Thanks everyone for the input. Good thing that I am not the only one having this problem. But given how many people wears contacts these days, I am guessing that most of the people do not have have this issue. Would be helpful to hear what goggle is being used from the ones not haveing this issue.
As far as general eye dryness, I am fine except when I am skiing. Did not have to use eye drops for all the years I have been wearing contacts. Even though it does not fix the root cause of the problem, looks like drops will help. I will definitely give that a try.
Regarding the comment that the goggle does not fit well, I checked the seal of the foam against my face, and it was good with no gaps or spaces in between. I don't know if anyone notice, there are openings all around the frame of the goggle that is covered by coarse foam to allow for air flow. I have checked a few different goggles from my friends, and they all have this. The size and shape may be different but the design is similar. I did try taping off about half of these openings, and it did not help. Maybe I will try taping off all of it as someone suggested to see what happens. The concern here is that if all air flow is blocked, the goggle will probably fog up.
As to the type of contact lens I wear, it is soft lens. But due to the strong myopia and astigmatism I have, they are much thicker than normal, especially on the bottom part. They also cost a lot more, and having weekly disposables is not an option. The best I can get is 4 pairs for a year costing a few hundred $. Nevertheless, bring a pair as backup is a good idea until I get this problem solved.
I understand the concept of lens curvature. Two years ago when my eye doctor check me out, I was told the fit was great, and I continued to use the same brand/perscription lens which I had been using for 3+ years prior. So I am not convinced that the curvature is the problem. The other things is the air flow/dryness problem affects one of my eyes a lot more than the other, which also leads me to think this particular pair of goggle may be faulty.
I think I will at least try a different brand/model of goggle to see if it makes any difference. Thanks for the suggestion of racing goggles. I will do some research and decide what to get.