Personally, just stick to your guns, man. What usually (emphasis on "usually", which means that it isn't "always" the case) happens is that the nerds in high school end up being the cool ones in college and the "real" world beyond that.
Besides, when I was in high school the "stoners" hated the jocks who hated the nerds who hated the popular people who hated the geeks and so on and so forth. Most of those labels get tossed out the window by the time you reach your late 20s.
I'd keep xc skiing, but you might also want to take up downhill or BC/AT or another snow sport that will get you out and on the hill with some other kids your age.
I would never go so far as to call xc skiing "gay" (that's just stupid), but it does tend to be a sport that appeals to older folks (i.e. everybody I know who xc skis is in their 30s and didn't pick it up until their late 20s at the earliest).
The bottom line is that when you're in Jr. and High School if you do anything "out of the norm" then people are gonna "hate" on you. Sadly, that's the way of the world. Of course if you become totally bada$$ at xc and end up winning an olympic gold in the future, just think who will have the last laugh?
one final note, since this is predominantly a downhill/alpine skiing forum, you might want to check out some online communities that cater specifically to xc skiing. You'll definitely get more "love" from them.
stay strong young grasshopper!