...couple of different threads running through this, including whether athletes tend to do better at their area of specialty, whether there are certain skills or fitness levels that someone should in general aspire to as a "cyclist," no matter what type of riding they specialize in, and whether certain types of bikes and riding are easier for people with low fitness levels and/or low skill levels.
Output-wise, you've got specialized fitness demands in racing that run along a line from BMX racing -- really incredible power -- to bike racers who abuse chemo drugs to keep their bodyweight low. Those are all bike racers.
Skill-wise, you also go from street bmx, dj, trials, and some types of freeriding, to the demands of riding a road bike. Riding a road bike does require far fewer bike-handling skills, but nordic racing also requires fewer skills than alpine and if you only ski alpine going out and skate skiing in particular is very humbling.
Most people know someone who is not real adventurous or fit who enjoys XC skiing every now and again, but who doesn't like alpine or telemark or snowboarding. Someone of them have lived in ski towns for a long time, and no one cares that they aren't seen on the lifts. If they're a cool person, they're still an integral part of the ski town. There are some people who live in ski towns who only play soccer all winter, that's also cool.
XC skiing is not a way, here at least, to try to display wealth or claim status or style the way some "man-seals" do thru their expensive road bike, yellow jerseys and talk on Monday am about their gnarly rides, so XC doesn't have that irritating component. You also don't have the ability on XC skis to claim priority over the rest of the population by riding two or three abreast, on places like PCH, while you suck and back up traffic the way people on bikes can do.
Skill-wise, you can look at each type of riding and come up with a list of what someone should be able to do to be "ok." For road biking, probably more people do come up short on the road bike list, but other than safety and etiquette issues it doesn't necessarily affect anyone else.