A lot of factor cause people to fall out of the sport.
The above tend to be a common reason among younger people, who are physically capable, and often still have an interest in the sport, but reduce or complete stop participating in skiing. Some do return at different stages in their life, when some of these factors change. Very common to see people in their 20s/30s/early 40s succumb to these reasons. Being in this age group myself and dealing with these challenges I totally get it. Getting married, buying houses, dealing with a career, having children, etc and only having so much leisure time and/or disposible income limits many. Let's me honest, most people on this board are die-hards, but many skiers are not. The time investment for skiing and related-travel is real, the money is absurd for many, and people have other things going on in their life. Its a real issue the industry has to deal with, and frankly I think in many places they could do a better job.
Pursue other interests
This can happen to any age group, but in my opinion tends to happen at two major junctions. Late teens/early 20s, and people in the 40s/50s. Often I see that children who grew up skiing, and often times even those like said were in race training programs give it up during their college years. They go away to school, get involved in other interests, and often if they don't have hardcore skiing parents (or enjoyed their family time) drift away from the sport. Sometimes its due to burn-out from the racing and training. (I had a similar issue with running that I walked away from the sport for 4 years and now am fully back into it). You also see this with people in the mid-life or even at retirement, maybe they aren't improving are looking for a change of pace. Maybe they want to get into another (expensive sport) or move to Florida.
Level of Fitness
This can happen too to any age group, but in more particular happens to women, or people that are older in their 50/60/70s. Some face an injury and struggle or don't want to come back. A bad accident can scare some. Some people don't want to risk injury as they get older.