Hi Gpaul; last time I was in Nanaimo, the economy was quite depressed, as it is many places throughout the island, due to logging slow-down. I was there in the off season however, but it did appear that there was an awful lot of business space permanently for rent. I wouldn't call it a small town (which is relative, I guess), but it has all the services one generally needs. As with all the SE portion of the island, weather is wonderful.
Victoria is certainly not in the million-plus class. I would guess the area has a population of about 400,000 but that could be off a little. The industries are not resource-based, so the economy is more stable and the atmosphere more cosmopolitan. Government (capital of BC), military, tourism, retirement services and the university come to mind as major employers. I am thinking it may what you are looking for, though a long drive from skiing.
I was in Whistler last week. We didn't bring our skis this time, but lots of people did, and still plenty of snow. Most gondola riders were mountain bikers! The Whistler gondola from mid-station down are down-hill bike trails, while skiers go to the top and ski the peak, red and emerald chairs. We mostly relaxed, shopped and rode bikes on the valley trails. Weird to see skiers and boarders while the tulips are blooming! I haven't tried the down-hill biking, but it looks like a blast! Its a must to rent specialty bikes with disc brakes and good suspension.
Have a great summer, my friend!