I was in BW and Silverstar for a total of three weeks this year with 3 kids under 12 plus a nanny.
In BW I would suggest you stay at either Stonebridge Lodge, Trappers Crossing or Sundance. That is provided you want apartment style accommodation. Make sure you get a three bedroom apartment plus loft. Trappers and Sundance are probably the newest and closest apartment complexes to the village. I would probably choose Stonebridge over Trappers because the views can be better, it also has a heated swimming pool which is worthwhile for the kids.
We actually stayed at a place called Sundance. It is a bit further away from the village but chose it because of the fantastic kids facilities. Heated swimming pool with slide and 2 hot tubs, theatre room with movies at 4.30pm and 7.00pm, games room with play stations, pool table. It is probably about a 200 mtr walk to the village centre with a bit of a dip in the middle. Not so convenient if you need to get small kids to the ski school at 9.00am every morning.
Weather wise BW can suffer from a bit of low fog and is sometimes refered to BIG WHITE OUT. Iv'e been their twice and had about 3 foggy days out of 15.
If your looking at house style accomodation then all of these are more remote from the village and so you would definately need a car IMO.
Over at Silver Star we stayed a bit further away from the village in a large 4 bed house called Mainstay. It was still within walking distance but it is a hill walk. We always get a hire car at the airport so staying a bit further from the village is never an issue for us and we occasionally took the walk to check out the cute western style themed houses nearby. If you go for the house option make sure you stay very close to the top of the Silver Queen chair lift. This chair is a really good beginners area and next to it is the tubing park, the ice skating area and the mini snow mobiles for the kids.
Our house Mainstay was well positioned for all this but there were a few houses right at the top of the chair that would have been in an even better location with even better views. There are also some town house accommodation at the bottom of the Silver Queen which would also be worth considering however they don't have the elevated views.
Down in the village there seemed to be mainly hotel style accommodation which I don't know too much about.
Someone mentioned Fernie, we spent 10 days in Fernie this year and had a great time. We stayed in a large three bed plus loft serviced apartment at Lizard Creek Lodge which was fantastic. Lizard is probably the pick of the accommodation choices in Fernie with a wonderful lounge bar area and adjoining restuarant. They also have their own ski rental facility, heated pool and hot tubs. We went to Fernie primarily to be involved in a special ski instruction camp run by a couple of Aussie guys. Easily the best ski instruction I have ever recieved, see this web site:
You asked about driving times. We drove from Fernie to Big White and that was about 9 hrs. Banff to Big White would be further than that I guess although it is probably a different route. I have not done Whistler to BW but it would be similar, about 8-9hrs. Big White to Silver Star is about 1.5hrs. Maybe a bit too much driving for just a three week stay I would say.
If you are driving with luggage between resorts you must rent something as large as possible IMO. Get a GMC Suburban or an equivalent. Heaps of space and they handle pretty well.
If you need a good travel agent in OZ to sought it all out I can recommend someone who did ours. She has extensive knowledge about BW, SS and Sun Peaks.