This thread has been a bit derailed by altitude concerns. I am in my mid-40s and have been making trips to Colorado since I was a teenager. I had absolutely no issues with altitude until I was 30...and now I can pretty much count on not sleeping all that well above 7500 feet, but that's really the only problem I experience. Every now and then I might feel a headache coming on, but just drinking more water than usual seems to immediately solve that issue.
Back to your original question of where you should go...I'm a huge Telluride and Aspen fan-boy, so those are the spots I would recommend since you haven't been to either of them. They are both a bit pricey, but worth it as you will not have crowds to deal with and have all sorts of wonderful terrain. I agree with @crank that staying in town is probably the way to go should you decide on Telluride (as opposed to Mountain Village).