|Yeah, I would think it would defeat the purpose. Unless it is below 0 out I think I could deal with wearing the suit and some thermals underneath for under a minute of racing. Then I could just put my jacket and pants/shorts back on.
Just wait until the doofus before you takes out 4 gates with him while falling and the course workers are all honest, good people, but they sure as hell don't know how to screw in a gate properly and they need to have the race's tech delegate show them how right there on the course and all of this time you are... in the start gate, basically in your underwear, while your coat is now at the finish line, waiting for you...
I usually wear two layers under the suit, a very thin, stretchy thermal top and pants followed by something more substantial, like a low-weight fleece, and I'm usually the one who is sweating like a pig in minus 10 slalom training while in shorts
I know of some people who just rely on a cotton t-shirt and the bib to get their heat from, but these people are crazy.
PS: It isn't during, or after the race, that you are cold, it is before, while waiting for your turn at the start. Dress very warm and undress at the very last second if possible. Being cold and miserable wrecks havoc on race mindset.