what is your backpack made out of? Silk? Can air move through it?
That is my best guess at why your stuff is getting so cold.
Perhaps you need a more outdoorsy backpack where the fabric is waterproof or at least windproofed, which will keep the cold air out. If your backpack has vents or holes designed to allow air to move through it (or water to drain out), then you should cover or plug those holes for skiing. I use a hiking backpack which has these drains, and I throw in a plastic grocery bag to line the bottom when skiing.
Also what else do you put in your backpack? I assume at least a layer of clothing. arrange that around your sandwich, and try to get the bundle closer to your body side so it has the most insulation to the elements.
If you have a water bladder, you can fill that with warmer water, or at least room temp water, which should keep the whole backpack warm.