You're right - no hard and fast rule...couple of things to check on your boots though, which would let you know if they are worn out or wearing out. First off, the sole condition - if it is gouged up in the toe and heel area (deep scratches) then your friction interaction with the binding may be in jeopardy - replace the toes and heels or replace the boot. If your boots are beginning to resemble elf shoes - turned up at the toe and heel from walking on hard surfaces (parking lots, etc) - time to replace them. If you have all of a sudden realized that shell fitting has a purpose (your feet are sliding back and forth and causing you to lose toe nails)...time for a new boot. If there are cracks in the plastic - time for a new boot.
Many boots have the 'born on' date stamped inside the cuff of the shell. Pull your liner and look for an embossed year (98) and that will tell you when your boot came off of the production line. Plastic has a shelf life that can be affected by temperature, treatment, salt (summer snow), wear and tear. If you find a number that is older than 6-7 years, you might want to think about replacing, IMO.