Someone who knows more about the voodoo that is DNS can correct me if I'm wrong, but I believe that in practice what happens is that the primary server gets queried for the DNS lookup, and if it responds in any
way the secondary/tertiary/quaternary servers don't get questioned.
So if the problem is that your primary server is wrong (as opposed to broken, in a power outage, etc) you will still get the wrong result despite your secondary server being correct.
So the key is to pick the DNS server you are most confident in and put it in the primary slot. I've got my ISPs servers for backup.
Basically what it comes down to is ISP's being cheap, incompetent, or both and having crappy DNS service...they'll ignore secondary nameservers, ignore TTLs, block domains they don't like, etc. DNS is really voodoo stuff though where "standards" are a joke and everyone does things a bit differently. Usually it just works
and most people take it for granted. Read the wikipedia article
to get an idea as to what a mess it is.