Just leave it alone. The worst thing you can do is try to do ANYTHING about it. And no - if you try to remove it all, you'll wish you hadn't.
Lenses don't cost that much anyway. If it bugs you that much, just buy a new lens.
To address one thing DCNB mentioned - people think that "lens cleaning cloths" were some kind of magic bullet for taking care of lenses. The failure here is that everyone was dry wiping the lenses over and over with the same cloth. They never cleaned the cloth. These cloths were a microfiber that picks up dust, dust, etc., and every time you used them, you were adding build up. Eventually, you were grinding that into the lens.
The best thing you can do is use a soft CLEAN cotton cloth (I use cut up t-shirts) and a mild LENS CLEANER (no Windex!!!). Lenses need to be damp (on the outside at least) to be cleaned while avoiding damage. Dry wiping is bad. Using the same old lens cleaning cloth is bad. Windex is TERRIBLE. Ski lenses are durable from impacts and so on, but really suck for scratch resistance. They are usually a softer polycarbonate, and are therefore pretty easy to scratch.
I should mention that I worked in an optical lab for 3 years, so that's where my otherwise uneducated spewing comes from.