Rubbing Compound on Brushed Metal Rims

I recently bought a pre-owned Toyota Tundra with factory 20" brushed metal rims. The rims have some kind of film on them that looks like someone maybe tried to polish them but didn't buff off the polish, and now its dried on. I've tried three different types of wheel cleaners, including one that is supposed to remove oxidation, but none have worked. I've gotten one clean using a plastic spatula to scrape it off, but it took two hours, and was not easy. My question is: would rubbing compound damage the metal rim, and leave swirl marks if I used it to remove this stuck on film? I do not want to mess up the wheels, but I got to get this shite off, its driving me crazy. Any help would be very much appreciated. Also, the rims are partially black so I dont think I can use any type of stripper because it may remove the paint on the black parts.