re: I think that they're four sides in this

You missed the point. The 2 basic reasons most folks who want Confederate symbols limited to museums, cemeteries, and battlefields are:

1. Confederates fought against the United States of America. Leaders like RE Lee and Stonewall Jackson broke the sacred oath they took at West Point Or when they entered the US Army). They are not singled out because they owned slaves.

2. Many if not most of these statues were erected as part of an attempt to whitewash Antebellum Southern history and glorify the movement for Jim Crow in the period 1880-1930.