The problem is that some invasive species are very harmful to the natural environment and humans as well.
Water Hyacinth chokes streams, kills fish, and turns otherwise navigable water ways into a vine-choked rat's nest.
Neutria Rat's destroy levies, endangering human settlements in low lying areas. They also eat marsh grasses which make marshy areas far more susceptible to coastal erosion.
Chinese Tallow trees grow extremely quickly and are resistant to most weed killers. They destroy fences, crowd in fields, kill crops, and choke out native trees. Because these native trees provide food for many herbivores (seeds and nuts), the killing of those trees by the new tallows (whose fruit is not eaten by native animals), leading to a ripple in the food chain that can cause a massive die off.
Boas are dangerous to people. They also push gators out of their habitat by supplanting their niche as apex predator. Humans lose a great resource in alligators due to their die off without gaining a new food and leather resource. Also, boas are much more dangerous to humans and pets than gators.
Wild boars dig up and destroy underbrush. This loss of brush destroys the habitats and food source for many prey animals, denying larger predators food. Boars are also highly aggressive towards humans and pets, causing a public danger. Once reaching maturity, they have no natural predators other than man, so if unchecked, they can reproduce almost exponentially.
All this, plus the environment is kinda like a jenga tower (to use a rough metaphor that is appropriate in THIS situation).
It may seem simple enough to just let the invasive species take over and eradicate some other species, but how does that affect the rest of the environment.
It may turn out that nothing happens and that the missing species was relatively unimportant and the environmental jenga tower stands strong.
It may turn out that the eliminated species was a crucial support and the environmental jenga tower comes collapsing down to devastating consequences.