Removing tile grout from brick

So the previous owners of my house laid ceramic tile over paver bricks (outside).

The tile is ugly and I've started removing it. The tile is popping up easy since it was not back-buttered, but the grout is not coming off of the paver bricks.

Is there any way to remove the grout from the paver bricks to restore them to their original look? Or will I just have to rip out the paver bricks as well and start new?

Thanks!