Correct me if I am wrong, but this is what an emulator does, and is all software.
There are certainly some exceptions I'm not aware of, but backwards compatibility on the PS2 and PS3 is/was handled via a physical chip on the motherboard.
From what I've read, the PS1 chip on the PS2 was actually the PS2's sound chip when playing PS2 games.
The PS2 chip on the PS3 motherboard was removed in later versions of the console to cut costs. There was software emulation at some point for select PS2 games but Sony removed it from later models because compatibility issues continued to crop up (and they probably got tired of supporting it). This is why the current PS3 has no backwards compatibility with PS2 games.
The problem with software emulation is exactly what Sony ran into. Designing software to emulate all of the various nuances of a physical piece of hardware requires a lot of trial and error and revisions.
You can see this by just running a PS2 or Gamecube emulator on your PC. The big titles usually run fairly well, but you'll run into weird glitches all the time on other games. It took a long time for the current SNES and Genesis emulators to get where they are today, with perfect (or near) emulation. I remember playing FF3 and Chrono Trigger on early versions of SNES9x and having to turn off various graphical layers to make some parts of the game playable.