We are in a spoiler thread so I will answer your query.
To put it in the most basic terms, when the Governor and Woodbury attacked the prison a lot of people were killed, including Lori and her baby with a single shotgun blast. The girl from Woodbury, named Lilly, that shot Lori was so upset she had killed a baby she then shot the Governor through his remaining eye. The prison was abandoned by the survivors and they headed back out on the road.
eta: I attempted to answer your question without giving away everything that happened and who all died, but that information will come out below, so if you want to read the comics some day and have some surprises left, you should exit the thread now.
This post was edited on 4/2 at 1:26 pm