Your assumption is that when people use it, they are strictly applying the dictionary definition, and not referencing an unwritten but well-understood cultural context; I would disagree with this basis. Indeed, in order to accept that a dictionary definition is even relevant here, we would have to believe that words never change their meaning once defined, or that they only change their meaning when that new definition appears in the dictionary and not apart from that event.

There is, in my opinion, a racial undercurrent to the word and that it is not (merely) a reference to the quality of roads or sidewalks in the ghetto, but to those who live there. That these qualities apply to the people who live there too.

BTW, from Wikipedia's page on "Ghetto":

"A ghetto is a section of a city occupied by a group who live there especially because of social, economic, or legal pressure."

Emphasis mine. Economic pressure, e.g., they can't afford to live elsewhere.