Naturally, I've wondered about the history of that building, so let's investigate.
As the 1888 notice below tells us, a freight transfer line was established in Birmingham by George C. Harris in 1881. That first office was located--"conveniently"--upstairs at 5 20th Street. By the time this item was published, the firm even had a telephone. The business was "well and favorably known" and moved household goods, freight and even safes, which they would position for the customer.
In the 1910 U.S. Census we can find some information about Mr. Harris. He was born in Alabama in 1849, and had been married to wife Jamie for 35 years. He owned his home free of any mortgage and could read and write. His household had a total of ten people, including George C. Harris, Jr. His occupation is listed as "transfer man". The Alabama Deaths & Burials Index 1881-1974 gives his death date as January 26, 1912.
As the BhamWiki entry on the company tells us, the firm's success continued in subsequent decades. By the late 1930's Harris Transfer Company had sixty trucks and three warehouses. You can see a photo of one of those trucks on that site. The company became one of the early partners with Allied Van Lines in creating a long-distance hauling system.
In the 1990's the company fell on hard times, filed for bankruptcy and was liquidated. Floors above the Fish Market are used in a self-storage business.
Source: North Alabama Illlustrated, 1888, p. 144
Here's another view of that sign with UAB's Jefferson Tower looming in the background.
A recent photo of the more or less outside dining area; main dining is to the right
