By the 1840s, due to increased Atlantic trade, the U.S. Custom Service had outgrown their office in the Exchange Building. Congress appropriated funds in 1847 and a waterfront site known as Fitzsimmon’s Wharf on the Cooper River was purchased for $130,000. When construction began and while digging the foundation workers found the remains of Craven’s Bastion, a Colonial-era fortification. Due to the marshy location, a grillage of timber was constructed to support the weight of the building. In 1859, with South Carolina’s secession becoming more possible, Congress refused to allocate funds and construction ceased.

In 1867, Congress revived the construction and it was completed in 1879. The original design included a large dome and four porticos, one on each side. However, due to post-War financial concerns, only the east and west porticos were completed, eliminating the north and the south.

In 1903, President Theodore Roosevelt appointed William C. Crum, MD, graduate of the Avery Institute, University of South Carolina, and Howard University Medical School as port collector. Crum was the son of Darius Crum, a German American, and Charlotte C. Crum, a free woman of color. He grew up near Orangeburg. Returning to Charleston, he began his practice and later joined the staff of the Crum joined the staff of the African American–operated McClennan Hospital and Training School for Nurse and eventually became the hospital’s chief administrator and Charleston’s most prominent black resident. In 1883 he married Ellen Craft, the daughter of the famous fugitive slave abolitionists William and Ellen Craft of Georgia. 

The New York Times and the New York Herald criticized the nomination as ill advised. In South Carolina, U.S. Senator Ben Tillman and the editor James C. Hemphill of the Charleston News and Courier jointly denounced Crum.  Tillman exclaimed, “We still have guns and ropes in the South.” James C. Hemphill wrote that Crum “is a colored man and that in itself ought to bar him from office.” Tillman temporarily derailed the nomination, but Roosevelt kept Crum in the position through interim appointments, until his January 1905 confirmation. There are remnants of Dr. Crum’s tenure inside the Custom House. There is evidence of a metal structure placed in the front of his office giving the collector a safe place to hide. As well as a desk believed to belong to Crum. Resigning his office in 1909 when President William Taft took office. In 1910 he was appointed the Minister to Liberia. by Pres. McKinley, a more traditional posting for an African American.

In 1908, Dr. Crum appointed Mr. Clarence O. Brown, an African American, as collector of customs, where he served for nearly half a century. He later received the Treasury Department’s Albert Gallatin Award for distinguished service.