Boston stockbroker Charles Hardy and his associates, who formed the Chatham Trust Company, built the Brick Block building in 1914. The building was home to the Chatham Trust Company offices for several years, and housed the post office until 1936. With residential apartments on the second floor, this was one of the first buildings to fulfill a growing need for both commercial and residential space on Main Street.

Over the course of almost 100 years, it has been home to a variety of businesses, including a billiard parlor, restaurants, jewelry stores, a meat market, a drug store, a photography studio, and several ice cream parlors. For many years this intersection was called Post Office Square, and was considered the “center of town“ in the growing Main Street area.

Charles Hardy and Associates also built Chatham Bars Inn in 1914, and Hardy was one of the financial backers for the Eastward Ho Golf Course. Harvey Bailey Alden designed the Brick Block, and Joseph W. Nickerson was responsible for the decorative brickwork. The Brick Block was placed on the National Register of Historic Places in 1979.

One of the early businesses in the Brick Block was the Mayflower Photography Studio, owned by Charles Smallhoff. In 1917, he moved his business across the street into the building at 475 Main Street, which had been home to tailor shops since 1885. Probably in the 1920's, the name was changed to the “Mayflower Shop”, and the business was expanded to more of a variety store. The Mayflower shop has had a couple of owners early on, but it was run by the Ward family for more than fifty years, from 1954 until 2011.