Originally named Femcliffe, this house was built on land which was part of a grant to W. Clarke in 1843. In 1848, it was sold to F. Mitchell, who built the two oldest rooms. In 1856 Ambrose Foss bought the property, and in 1857 Jules Joubert built the major portion of the house for Captain Henry Brown. A subdivision map of about 1858 shows the name of Jules Joubert on this house as well as 1 McBride Avenue (see also 25 Ernest Street). Surrounding streets are named for Joubert’s wife Ada (married in 1855) and their quick succession of children, Ferdinand, Ernest and Madeline. Henry Brown, who donated one third of the cost of building the Primary School in Alexandra Street, lived here from 1861 until 1866, when he let the house to Dr William Hall Palmer. In 1873 Dr Palmer moved to 1 McBride Avenue and Henry Brown returned to this house. Sands Directory described Brown variously over the years as JP, squatter and boatbuilder. By 1885 William Proctor, solicitor and later MLA, father of artist Thea Proctor, owned the house and lived here for many years.

On March 14, 1861, the proclamation of Hunters Hill as a municipality was read to residents in a ceremony held in this house; the proclamation had been signed by the Governor on January 5, 1861.