Figtree House was built in three stages. Mary Reibey (Reiby), a well-known merchant, named the property after a Port Jackson Fig, beside which she had two small cottages built in 1836, probably to house convict labour for the farm. It is said that she delighted in using the farm with orchard as a retreat from Sydney, but in 1838 the farm was offered for sale (see Figtree Area). It was not sold, and Joseph Fowles leased the farm as did subsequent tenants, until Didier Joubert purchased the entire property (which Mrs Reibey had increased by acquiring neighbouring land grants) in 1847 for £500, paid in instalments over four years.

In 1848 Joubert added a two-storey stone wing and paved verandah, making one house out of the two cottages. It has been written that this was the first home of the Jouberts in the district and that they lived here while building St Malo (now demolished). It has also been written that they lived in Reibey Cottage until St Malo was ready for occupation. Certainly Sands Directory, when listing occupants, does not distinquish among the buildings on Figtree Farm until after Didier Joubert’s death in 1881. His son Numa Joubert added the timber tower, which was constructed by shipwright Joseph Howard, who lived here for some years. Because of its tower, this house has long been a landmark in Hunters Hill, associated through many decades with the ferry service (1871-1905) and when that ceased, with boatsheds and a jetty where the Figtree Bridge is now. Figtree House is the sole remaining building from Figtree Farm.