The Armistead House was built in 1890 by Cary Peyton Armistead, a prominent lawyer and head of the local Democratic Party, for his wife Eudora and their five children. Cary’s father, Robert Henry Armistead, owned much of the land surrounding the Governor’s Palace in Williamsburg. In 1889, Cary Peyton Armistead purchased the property known as the Morrison House, which had previously been the colonial-era Richard Charlton’s Coffeehouse. Armistead removed the 18th-century building but incorporated elements such as the original foundation, timber, and bricks into his new home. This house served as his primary residence and law office until his death in 1901. The southwest parlor, now the lobby of the Armistead House, originally functioned as his office with its own entrance.

Cary Peyton Armistead’s two daughters, Dora and Cara, never married and were the last of his five children to live in the house. They resided there well into the late 20th century. After Cara's death, the house became known as the “Dora Armistead House,” likely to distinguish it from the nearby Greek Revival Bowden-Armistead House.

During the restoration of Williamsburg to its colonial period, John D. Rockefeller, through his agent W.A.R. Goodwin, sought to purchase the Armistead property. However, the Armistead family refused to sell, making their home one of the few post-colonial structures remaining in the recreated 18th-century landscape. This refusal reportedly caused tension between the Armistead heirs and the Colonial Williamsburg Foundation for decades. From the 1940s to the late 1970s, the Armisteads rented rooms in the house to tourists, taking advantage of their prime location in the historic area.

Eventually, the property was transferred to the Colonial Williamsburg Foundation with the stipulation that the building could not be demolished. In 1995, the Armistead House was moved to its current location, surrounded by other dwellings from the same period.

Julie and Clyde Nordstrom, longtime Williamsburg residents and graduates of The College of William & Mary, acquired the Armistead House in its unrestored state from the Colonial Williamsburg Foundation after the house's relocation. They began an extensive preservation, renovation, and restoration project, which was led by their son, Dennis Nordstrom (William & Mary class of 1983), beginning in early 2019. The house was added to the National Register of Historic Places on August 10, 2020. The restoration was completed, and the Armistead House was acquired by its current owner for operation as a historic inn in early 2022.