On August 10, 1896, Lot #2 of the former U.S. Arsenal was deeded to Josephine Poe, wife of Edgar Allen Poe. This was the land on which E.A. Poe began construction of his two-story frame house, along with a barn, woodhouse, smokehouse and wellhouse in 1897. Poe, a successful businessman, politician and civic leader, was one of several person to settle in the Haymount area at the turn of the century.

Built by Ruffin Vaughn, the Poe House exhibits rare Eastlake detailing. Other architecturally significant elements of the house include the entrance bay, a wrap-around porch, exterior sawnwork, tongue-and-groove wainscoting and bullseye molding throughout the interior. The house is owned by the Museum of the Cape Fear. Following its recent restoration, it serves as a historic house museum interpreting late nineteenth and early twentieth century social, cultural and family history.