Built in 1931 for Fifth Avenue Bank of New York (later the Bank of New York), 909 Madison Avenue replaced a Queen Anne Style rowhouse designed in 1886 by owner/architect Charles Buck & Company as one of a row of four. The present structure was designed by architects Schultze & Weaver in the Neo-Federal style. Located on New York's Upper East Side, the now landmarked building is just blocks from the Metropolitan Museum of Art, the Frick Collection, and the Solomon R. Guggenheim Museum.

In January 2017, the New York gallery expanded to occupy the full premises at 909 Madison Avenue, where since its opening it has presented critically acclaimed exhibitions on the upper two levels. With the expansion, Lévy Gorvy significantly increased its exhibition space through the addition of the ground floor, while expanding its private viewing rooms and research facilities on the lower level. The gallery commissioned the renowned architect Bill Katz to redesign the building’s ground floor with the goal of enhancing the presentation of both intimate and large-scale projects.