Born in Kilmarnock, Scotland, Hart moved to Albany, New York with his family at the age of two. There, he trained as an apprentice under a sign and carriage maker, but returned to Europe for artistic training in 1851 at age twenty-three. He studied in Munich under Friedrich Wilhelm Schirmer at the Kunstakademie Düsseldorf. Before returning to America permanently in 1853, Hart exhibited his first work at the National Academy of Design in 1848. He was elected an associate in 1857 and a became a full academician in 1859. He would exhibit there for over forty years, and serve as Vice President from 1895 to 1899, ending his tenure just two years before his death.

Based in New York City, James McDougal Hart-like his brother William-painted in the style of the Hudson River School. Unlike his brother, who painted mostly small and mid-sized works, James favored painting larger landscapes, a likely result of his exposure to the large realist works of the Düsseldorf School. He was known for his renderings of livestock, specifically cattle. His major landscapes are considered some of the most important works to come out of the Hudson River School; a notable example is his Summer in the Catskills, now in the Thyssen-Bornemisza Museum in Madrid.