Born in an agricultural district of the Ukraine, Olinsky displayed a precocious artistic ability at a very young age. At ten years old, he was invited to join the drawing class at the university in Elizabethgrad, near Odessa. Here, he worked with students twice his age. In 1890, his family emigrated to the United States to settle in New York City. At sixteen he won admission to the National Academy of Design to study under Francis Coates Jones, Edgar Melville Ward, Charles Yardley Turner and George W. Maynard. In 1900, a friend recommended Olinsky to John La Farge who was looking for a young man to "correct" his drawings and assist in the studio. Olinsky appreciated the opportunity to work with such an exceptional artist who was arguably the most versatile of his era. He renewed his contract with La Farge two or three times until 1908, when Olinsky said that "it got so that unless I found out what I could do by myself, I would give up art."

So it was that Olinsky took his new wife on a tour of Italy and France to study the old masters in the Louvre and paint scenes along the canals of Venice. In 1910, Olinsky returned to New York in time to serve as pallbearer at La Farge's funeral. After this sad event, Olinsky set up a studio in Washington Square and began to paint his wife and daughters. These pictures drew such attention that Olinsky now found himself in the portrait business.