Hermes and his sister Vasso were eventually raised Catholic by his mother, the former Mary Huston. Penniless, they moved to the poorest area of New York where Hermes learned his first tap dancing steps on the streets from local black children. At their lowest ebb, the family had only some potatoes and coffee for their meal. They vowed to commemorate that day and every year on June 13, they wrote down what they did and ate that day in a journal. The date became a family celebration. Eventually the family headed West with "Sammy", a black boy they met at a gas station, to Los Angeles, California.
Pan's career began with an appearance as a chorus boy in 1928 in the Marx Brothers Broadway production of Animal Crackers. He also danced in partnership with his sister Vasso, who subsequently appeared in the chorus of many of the Astaire-Rogers pictures. He first met Ginger Rogers in 1930, when he appeared as a chorus singer in the Broadway musical Top Speed.
He retained links with his relatives in Greece and made an extensive visit in the 1970s to meet them.
Fred Astaire, whom he physically resembled, on the set of Flying Down to Rio (1933), in which he worked as an assistant to dance director Dave Gould.
While Astaire was trying to work out a series of steps for "The Carioca" number, it was suggested to him that Pan had a few ideas and Pan was invited over. Hermes Pan demonstrated a brief break he had picked up from his street days in New York.
From then on the two began a lifelong professional collaboration and friendship which included all the RKO Astaire pictures, including A Damsel in Distress (1937) in which Ginger Rogers did not appear, and for which he was awarded the 1937 Academy Award for Best Dance Direction. He had previously received Academy Award nominations for the "Top Hat" and "The Piccolino" numbers from Top Hat (1935) and for the "Bojangles of Harlem"' number from Swing Time (1936).
The Astaire-Pan collaboration, involving 17 out of Astaire's 31 musical films and three of his four television specials, is widely accepted as one of the most important forces in dance choreography of 20th century film and television musicals. Astaire called Pan his "ideas man", and while he generally choreographed his own routines, and sometimes worked with other choreographers, he greatly valued the assistance of Pan not just as a source and critic of ideas, but also as a rehearsal partner for the purposes of fine-tuning a routine.
Given Astaire's obsessive rehearsal habits, this was no mean task. Pan also performed the essential function of rehearsing Ginger Rogers, whose many other commitments during the filming of the Astaire-Rogers musicals often conflicted with Astaire's rehearsal schedule. In addition, he recorded Ginger's taps in post production in some numbers.
Pan continued to collaborate with Astaire right up until the latter's last musical picture, Finian's Rainbow (1968), which was a disaster on a number of fronts, not least for Pan himself. The young director Francis Ford Coppola had no prior experience with musical films, and proceeded to ride roughshod over Astaire and Pan's plans for the film's dance routines, reintroducing the style of dancing camera of the early 1930s which Astaire had done so much to banish from the Hollywood musical. Eventually, Coppola fired Pan, who had a small walk-on part in the film; Coppola has since acknowledged his own primary responsibility for the film's artistic failure.
Pan was very close friends with Rita Hayworth and he served as a pallbearer at her funeral.
According to John Franceschina in his book Hermes Pan: The Man Who Danced with Fred Astaire, it was well known in the movie industry that Pan was gay, but it never became public and the identity of his partners is not known.