A Washington native, Duffey started the Seldom Scene in 1971 after about a decade of playing with Charlie Waller & the Country Gentlemen, a bluegrass group from the same general region. The mandolinist was also a musical instrument repairman, a profession that managed to provide him with an alternative to being on the road all of the time, something with which he apparently had very little patience. One reason to form the new band was the possibility for it to work regularly without straying too far from the Washington, D.C./Virginia/Maryland axis. Certainly there would be gigs enough within a two- or three-hour driving ratio, especially with bluegrass coming off a peak in new popularity from the late '60s. He enlisted only other players whose demanding day jobs would prevent them from whining about wanting to go on long road trips. These original players were physician John Starling, mathematician Ben Eldridge, graphic artist Mike Auldridge, and National Geographic mapmaker Tom Gray. Also known as mandolinist John Duffey, banjo player Ben Eldridge, guitarist John Starling, bassist Tom Gray, and Dobro player Mike Auldridge. But the part-time nature of each of these players' musical focus in the early '70s would have probably dictated that they would have listed their day jobs first. The group was even called the Seldom Scene as a joke about the fact that they wouldn't be seen much on-stage. What happened was the opposite. By staying right out on the edge in an age of great musical adventurism among the audience, the group became much in demand, as well as producing some of the best-selling progressive bluegrass records in the history of the genre. Another part of Duffey's success as a member of co-operative bands was his belief that democracy worked in music ensembles, regardless of whether it seemed to be working in society. The mandolinist's philosophy seems to have been borne through his loyal membership to just two different bands over the 40 years of his career. He grew up in a musical family, although what he was exposed to at first was about the farthest one could get from bluegrass. His father was a professional singer who at times worked for the Metropolitan Opera. As a young man, Duffey became attracted to the music of Appalachian migrants in the area. He was not particularly concerned that amongst the classical or so-called legitimate music crowd such sounds had very little status. Despite his own lack of enthusiasm for so-called hillbilly music, Duffey senior realized that his son seemed to have inherited an exceptional singing voice, with a range of about four octaves. His father went ahead and taught him the voice and breathing techniques of a classical opera singer. Duffey continued his love affair with Appalachian music, but since he realized he wasn't in any way a native of that area, he focused on expanding the concept of the music to include people like him. He created new repertoire from modern and ancient sources and developed innovative vocal harmonies. What he did was definitely pleasing to the large new bluegrass audiences, although many purists found the new developments and outlooks being expressed in the music revolting. The controversy became part of the reason crowds packed several long weekly house band stints the group maintained in the D.C. area. Members of Congress in suits would be rubbing noses with college students, as well as disgruntled members of the bluegrass "walking dead," (i.e., those who wanted everything to be done the way Bill Monroe done done it).

Duffey's professional career began with a car crash in 1957 that injured mandolinist Buzz Busby. The banjo player in the same band as Busby, Bill Emerson put out a bee that he was looking for a substitute mandolinist so the band wouldn't have to cancel its schedule of club dates. In his search for potential players, Emerson found both young guitarist Waller and Duffey, as well. It was the mandolinist who came up with a name for the new group. He pointed out that many bluegrass bands at this time were coming up with names like the Mountain Boys. "We're not mountain boys," he said. "We're gentlemen." Duffey wound up staying with the Country Gentlemen for about a decade, as the group rode the new wave of folk music thrill-seeking. Many of the innovations of the Seldom Scene are foreshadowed by this earlier group, such as a diverse selection of material that could include gospel, jazz, and folk influences. By the late '60s, Duffey was working as an instrument repairman at an Arlington music store when the Seldom Scene was formed. In addition to much research, collecting, and arranging old songs and poems for the group, Duffey also composed his own music. Some of his best pieces include "The Traveler," which was dedicated to his wife, and the eerie "Victim to the Tomb."

As this group became more and more popular, Duffey's imposing stage personality came more and more to the front. Known for being able to shut the lid on just about any heckler, Duffey has been described as one of the most riotous personas in bluegrass, famous for his politically incorrect jokes and onstage shenanigans. Although the pre-bluegrass genre of old-time music was known for crazy stage shows and broad humor, bluegrass by this time had developed into a style most often represented visually by bandmembers who stood straight as a board, their faces expressionless no matter what they were picking. And the new comic approach presented by Duffey was much more sophisticated than the laugh-grabbing days of blackened teeth and hillbilly yuck yucks.

Duffey, along with former boss Waller and the other original Country Gentlemen, were inducted into the International Bluegrass Music Associations' Hall of Fame. The Seldom Scene continued to be active up to the end, playing in Englewood, New Jersey, just days before Duffey's death. They were also working on arrangements for a new recording, including an adaptation of the Delta blues number "Rollin' and Tumblin'," for a progressive bluegrass band, of course. The posthumous Always in Style project was released under Duffey's name on the Sugar Hill label. Although the best recorded legacy of the mandolinist's work are on the recordings of the groups he was in, he also pops up here and there as a session picker, including on a Linda Ronstadt album.