McPhee's first recordings as leader appeared on the CjR label, which he founded in 1969 with painter Craig Johnson. These include Underground Railroad by the Joe McPhee Quartet in 1969, Nation Time by Joe McPhee in 1970, and Trinity by Joe McPhee, Harold E. Smith, and Mike Kull in 1971. By 1974, Swiss entrepreneur Werner X. Uehlinger had become aware of McPhee's recordings and unreleased tapes. Uehlinger was so impressed that he decided to form the Hat Hut label as a vehicle to release McPhee's work. The label's first LP was Black Magic Man, which had been recorded by McPhee in 1970. Black Magic Man was followed by The Willisau Concert and the landmark solo recording Tenor, released by Hat Hut in 1976. The earliest recordings by McPhee are often informed by the revolutionary movements of the late '60s and early '70s; for example, Nation Time is a tribute to poet Amiri Baraka and Joe McPhee & Survival Unit II at WBAI's Free Music Store, 1971 (finally released as a Hat Art CD in 1996) is a sometimes anguished post-Coltrane cry for freedom and liberation. But for the most part, McPhee's later artistic endeavors have been driven less by politics and more by the joy of sonic exploration for its own sake.

As the 1980s began and with a number of Hat Art recordings under his belt, McPhee met composer, accordionist, performer, and educator Pauline Oliveros, whose theories of "deep listening" strengthened his interests in extended instrumental and electronic techniques. McPhee also read Edward de Bono's book Lateral Thinking: A Textbook of Creativity, which presents concepts for solving problems by "disrupting an apparent sequence and arriving at the solution from another angle." De Bono's theories inspired McPhee to apply this "sideways thinking" to his own work in creative improvisation, resulting in the concept of "Po Music." McPhee describes "Po Music" as a "process of provocation" that can be used to "move from one fixed set of ideas in an attempt to discover new ones." He concludes "It is a Positive, Possible, Poetic Hypothesis." The results of McPhee's application of Po principles to creative improvisation can be heard on several Hat Art recordings, including Topology, Linear B, and Oleo & a Future Retrospective.

Although his work was well documented on Hat Hut, McPhee had never secured a contract with a United States-based label and was still a relative unknown in his home country as the 1980s progressed and he went on hiatus to care for his aging parents. McPhee re-emerged into the performing and recording world during the 1990s and finally began to attract wider attention from the North American creative jazz community. He has since been performing and recording prodigiously as both leader and collaborator, appearing on such labels as CIMP, Okkadisk, Music & Arts, and Victo. In 1996, 20 years after Tenor, Hatology released As Serious as Your Life, a second solo recording (this time featuring McPhee performing on various instruments). McPhee also began a fruitful relationship with Chicago reedman Ken Vandermark, engaging in a set of improvisational dialogues with Vandermark and bassist Kent Kessler on the 1998 Okkadisk CD A Meeting in Chicago. The Vandermark connection also led to McPhee's appearance on the Peter Brötzmann Chicago Octet/Tentet three-CD box set released by Okkadisk that same year.

As the 1990s drew to a close, McPhee formed a trio with bassist Dominic Duval and drummer Jay Rosen and premiered the new ensemble at a New York City jazz festival. Given the lack of press attention they received, McPhee, Duval, and Rosen decided that an apt title for the group would be Trio X. A number of Trio X recordings have since been released on Cadence Jazz and CIMP, including Rapture (1998), The Watermelon Suite (1999), On Tour: Toronto/Rochester (2001), In Black and White (2002), Journey (2003), and The Sugar Hill Suite (2004), and the band has received favorable critical notice for these recordings as well as for its live concert and festival appearances. In October 2008 Trio X embarked on a tenth anniversary tour in the U.S. and also released a limited-edition seven-CD box set documenting seven concerts recorded in the Midwest and New York state. The 21st century saw McPhee release No Greater Love (2000), Voices: Ten Improvisations (2008), and Tomorrow Came Today (2009), among others. With a career now spanning over 30 years and appearances on over 100 recordings, Joe McPhee has shown that emotional content and theoretical underpinnings are thoroughly compatible -- and in fact, a critically important pairing -- in the world of creative improvised music.