He was one of the founding members of the Jazz Crusaders, the band which became simply The Crusaders in 1971, and remained a part of the group until its final album in 1991 (not including the 2003 reunion album Rural Renewal).
Sample began playing the piano when he was five years old. Since the early 1980s, he has enjoyed a successful solo career and has guested on many recordings by other performers and groups, including Miles Davis, George Benson, Jimmy Witherspoon, B. B. King, Eric Clapton and Steely Dan. Sample incorporates jazz, gospel, blues, Latin, and classical forms into his music.
In high school in the 1950s, Sample teamed up with two friends, saxophonist Wilton Felder and drummer "Stix" Hooper, to form a group called the Swingsters. While studying piano at Texas Southern University, Sample met and added trombonist Wayne Henderson and several other players to the Swingsters, which became the Modern Jazz Sextet and then the Jazz Crusaders, in emulation of one of the leading progressive jazz bands of the day, Art Blakey's Jazz Messengers. Sample never took a degree from the university; instead in 1960, he and the Jazz Crusaders made the move from Houston to Los Angeles.
The group quickly found opportunities on the West Coast, making its first recording, Freedom Sounds in 1961 and releasing up to four albums a year over much of the 1960s. The Jazz Crusaders played at first in the dominant hard bop style of the day, standing out by virtue of their unusual front-line combination of saxophone (played by Wilton Felder) and Henderson's trombone. Another distinctive quality was the funky, rhythmically appealing acoustic piano playing of Sample, who helped steer the group's sound into a fusion between jazz and soul in the late 1960s. The Jazz Crusaders became a strong concert draw during those years.
While Sample and his band mates continued to work together, he and the other band members pursued individual work as well. In 1969 Sample made his first recording under his own name; Fancy Dance featured the pianist as part of a jazz trio. In the 1970s, as the Jazz Crusaders became simply the Crusaders and branched out into popular sounds, Sample became known as a L.A. studio musician, appearing on recordings by the likes of Joni Mitchell, Marvin Gaye, Tina Turner, B. B. King, Joe Cocker, Minnie Riperton and Anita Baker. In 1975 he went into the studios with jazz legends Ray Brown on bass, and drummer Shelly Manne to produce a then state-of-the-art recording direct to disc entitled The Three. About this time Blue Note reissued some of the early work by the Jazz Crusaders as “The Young Rabbits.” This was a compilation of their recordings done between 1962-68.
The electric keyboard was fairly new at the time, and Sample became one of the instrument's pioneers. He switched to electric keyboard for his recordings with the Crusaders themselves, and the group hit a commercial high-water mark with the hit single "Street Life" and the album of the same name in 1979. In 1978 he did a joint session with guitarist David T. Walker, Swing Street Café, which had all the feel of a live set done in a back street joint in Texas.
The Crusaders, after losing several key members, broke up after recording Life in the Modern World for the GRP label in 1987. Despite the disbanding of the Crusaders, the members would join each other to record periodically over the years; releasing Healing the Wounds in the early '90s. Felder, Hooper, and Sample recorded their first album, called Rural Renewal, as the reunited Crusaders group in 2003 and did a live concert in Japan in 2004.
Since Sample's Fancy Dance (1969), he has recorded several solo albums, including the George Duke produced Sample This.
GRP also released Joe Sample Collection, and a three disc Crusaders Collection, as testament to Sample's enduring legacy. The pianist's most recent recordings are The Song Lives On (1999), featuring duets with singer Lalah Hathaway, and The Pecan Tree (2002), a tribute to his hometown of Houston, where he relocated in 1994. His 2004 album on Verve, Soul Shadows, paid tribute to Duke Ellington and Jelly Roll Morton, and pre-jazz bandleader James Reese Europe. In 2007 he recorded Feeling Good with vocalist Randy Crawford.
Some of his works are featured on The Weather Channel's "Local On The 8s" segments and his song "Rainbow Seeker" is included in their 2008 compilation release, The Weather Channel Presents: Smooth Jazz II. Nicole Kidman sang his song "One Day I'll Fly Away" in the Baz Luhrmann film Moulin Rouge!.