Another Man’s Song, Their Crown: The Battles Behind The Isley Brothers’ ‘Love the One You’re With’
In May 1971, the Isley Brothers did something that looked, from the outside, like surrender. The most self-reliant group in soul music — a family that had survived a decade of label-hopping, written its own biggest hit, and built its own record company to release it — put out a single written by somebody else. Not just anybody: Stephen Stills, the golden boy of the Laurel Canyon rock aristocracy, whose “Love the One You’re With” had peaked at number fourteen on the pop chart only weeks before. The Isleys’ version was the lead single from an album containing not one original composition. For a group whose entire identity was ownership, it was a startling move — and it was a declaration of war.
Because the Isleys did not cover “Love the One You’re With” the way supplicants cover a hit. They took it to church, slowed its grin into a testimony, wrapped it in handclaps and harmonies honed in Cincinnati sanctuaries, and sent it back up the charts — number three R&B, number eighteen pop, higher in soul’s home territory than Stills himself had reached anywhere. The battles behind the record are the battles of Givin’ It Back, the 1971 album it fronted: a fight for artistic legitimacy, for control of a family business still distributed by somebody else’s trucks, and for the right of a Black vocal group to claim the rock repertoire as its own birthright — which, the Isleys had better reason than anyone to argue, it was.
A Party Line, Borrowed Politely
The song began as somebody else’s sentence. Billy Preston — keyboardist, gospel prodigy, the man the Beatles had brought in to keep the peace during Let It Be — had a catchphrase he liked to dispense as romantic counsel: “If you can’t be with the one you love, love the one you’re with.” Stephen Stills heard him say it at a party, recognized a chorus when he heard one, and did the honorable thing. As Stills explained years later: “This song has been very good to me. The title came from a party with Billy Preston. Then I asked him if I could pinch this line he had, and he said, ‘Sure.’ So I took the phrase and wrote a song around it.”
Musicologists have noted an older ancestor for the sentiment: the 1947 musical Finian’s Rainbow contains the line “When I can’t be with the girl I love, I love the girl I’m with,” and Preston, raised onstage and in church, may well have absorbed it along the way. Either way, the phrase had already lived at least two lives — Broadway and Preston’s party patter — before Stills set it to music. It would live several more. Songs built on borrowed wisdom invite borrowing; that is practically their design. What no one foresaw was that the most consequential borrower, after Stills himself, would be a family of brothers from Cincinnati who treated other people’s material the way jewelers treat other people’s stones.
Stills wrote the song out, verses and all — a rose in a fisted glove, an eagle flying with a dove — and built the track on a rolling, Latin-tinged groove with steel drums he taught himself to play for the occasion. “My favorite part is the steel drums,” he admitted. “I played them before a little bit, but I just kept diddling around till I found the right notes.” The result was the defining single of his solo career. The Isleys were listening.
The Bar Stills Set
Released in November 1970 as the lead single from his debut solo album, Stephen Stills, “Love the One You’re With” entered the Billboard Hot 100 that December and climbed through the winter to number fourteen, holding that peak for two weeks in early 1971. The album went to number three and gold. It was a landmark production — recorded largely in London with a guest list that reads like a hall of fame induction class, and notable in rock history as the only album on which both Eric Clapton and Jimi Hendrix played. Stills’ own supergroup, Crosby, Stills, Nash & Young, was fracturing into solo careers; his record was the splinter that flew furthest.
This is the bar the Isley Brothers chose to jump over. Not a fading standard, not an album cut ripe for discovery — the current signature hit of one of rock’s biggest stars, still warm from the charts, in the spring of the very year it had peaked. Covering it invited direct comparison in a way that covering Dylan or James Taylor — whose songs they also took on — did not. Stills’ version was folk-rock perfection: loose, sunny, communal, a campfire singalong for the free-love era, its chorus engineered for instant join-in.
The timing sharpened the insult, or the compliment — depending on which camp you asked. The four members of CSNY had scattered into a famous race of solo albums across 1970 and 1971, and of all that splintered supergroup’s solo singles, “Love the One You’re With” was the biggest. For the Isleys to cut it in the spring of 1971 was to walk into the middle of rock’s most publicized family argument and out-sing it, in public, with a family of their own.
The Isleys’ wager was that the song had a second nature its author had only half-revealed. Underneath the campfire cheer sat a piece of folk philosophy with deep roots in Black American saying — Preston’s line was, after all, the source — and in gospel consolation: make a way out of no way, bless the portion in front of you. The brothers had been singing that theology since childhood. They did not need to learn the song. They needed to repatriate it.
The Covered Become the Coverers
The album’s title was the thesis. Givin’ It Back — after years of watching other acts cover Isley Brothers material, the family announced, it was returning the favor. The group’s songs had been a public resource since the beginning of their career; the Beatles’ version of “Twist and Shout” alone had made the brothers’ composition-era repertoire a foundation stone of the British Invasion, and singers of every description had built hits and album tracks out of the Isley catalogue for a decade. The brothers had profited unevenly from that traffic. Now, in 1971, they would run the trade route in reverse.
Wikipedia’s summary of the album puts it plainly: after years of having other acts cover their most famed material, the Isleys decided to do the same to music made famous by artists such as Stephen Stills, Eric Burdon and Neil Young. But the framing as payback undersells the ambition. Givin’ It Back was the first Isley Brothers album built entirely from outside material, and its selections were not random: Dylan, Hendrix, Neil Young, James Taylor, Bill Withers, Eric Burdon and War, and Stills twice over. This was the contemporary rock canon, the 1970-71 syllabus of serious popular music — and a Black family group from the R&B circuit was claiming the whole syllabus as its rightful classroom.
The battle lines were industrial as much as artistic. American radio and retail in 1971 still sorted music by the race of its makers with bureaucratic thoroughness. A rock album by a soul group was a category error the business barely had bins for. The Isleys’ answer was to make the category error at the highest possible level of craft, on their own label, with their name over the door.
The Old Cover Game, Played in Reverse
There was also an older game being replayed. In the 1950s, the American record business had run on cover battles: a Black R&B act would cut a hit, and within weeks a white pop act — Pat Boone softening Little Richard and Fats Domino, the Crew-Cuts sanding down the Chords, Georgia Gibbs answering Etta James and LaVern Baker — would rush out a whitened version engineered to beat the original on pop radio, which it usually did. The originals’ makers watched their songs, and a large share of the money, change trains at the color line. It was the industry’s original sin, and everyone in the Isley Brothers’ generation had lived through it as young professionals. The brothers themselves had felt the draft of it: their own songs kept becoming other people’s pop property.
By 1971 the cover-battle economy was supposed to be dead — killed by the Beatles’ generation, which had the grace to point listeners back to the originals. What the Isleys did on Givin’ It Back was resurrect the form with the polarities reversed. A Black group, on its own label, was now the one moving fast on a white artist’s current hit — not to soften it for a sheltered market, but to deepen it for a knowing one, and not to steal the pop chart from the original so much as to annex the R&B chart for it and dare the pop chart to argue. Where the fifties covers had subtracted soul for safety, this one added soul as argument.
The reversal had a wit to it that the brothers surely savored. “Love the One You’re With” was itself built on a line borrowed from Billy Preston — a Black sideman’s wisdom that a white star had, with permission and credit, carried to the bank. The Isleys were not even the first borrowers in the chain. They were the ones who brought the borrowed thing home, stamped it in church harmony, and posted it back up the charts. If the fifties game was theft wearing a smile, this was restitution wearing one.
Their Own Label, Borrowed Trucks
To understand why the Isleys needed Givin’ It Back, you have to understand what T-Neck Records was — and what it wasn’t. The brothers founded the label in 1969, naming it for Teaneck, New Jersey, the town where they were then based, and launched it with the self-written, self-produced “It’s Your Thing,” a number-two pop and number-one R&B smash that won a Grammy and humiliated Motown, which had let the group go and then watched them flourish independently. T-Neck was proof of concept: the family could write, produce and own.
But owning the label is not the same as owning the pipeline. T-Neck’s records reached the public through a distribution deal with Buddah Records, the New York hit factory run by Neil Bogart. Buddah pressed, shipped and promoted; the Isleys created. It was a workable arrangement and, through the early seventies, a productive one — a run of albums including The Brothers: Isley, Get Into Something and Brother, Brother, Brother. Yet every independent label owner of the era learned the same arithmetic: the distributor’s priorities are the distributor’s catalog, and the further a record travels from the distributor’s idea of what you are, the harder it is to move. The Isleys’ idea of what they were, in 1971, was a rock band. Buddah’s bread and butter was R&B singles.
Self-production was the family’s counterweight. Givin’ It Back was produced by Ronald and Rudolph Isley themselves — no outside producer interposing a “commercial” judgment between the brothers and the experiment. If the album failed, it would fail as a pure expression of the family will. That autonomy was the point of T-Neck, and Givin’ It Back was the fullest use they had yet made of it: an album no major-label A&R department of 1971 would have approved, made precisely because no A&R department had to.
The arrangement still had four years to run — the Epic distribution deal that would power the 3+3 era began in 1973, and it is no accident that the group’s full transformation waited for a distributor built for albums. But the artistic declaration came first, on Buddah’s trucks, in 1971: we are an album band, we are a rock band, and here is the proof, pressed on our own imprint.
Nor was the experiment launched from a position of commercial comfort. The group’s pop momentum since “It’s Your Thing” had been cooling into the turn of the decade: the singles from the Get Into Something period — “Keep On Doin’,” “Freedom” — had done their work on the R&B chart, reaching the teens, while stalling in the seventies on the Hot 100. The Isleys in early 1971 were respected, solvent and stuck: too established to vanish, too categorized to grow. Read against that stall, Givin’ It Back is not a rich man’s indulgence. It is a leveraged bet by a family firm whose core market had plateaued — a decision to spend its credibility on the one move that could change its category before its category finished changing it.
Spring at Mediasound
The album was recorded in 1971 at Mediasound Studios in New York City, and its personnel list captures a group in the act of changing shape. The credits still read like a vocal trio’s album — Ronald out front, O’Kelly and Rudolph behind him — but the band underneath them is mutating. Marvin Isley, nineteen, plays bass. Chris Jasper sits at the piano. And on guitar, two names appear where a session man used to be: Chester Woodard, the group’s road guitarist, and Ernie Isley.
Ernie was nineteen years old in 1971, a decade and a half younger than O’Kelly, and Givin’ It Back is where he first performs on an Isley Brothers album — “first performing,” as the group’s histories put it, on the record that would relaunch the family as a band. He and Woodard split the lead guitar work across the album, including on “Love the One You’re With,” in a twin-guitar weave that was itself a rock-band gesture. Behind them sat George Moreland on drums and percussion, Milton Westley on organ, John Mosley on flute — and, for the closing track only, a guest percussionist whose presence was a quiet seal of authenticity: Buck Clarke, a veteran jazz conguero, played congas on “Love the One You’re With.”
Even the album’s smallest credit tells the family story. On “Cold Bologna,” the guitar is played by the song’s writer, Bill Withers — a former aircraft mechanic who had not yet released his own debut album and whose funky blues the brothers championed before the world knew his name. The Isleys were not merely covering their contemporaries. They were convening them.
Taken to Church
What the Isleys did to “Love the One You’re With” can be heard in the first thirty seconds. Stills’ original bounces in on sunshine and steel drums, a song already smiling. The Isleys’ version rises instead — organ and piano laying a church floor under the groove, the tempo eased back, the brothers’ voices entering not as a campfire chorus but as a congregation. Critics have reached for the same phrase for fifty years: Wikipedia calls it their “unique gospel-driven rendering,” and the gospel is not a metaphor. It is the arrangement’s literal source code — the call-and-response reflexes, the testifying lead, the sense that the chorus is not a hook but an affirmation, repeated until believed.
The transformation re-reads the lyric. In Stills’ hands, “if you can’t be with the one you love, love the one you’re with” is a shrug set to music — free love’s pragmatic motto, charming and a little cynical. Sung by Ronald Isley over a groove built like a sermon, the same words become consolation: counsel for the down and confused, as the opening verse addresses them, from a tradition that had centuries of practice at finding gratitude inside deprivation. The song’s images — that rose in a fisted glove, the eagle flying with the dove — stop sounding like hippie whimsy and start sounding like parables. Same text; different testament.
Technically, the record is a marvel of economy: three minutes and forty seconds, the album’s closer, and its most concise statement. The twin guitars chatter and testify, Buck Clarke’s congas keep the Latin pulse that Stills’ steel drums had sketched, and the brothers stack the final choruses into a wall of harmony that lifts the song clean out of the folk-rock category. It is not a cover so much as a conversion experience — the song enters as Stills’ and leaves as the Isleys’, permanently.
Notice, too, the personnel diplomacy in that percussion credit. Gary Jones played congas on the album’s other tracks; for the closer, the brothers brought in Buck Clarke — a jazz elder whose career reached back through the music’s earlier generations. Assignments like that are how bands declare priorities without saying a word. The Stills remake, the record the whole album would be judged by, got the veteran. When the needle lands on side two’s final groove, what you hear is a group that has staffed the song like a title fight.
The Second Stills Song
One detail of Givin’ It Back is easy to skate past and impossible to overstate: the Isleys covered Stephen Stills twice. Tucked on side two, two tracks before the closer, sits “Nothing to Do But Today,” another cut from Stills’ solo debut, delivered by the brothers as polished, easy-rolling soul. No other writer on the album gets two songs. Dylan gets one, Hendrix one, Young one, Taylor one, Withers one. Stills gets an audit.
The double booking reframes the closing cover’s intent. This was not a group rifling the charts for a sure thing; if it had been, one Stills song would have sufficed, and they’d have chosen a safer moment to take it. It was a deliberate measuring of themselves against the most celebrated harmony-singing songwriter of the new rock era — group against group, family against supergroup. Crosby, Stills, Nash & Young had built their legend on stacked harmonies; the Isleys had been stacking harmonies since before CSNY’s members left school. Covering Stills twice was a way of saying, on the record, in the most courteous possible terms: nice blend. Ours is older.
It also made “Love the One You’re With” land differently. Arriving at the end of an album that had already bent Stills’ material once, the closer plays not as a hopeful remake but as a summation — the second and decisive exhibit in a case the whole album had been building. By the time Ronald reaches the final choruses, the verdict the record wants has been thoroughly prepared: these songs are not being visited. They are being re-zoned.
The Album Around It
“Love the One You’re With” closed Givin’ It Back, but the album had opened with its boldest provocation. “Ohio” / “Machine Gun” — nine minutes and thirteen seconds — spliced Neil Young’s furious response to the Kent State shootings of May 1970 to Jimi Hendrix’s “Machine Gun,” the anti-war broadside from Band of Gypsys, and folded a prayer into the junction. For the Isleys, the Hendrix half was personal: Jimi had been their sideman in 1964-65, living with the family before fame, and he had been dead less than a year when these sessions began. Ernie and Chester Woodard’s guitar work on the medley is, among other things, a memorial service conducted at high voltage.
The rest of the program moved between protest and tenderness. “Lay Lady Lay” stretched Dylan’s country miniature past ten minutes into a slow-burning soul meditation. “Fire and Rain” took James Taylor’s confessional into the sanctuary. “Spill the Wine” kept Eric Burdon and War’s Latin funk close to the original and became a hit single in its own right, reaching number forty-nine pop and number fourteen R&B. “Nothing to Do But Today” gave Stills a second track on the album — the brothers were not borrowing a hit, they were auditing a writer. And “Cold Bologna” handed the guitar chair to a still-unknown Bill Withers, repaying an unknown’s song with a master’s band.
Seven tracks, forty-one minutes, no originals — and, as AllMusic’s Bruce Eder wrote in the review that has followed the album through every reissue, the group sounds anything but unoriginal: the reworkings “show no lack of originality,” and the closer is “sent soaring to heights that the Stephen Stills original could only gaze up at.” Reviewing the album decades later, critics would call it a time capsule. In 1971 it was something braver: an argument, made in other men’s songs, about who the Isley Brothers were.
Number Three, Number Eighteen
The single preceded the album by four months — released in May 1971, while Stills’ original was still a recent memory — and it performed like a challenge answered. The Isleys’ “Love the One You’re With” entered the Billboard Hot 100 in the issue dated June 19, 1971, at number seventy-two, and climbed through the summer to peak at number eighteen on August 14, spending eleven weeks on the chart. On the R&B chart it reached number three. On the Cash Box survey it did better still, reaching number ten. It was the group’s fifth top-forty pop single in the United States, and it came closer to Stills’ own number-fourteen peak than a cover released in the original’s immediate wake had any right to.
Read the two chart runs side by side and the battle resolves into something like a split decision — with territory divided by the industry’s own map. Stills took the pop chart by four places. The Isleys took the R&B chart outright, in a field where Stills’ record had never been a factor, and matched him nearly even on pop while playing away from home. Each version was the definitive one in its own country, and each country, in 1971, still pretended the other was foreign.
There is a footnote in the numbers that flatters the brothers most. Covers usually feed on distance — on the original being old, or foreign, or forgotten. This one fed on proximity. Every radio listener who heard the Isleys’ record in the summer of 1971 had Stills’ version in their ears from the winter. The comparison was the marketing, and the brothers won enough of it to matter.
An Album That Arrived Late to Its Own Hit
The album itself, released September 25, 1971, had a stranger fate. By the time Givin’ It Back reached the shops, its lead single had already peaked, and the LP was left to sell an argument rather than a moment. It reached number thirteen on Billboard’s R&B album chart — respectable, not spectacular — and its singles after the first (“Spill the Wine” in September, “Lay Lady Lay” in November) kept the group’s name warm rather than hot. Robert Christgau, grading the era’s output in his consumer guide, gave it a B: the sound of a tough critic conceding the craft while withholding the crown.
The reissue era has been kinder than the release season was. When the album returned in 1997 as part of Epic/Legacy’s Rhythm & Soul series, with Bruce Eder’s notes framing it as a misunderstood triumph — “as much a time capsule as an album,” in his phrase — the reevaluation began in earnest. Pitchfork, reviewing the group’s 23-disc box set The RCA Victor & T-Neck Album Masters (1959–1983) in 2015, rated Givin’ It Back at 8.3 out of 10. Albumism’s fifty-fifth anniversary tribute called it “100 percent an Isley affair” despite containing no Isley compositions, and singled out “Love the One You’re With,” with Buck Clarke’s congas, as the album’s defining moment.
The pattern is familiar from the group’s whole career: the Isleys’ records were often worth more later than the market first allowed. But Givin’ It Back is a special case, because its modest commercial showing disguised a permanent change inside the band. The album that the charts graded a B had quietly re-founded the Isley Brothers as a self-contained rock group. Within two years, that group would be unstoppable.
Proven on Stage
If any doubt remained about whose song it had become, the brothers settled it in public. When they recorded their live album The Isleys Live at New York’s Bitter End in 1973, “Love the One You’re With” made the set — alongside the “Ohio” / “Machine Gun” medley, which the contemporary accounts describe expanding into one of the great live statements of the era — and the performance has been singled out by listeners ever since. Ronald, introducing the song, plays the whole philosophy for a laugh before he sings it: as one widely cited account of the live rendition has it, he teases the audience — “Look, girls, when you can’t be with the one you love … just wait?” — puncturing the lyric’s free-love license with a preacher’s son’s timing, then lets the band prove the song could carry a room of believers anyway.
The live version matters because it shows the song fully naturalized. On stage there is no Stephen Stills in the comparison at all; there is only the Isley Brothers’ congregation and a chorus they have made into a communal rite. PopMatters, reviewing the group’s box set four decades later, singled out The Isleys Live as nearly perfect and its inclusion of “Love the One You’re With” as a sweetener on an already loaded record. The borrowed song had become a set-list fixture of the family liturgy — the surest sign of repossession a repertoire can offer.
The Song Refuses to Settle
Stills’ phrase, meanwhile, completed its escape from any single owner. Aretha Franklin built it into her legendary Fillmore West performances of 1971 — the same season as the Isleys’ single — with Billy Preston himself, the phrase’s originator, at the organ: the line coming home to its maker in a cathedral of soul. The Meters cut it. Bob Seger cut it. Firefall, Bucks Fizz — the song became one of those standards-in-waiting that every genre tries on, and in 1994 Luther Vandross remade it for Songs, his album of interpretations, introducing Preston’s party wisdom to the quiet-storm generation as a velvet R&B single in its own right.
Through all those lives, the Isley Brothers’ version holds a particular office: it is the proof of concept. Every subsequent singer who has treated “Love the One You’re With” as a soul song rather than a folk-rock song is working the claim the brothers staked at Mediasound in 1971 — that the song’s natural habitat was the church as much as the campfire, that its shrug contained a sermon. Vandross, who revered the Isley ballad tradition above almost any other, was not covering Stills in any simple sense. He was covering the record that had shown him what the song could hold.
And the phrase itself outgrew music entirely, passing into the language as a proverb — quoted in contexts Preston never partied in, usually by people who could not name Stephen Stills, let alone Billy Preston. Songs dream of this fate. Almost none achieve it. Fewer still achieve it twice, in two arrangements, in the same calendar year.
The Bridge to 3+3
Inside the family, Givin’ It Back did its quiet work. It blooded Ernie Isley as a recording guitarist while he was still a teenager. It installed Marvin’s bass and Jasper’s piano inside the group’s recorded sound. It demonstrated, on the record-buying public’s dime, that the brothers could hold an audience through ten-minute Dylan covers and nine-minute protest medleys — that the album, not the single, was their natural unit. And it sharpened the lesson of the Buddah years: distribution is destiny, and the family needed a partner built for the music it now intended to make.
The sequence that follows is the payoff. Brother, Brother, Brother in 1972 consolidated the new band. In 1973 the Isleys moved distribution to Epic, made Ernie, Marvin and Chris Jasper full members, and released 3 + 3 — the album where the covers strategy turned back into original songwriting with rock grammar fully absorbed, “That Lady” riding Ernie’s guitar to the top ten. The 3+3 era’s triumphs, from The Heat Is On to Go for Your Guns, all stand on the foundation poured at Mediasound. Even the circle of repertoire closed: the group that had “given back” Stills and Dylan in 1971 spent the rest of the decade as the act other artists covered.
“Love the One You’re With” sits at the hinge of that story. It was the hit that financed the argument — proof, delivered up the charts, that the experiment was not self-indulgence. Without its number three, Givin’ It Back might be remembered as a noble detour. With it, the album stands as what it was: the sound of a family changing its own mind about what it was allowed to be, in public, at full volume.
The One Battle Nobody Won
Honesty requires a footnote about the money, because for all the battles the Isleys won with this record, the oldest one stayed lost. The publishing on “Love the One You’re With” belongs to its writer: every copy of the Isleys’ hit, every later cover their version inspired, sent its songwriting royalties to Stephen Stills. And Billy Preston — whose party remark had supplied the title, the hook and the philosophy — held no writer’s credit at all, then or ever; a spoken line, however immortal, is not a copyright, and his reward was the story itself. The brothers, who had built T-Neck precisely to escape the economics of making other men rich with their labor, spent the summer of 1971 doing a refined version of exactly that.
And yet the ledger is not the whole account, because the Isleys were never only buying chart positions with Stills’ song. They were buying a category change — proof that a family R&B act could stand at the center of the rock conversation — and category changes compound for decades, long after a single’s royalties have been spent. 3 + 3, The Heat Is On, Go for Your Guns: the fortune the Isleys made in the seventies was made with their own copyrights, in a market position this borrowed song had helped to clear. Preston’s line proved prophetic in a sense he never intended. Unable to be with the audience they loved — the album-rock audience on the far side of the industry’s wall — the brothers loved the one they were with, and used its song to go and get the other one.
Why It Endures
Some covers are tributes. Some are thefts. The Isley Brothers’ “Love the One You’re With” is a repossession. The song’s wisdom had been overheard at a Black musician’s party, polished into a hit by a rock star, and returned — within months, at chart altitude — to the gospel and soul tradition that had supplied its philosophy in the first place. The brothers hid nothing about the transaction. They put it in the album title: Givin’ It Back.
What endures, finally, is the singing. Ernie and Woodard’s guitars have been imitated, Buck Clarke’s congas absorbed into the record’s DNA, but the thing no later version has replicated is three brothers from Cincinnati leaning into a borrowed chorus as if it were a hymn they had known since childhood — because, in every way that mattered, they had. Stephen Stills wrote a fine song about settling for the love in front of you. The Isley Brothers, who had spent a career refusing to settle — for other people’s labels, other people’s categories, other people’s definitions of what a soul group could play — sang it like an anthem of gratitude anyway. That tension, between the lyric’s contentment and the singers’ ambition, is the battle inside the record. It is why, half a century on, the song still sounds less like advice than like a verdict: the crown fit the borrowers better than the man who lent the line — and everyone, starting with Stills himself, has had to live with the one they’re with.