Rufus Scott is a Black jazz musician living in Harlem who spends his days wandering the streets of New York. He has made a mess of his life following a catastrophic relationship with a white, Southern woman named Leona. Rufus and Leona met at a jazz show and quickly fell in love. However, their relationship quickly soured after Rufus sexually, physically, and emotionally abused Leona. In particular, Rufus became mad whenever he did not feel Leona properly understood Black suffering in the United States. Eventually, Rufus went too far with his abuse, and it drove Leona insane. The police found her wandering the streets and had her shipped to an asylum in the South, where she lives out the rest of her life. Rufus now fears that his caustic relationship with Leona ruined his friendships with Vivaldo and Cass, the only people outside of his family who care for him. Not knowing what else to do, he has been roaming the streets and performing sex work for money.
On a particularly grim day, Rufus decides he cannot stand his life anymore and, out of desperation, he goes to see Vivaldo. Vivaldo tries to comfort him as best he can and buys him some food. Later that day, Rufus and Vivaldo go out to a bar together where they see Cass and her husband, Richard. Cass tries to comfort Rufus, while Vivaldo meets up with an old girlfriend, Jane. After Cass leaves, Rufus slips out of the bar, feeling hopeless about life. He makes his way to the George Washington Bridge and jumps off it, killing himself.
Vivaldo and Cass attend Rufus’s funeral; they are the only white people in attendance. Following the funeral, Vivaldo grows close to Rufus’s sister, Ida, and the two of them start a romantic relationship. One day, they attend a party at Richard and Cass’s house to celebrate the release of Richard’s new novel. There, Ida meets Steve Ellis, a television producer who promises he can make her a star. Vivaldo does not like Ellis and is jealous of the attention he pays to Ida. His jealousy leads to a fight between himself and Ida—the first fight of their relationship.
Not long after the party, Cass receives word that Eric, a friend who has been living abroad in France, plans to return to New York to star in a Broadway show. Eric is Rufus’s former lover who left New York after enduring abuse from Rufus. Eric has been living with his new boyfriend, Yves, who he plans to bring to New York after he gets settled. When Eric arrives in New York, he grows close to Cass and Vivaldo. Eventually, he starts an affair with Cass, who has grown apart from Richard following the success of his novel. Eric and Cass display their affair quite openly, confident that Richard will not discover it. Their brazenness shocks Ida and Vivaldo, though they keep it a secret from Richard.
Additionally, Ida and Vivaldo have troubles of their own. They fight regularly, often over issues of race and jealousy. Ida tends to blame Vivaldo for the sins of all white people and claims he does not understand the struggle of Black people in America. Meanwhile, Vivaldo is jealous of Ida and her new relationship with Ellis. Vivaldo does not respect Ellis and hates that Ida is going to see him on a regular basis in the hopes of becoming a star.
One night, Cass and Ida go to a bar with Ellis while, separately, Eric and Vivaldo spend time together. Ida dances with Ellis at the bar, leading to raised eyebrows from everyone around them except Cass, who believes their relationship is innocent. Cass leaves the bar and goes home where she finds Richard waiting for her. Richard confronts her about her affair, though he mistakenly assumes she is having sex with Vivaldo, not Eric. Cass refuses to give in at first and says she has been spending her time with Ida. Richard calls her bluff, saying he knows she is lying because he knows Ida is having an affair with Ellis. At this point, Cass gives up and tells him she is sleeping with Eric, sending Richard into a rage.
Meanwhile, Vivaldo and Eric get drunk together and go back to Eric’s apartment. After some conversation about the state of their lives and relationships, they crawl into bed together. In the morning, Vivaldo and Eric have sex together, making them both feel ecstatic. However, Eric quickly gets pulled away when he receives a phone call from Cass explaining that Richard has discovered the affair. He agrees to go and meet her so they can do damage control.
Vivaldo returns home, where he finds Ida. Together, the two of them have a conversation about their relationship, which ends with Ida revealing that she is having an affair with Ellis. She says she had hoped to take advantage of Ellis, but, in the end, he took advantage of her. Vivaldo and Ida end their relationship amicably and, for the first time in a long time, Vivaldo makes progress on the novel he is working on.
Ultimately, Cass decides to take some time apart from Richard, though she thinks they may be able to patch their relationship up eventually. Meanwhile, Eric decides that, although his affairs have been fun, Yves is the person he wants to be with. Not long after, Yves flies to New York. Eric meets him at the airport, eager to start a new chapter in his life.
