While in London in the early 1980s, the unnamed narrator recounts a series of stories and memories to his cousin Ila and his uncle Robi. The stories and memories belong to the narrator; his uncle Tridib; and his grandmother, Tha'mma. The memories begin in the early twentieth century when Tridib's grandfather, Mr. Justice Chandrashekhar Datta-Chaudhuri, befriends Lionel Tresawsen at séances in London.
Tha'mma was born in 1902 in Dhaka, British India. As a young girl, Tha’mma’s father and her uncle, Jethamoshai, begin feuding, so they split their huge communal house in half with a wall. The two sides of the family stop speaking to each other, and Tha'mma tells her younger sister, Mayadebi, that Jethamoshai's family lives in "the upside-down house," where they do everything upside down and backwards. After Tha'mma and Mayadebi marry (Mayadebi marries the Shaheb, Justice Datta-Chaudhuri's son), they lose contact with Jethamoshai. Tha'mma follows her husband as he works on the railroad until he dies in 1936. At this point, her son, the narrator's father, is still a child. Tha’mma becomes a teacher and refuses to accept help of any sort from her family. Though Tha'mma had been very interested in the terrorist movements against British rule in her youth, when the Partition happens in 1947, it means little to her. However, she never returns to Dhaka since it becomes the capital of the Muslim country East Pakistan.
The Shaheb is a wealthy diplomat, and in 1939, he ends up needing a special medical operation that can't be performed in India. Mrs. Price, Lionel Tresawson's daughter, invites the Shaheb and his family to live with her in London so that he can receive medical attention there. Tridib, who is nine years old, accompanies his father, while his older brother, Jatin, stays in school in India. Tridib loves London and is fascinated by Alan Tresawsen, Mrs. Price's brother, and his friends Dan, Mike, and Francesca. In the time leading up to World War II and the early days of the Blitz, Tridib spends his days exploring bombsites and listening to Snipe, Mrs. Price's husband, tell stories. In 1940, a bomb hits Alan's house on Brick Lane, killing him and Dan. Later that year, Tridib's family returns to India.
Over the next decade, Mayadebi and the Shaheb have a third son, Robi. The narrator's father marries the narrator's mother, who soon gives birth to a son, the narrator. Jatin marries a woman affectionately known as Queen Victoria, and the couple has a daughter named Ila, who is the narrator's age. Mrs. Price, whose daughter May was an infant when Tridib was in London, has a son named Nick. Ila's parents are wealthy, and she spends her childhood traveling around the world for her father's work. The narrator, on the other hand, never gets far outside of Calcutta. Instead, he spends his time listening to Tridib tell stories about London and other faraway lands. Tridib teaches the narrator to use his imagination and explains that the world in one's imagination can be just as real as the outside world. Ila doesn't understand this—she sees too much of the world to understand how one's imagination can be anywhere as good.
For a time, Ila's family lives with the Prices in London. When she's eight, her family visits Calcutta for a festival. The narrator convinces Tha'mma to allow his family to accompany Ila's to their family home in Raibajar. When they meet Ila's family in Gole Park, the narrator's mother is shocked that the narrator, who spent weeks asking after Ila, is too shy to talk to her. The narrator feels as though his mother betrayed him by making it clear that he needs Ila more than Ila will ever need him. Regardless, the family piles into the Shaheb's two cars and drive for hours. When they reach the massive house, Ila leads the narrator into a half-underground storage room, which stores a massive table that Tridib's grandfather shipped back from London. Ila decides that they're going to play a game called Houses, which she plays with Nick in London. She informs the narrator of who Nick is, and the narrator understands that Nick is his competition for Ila's affection. Ila draws a map in the dust of Mrs. Price's house and adds a room for Magda, her doll, who is the baby for the purposes of the game. When everything is set, Ila tells the narrator what "happened" to Magda at school that day: the ugly school bully chased the beautiful blonde Magda home, yelling slurs at her—but Nick Price saved her from being beaten up. When Ila starts to cry, the narrator is angry and doesn't understand why she's crying. Finally, Tridib walks in with the children and listens to the narrator tell Ila's story. He encourages the narrator to not call Ila dumb for crying like the story is real, and he insists that everyone lives in stories.
In 1959, Tridib and May, who is nineteen at the time, begin writing to each other. They exchange photos after a year. In 1963, Tridib sends May a very long letter recalling an experience he had as a boy in London, when he watched two strangers have sex in a bombed cinema. He tells May that he wants to meet her like those strangers did—as strangers in a ruin. May is flustered, but she makes plans to visit Tridib in India. Around the same time, Tha'mma, who is retired and has time on her hands for the first time in her life, receives word that her uncle Jethamoshai, who is in his nineties, still lives in the family home in Dhaka. She believes that it's her duty to bring Jethamoshai home to India. Not long after this comes to light, the Shaheb receives a job posting in Dhaka, and he, Mayadebi, and Robi move there. Finally, Mayadebi invites Tha'mma to visit, and they make plans to try to save their uncle from the growing unrest in the Muslim-majority city. May makes plans to travel to Calcutta and then to Dhaka with Tha'mma. Tridib decides to accompany them to Dhaka.
The narrator joins Tridib and his father to pick May up from the train station. Over the next few days, the narrator accompanies Tridib and May as they drive around and see the sights. He shows her the table in Raibajar, and she tells him that Ila was a victim of bullying, but Nick never saved her. When they visit the Victoria Memorial, May becomes suddenly emotional. Tridib tells her that it's their ruin, which puzzles the narrator. He understands that there's a relationship between May and Tridib that he won't understand. Not long after that, on January 4, 1964, Tridib, May, and Tha'mma leave for Dhaka.
A few days later, the narrator experiences a harrowing bus ride home from school as the driver tries to protect the dozen boys from the angry mobs in the streets. Meanwhile, in Dhaka, the Shaheb warns Mayadebi and Tha'mma that trouble is brewing there, but Tha'mma insists on seeing Jethamoshai anyway. Thirteen-year-old Robi is excited to see "trouble" and goes with them to the old house in Dhaka. There, a Muslim mechanic named Saifuddin greets them and explains that a rickshaw driver named Khalil cares for Jethamoshai. When Khalil arrives, he leads his guests into the house. Jethamoshai doesn't recognize his nieces, but he tells Tridib that he's waiting for his family to return so that he can take them to court and gain full ownership of the house. The driver races to the door and says that there's trouble, and they have to leave. Khalil agrees to drive Jethamoshai in his rickshaw to Mayadebi's house. When they're in the car, they turn a corner and come face to face with a mob. It surrounds the car and breaks the windshield. When the mob descends on the rickshaw, Tha'mma tells the driver to go, but May gets out to try to save Jethamoshai. Tridib follows her, but Tridib, Jethamoshai, and Khalil are all brutally murdered by the mob. The narrator's parents tell him later that Tridib died in an accident. The following year, Tha'mma gives her beloved gold chain away to fund the war with Pakistan and appears crazy to the narrator. His mother explains that Tha'mma hasn't been the same since "they" killed Tridib.
In college, the narrator continues to both love Ila and find her frustrating, as she never understands why he is so insistent on remembering Tridib's stories or their own childhood antics. Once, during a summer holiday, she convinces the narrator and Robi to go with her to a nightclub. Robi doesn't want to go, but at the club, he forbids Ila from dancing with another man. She screams at them that she lives in London so she can be free of this kind of oppression. The narrator tells this story to Tha'mma on her deathbed, and it makes her extremely angry: she doesn't think Ila's kind of freedom is real. In her anger, Tha'mma writes a letter to the dean of the narrator's school the day before she dies, telling the dean that the narrator visits prostitutes and should therefore be expelled.
After seeing a lecture in Delhi, the narrator realizes that although he never connected the events as a child, the riot he experienced in Calcutta and the riot that killed Tridib in Dhaka was part of the same political uproar. As he studies Tridib's atlas, the narrator discovers that borders are meaningless and actually helped create the climate that brought on the riots in the first place. The narrator goes on to pursue an advanced degree in London. At one point, Ila takes Robi and the narrator to visit Mrs. Price and introduces them to Nick. The narrator shows off the power of Tridib's stories by leading his friends around London and through Mrs. Price's house based off of the mental maps Tridib created for him. Ila, Robi, and the narrator have dinner at an Indian restaurant afterwards, and Robi admits that he has a recurring nightmare about the riot in Dhaka in which he can never keep Tridib from getting out of the car. The narrator also reconnects with May, who plays oboe in an orchestra. They spend Christmas with Mrs. Price, and May suggests that Nick is lying about leaving his job in Kuwait: she believes he embezzled money. There's a blizzard that night, so Ila and the narrator stay at Mrs. Price's house in the cellar. Ila undresses in front of the narrator, not realizing his feelings for her, but she spends the night with Nick.
Back in London a few years later, Ila marries Nick. At their party, the narrator gets very drunk and May offers to take him home and put him to bed. The narrator assaults May but feels horrible about it in the morning. She takes him with her while she collects money for her "worthy causes," and on a break, she talks about her relationship with Tridib. As the narrator prepares to return home a few months later, Ila confides in him that Nick is cheating on her, though she refuses to leave him. The night before the narrator leaves, he has dinner with May. At dinner, May tells the narrator about the riots and asks if he thinks that she killed Tridib. May tells him that she used to think she did, but she knows now that Tridib sacrificed himself and knew he was going to die. She asks the narrator to stay the night and he accepts, glad to finally understand the mystery of Tridib's death.
