In rural North Carolina, the unnamed narrator (who is referred to as “Brother”) describes the season in which the scarlet ibis landed in the tree in his family’s front yard, when summer was finished but autumn had not yet begun. He remarks that he’s surprised the memory is so clear to him, as is his memory of his brother, Doodle.
Brother flashes back to when Doodle was born. Brother is six at the time and is immediately disappointed by Doodle. Doodle is born with a large head and tiny body, and his doctor doesn’t expect him to live more than a few days, though Brother’s Aunt Nicey believes that he will. Doodle’s parents have a small coffin built for him, but he survives infancy and they decide to name him William Armstrong.
Brother confesses how he had wanted a brother with whom he could run and play, but his parents tell him that Doodle would never be able to do those things. When he is two years old, Doodle starts to crawl, at which point Brother decides to call him Doodle, because he crawls backwards like a doodlebug.
Doodle’s father builds him a go-cart so that Brother can take him out to play. Even though Doodle has many restrictions on what he can do, Brother essentially ignores them. Brother takes Doodle to Old Woman Swamp, where they enjoy each other’s company, but Brother is also sometimes mean to Doodle. Brother brings Doodle up to the barn loft to show him the coffin his parents had made for him, and won’t let him leave until he touches it. Doodle does so, screams in terror, and as Brother carries him down the ladder Doodle begs his brother not to leave him.
When Doodle is five years old, Brother decides to teach Doodle how to walk because he is ashamed of having a brother of that age who cannot. Although Doodle initially doesn’t understand why he needs to learn, Brother attempts to teach him every day that summer. Doodle repeatedly falls to the ground, unable to stand, but after much perseverance, Doodle learns to walk. They decide to show their parents and Aunt Nicey, who are overjoyed.
Brother, now believing that he can teach Doodle to do anything, sets out to begin a development program for Doodle, teaching him to run, swim, climb trees, and fight. They work through the spring and summer, and Doodle makes some progress, but Brother worries that he still will not be able to keep up with the other boys in school. After a particularly strenuous day, Doodle collapses and begins to cry.
A few days before school begins, the family notices a scarlet ibis in a tree in their yard. Their bird book reveals that the ibis is not native to the area and must have been carried there by a storm. Suddenly, the ibis tries to fly, but its wings are uncoordinated and it crashes to the ground, dying. Doodle is very moved by the death of the ibis and solemnly buries it.
After burying the ibis, the two boys go outside to practice swimming, but Doodle is too tried to swim, so Brother makes him practice rowing instead. Soon, a storm seems to be approaching, and Doodle is too tired to carry on so the boys start to return to their house. It begins to rain heavily, at which point Brother, frustrated with Doodle’s failure, starts to run as fast as he can away from Doodle, who cannot keep up. After a while Brother stops and waits for Doodle, but Doodle does not appear. Brother turns back, only to find Doodle limp on the ground and bleeding from the mouth. The story ends with the image of Brother shielding Doodle’s dead body from the rain like his own “fallen scarlet ibis.”