Jude and Sue’s private life becomes more of a subject for gossip, and soon everyone in town knows that they aren’t really married. The couple pretends to go off to London and get married, but people still want the scandal to continue so they disbelieve it. An “oppressive atmosphere” develops around the couple and they feel unwelcome and condemned.
The couple’s “dreamy paradise” finally comes up against the harsh reality of a cruel, judgmental society, and the delicate and sensitive Jude and Sue cannot stand up against their neighbors’ condemnation.
One day Jude is hired to reletter the Ten Commandments at a nearby church, and Sue comes along to help him. While they are working Little Father Time comes in, crying that other children mocked Sue in front of him. Jude and Sue then overhear some church members discussing their marital status and telling condemning stories. Sue finally breaks down, as she cannot understand why people hate them for choosing to “live their own way.” Soon afterward the contractor fires the couple, wishing to avoid a scandal.
Hardy now delves into more specific, heart-rending tragedies – he has shown how innocent and justified Jude and Sue are in living the way they do, and now he will illustrate how society unjustly tramples over them for daring to live in an unorthodox way. The cruelty of others is especially hurtful to Little Father Time, who is a blank canvas being filled with unfairness.
Later Jude is nudged out of a workers’ union, and the couple decides to move away. They sell all their furniture at auction, and Jude and Sue remain upstairs with Little Father Time. They overhear all the townspeople discussing their personal lives. Jude and Sue decide where to move next, recognizing that Jude will now be unable to find church-related work.
Though sexuality is never mentioned in “respectable” society, it is everywhere under the surface. Judgment runs so deep that seemingly secular activities like a workers’ union are affected by someone’s harmless personal decisions.
Jude and Sue leave the house just as the auctioneer is selling two pigeons Sue kept as pets. A poulterer buys them to use for pies, which greatly upsets Sue. Later that evening Sue passes the poulterer’s shop and sets the pigeons free.
Sue is again associated with freedom and birds, as she takes this small action against the injustice of her surroundings. She also shows her affinity with Jude in her sympathy for animals.
Afterwards Sue feels guilty and confesses to Jude, and she laments aloud that the law of Nature is “mutual butchery.” Little Father Time asks if this is true, and Sue affirms it. Jude lists all the towns where they cannot go, as they are known and condemned there, though they have “wronged no man” but only “done that which was right in our own eyes.”
Sue repeats Jude’s childhood revelation (while doing his job scaring crows) as the couple grows more depressed and pessimistic. This last statement is a sort of thesis for the novel – people should not be punished or condemned for living in an unorthodox manner, as long as they cause no harm to anyone.