Caesura gives readers a sense of the speaker's troubled mind. It does this either by fragmenting sentences, or by balancing different elements of a sentence to give what's being said the sound of cold, hard logic. For example, take a look at the full stops in lines 12 and 17:
To think that I do not have her. To feel that I have lost her.
[...]
This is all. In the distance someone is singing. In the distance.
Full stops like these slow the poem down to a near halt, dramatizing the speaker's inability to fully comprehend the break-up. It shows him dwelling on things; the short, declarative statements make it seem like the speaker is trying to reason out what happened, to convince himself, bit by bit, that his relationship is really and truly over.
The pauses in lines 25 and 26 work similarly:
Another’s. She will be another’s. As she was before my kisses.
Her voice, her bright body. Her infinite eyes.
The pauses created by all these caesurae (which combines in line 26 with asyndeton) again suggest that the speaker is trying very hard to slowly, steadily reason out what happened, to remind himself of his current, lonely reality, and even to contain his emotional response to that reality. Those pauses prevent the poem from gaining much momentum, in turn suggesting that the speaker is trying to keep it together; he could write "the saddest lines," but he also doesn't want to let his feelings spill out of him uncontrollably.