Alliteration adds to the poem's musicality, and it often reinforces its meaning as well. The poem begins with alliteration, in fact:
Well, World, you have kept faith with me,
Kept faith with me;
The repeated /w/ sound heightens the emphasis on those first two stressed syllables—"Well, World"—drawing out the speaker's dryly ironic tone. That same sound then repeats twice in "with [...] with"; in fact, the whole phrase "kept faith with me" repeats (an example of the device epizeuxis). In general, the poem's alliteration is part of its pattern of intensive repetition, which adds to its song-like quality.
For another example, listen to the swing of lines 9-10:
'Twas then you said, and since have said,
Times since have said,
The insistent /s/ alliteration (sibilance), triple repetition of "said," and so on make these lines sound a lot like song lyrics. Poems don't usually repeat themselves to this extent, but songs often do.
Sibilance appears in other lines as well. Soft, repeating /s/ sounds make the phrase "smooth serenity" (line 14) that much smoother to say aloud. By contrast, the three straight /s/ words in line 23 ("stem such strain") sound a little labored, in keeping with the description of "strain."
Why the difference? For one thing, three words sound weightier than two; for another thing, two of the alliterative words in line 23 share a double consonant sound, /st/, rather than /s/ alone. Also, the two /st/ syllables are both accented by the meter ("stem such strain"), whereas only one /s/ syllable was accented in line 14 ("smooth serenity"). All these factors make the alliteration in line 23 sound slower, heavier, and more "strain[ed]."