The poem begins with "We," a plural personal pronoun. This instantly makes the reader question who, exactly, is contained within this group—that is, whom the speaker is addressing. The first line also establishes the past tense as the dominant tense of the poem, suggesting that what is to follow is a memory (shared by whomever this "we" consists of).
Though the language is simple almost the point of banality in the first line, the mention of "that" particular day draws the reader in. That is, though the setting seems unremarkable, something about that specific day has remained significant in the speaker's psyche. The simplicity of the language is also part of the "neutral tones" of the title, suggesting a psychological conflict between the emotional moment that transpires and the apparent indifference of the natural world.
The subsequent lines further reflect the idea of "neutral tones" by suggesting a distinct lack of color. The sun is not warm and yellow, but white, suggesting that it (and by implication, the love affair) has had its life drained away. The whiteness of the sun gives the speaker the sense that the sun has been told off by God; it's as if the sun is not allowed to shine because of some wrongdoing.
Two interesting points arise here. Firstly, Hardy lived during an age of increasing religious pessimism and did himself make the move from Evangelicalism to atheism over the course of his life. With a sense of meaninglessness running through the poem that possibly pertains to this loss of faith, the presence of God here might be a signal of some kind of universal malign intent—a sense of love being doomed to failure. The second point of note is the question of why the sun could incur the wrath of God in the first place. If the sun is usually taken as a symbol of life, warmth, and moral goodness, perhaps it is guilty of providing false hope for love in a world that cannot sustain it. The impotence of the sun is also perhaps suggestive of the "fire" of love going out between the speaker and the person whom the speaker is addressing.
In lines 3 and 4, the deadness of the natural environment hints at the death of the relationship that following stanzas will recount. It's worth noting that the ash tree is particularly useful for firewood, again hinting at a metaphorical "fire" that has been snuffed out. The pun between "ash" and "gray"—as in, the ash left after a fire—supports this subtly.
There is also something paradoxical about the natural imagery used in the poem. On the one hand, it does reflect the deadness of the relationship as mentioned above. On the other, it is also anti-symbolic—nature is just being nature and doesn't have any response to the lovers' scenario. This indifference mirrors the growing indifference between the two people in the poem.