"lead but to the grave..."See in text(Text of the Poem)
Gray makes power, beauty, and wealth meaningless as all paths in life lead to death. If anything, these symbols of wealth and power are even more ridiculous because both poor and rich alike are destined to die. Here, Gray uses death to erase the distinction between the rich and the poor that he established through imagery at the beginning of the poem.
This line marks a turning point within the poem at which Gray stops describing the farmer and begins addressing an imagined audience. Gray presupposes the audience's opinion of the characters he is describing and contradicts it.