Plum trees are fast-growing and extremely adaptable trees that flourish without requiring human intervention. The symbol of the plum trees here thus underscore the theme that nature is resilient and does not depend on the continued existence of mankind in order to survive and thrive.
Recall that the rhyming couplets and alliteration give the poem a symmetrical and cyclical sound, reminding the reader of the cyclical quality of nature. Here, the “circling” of the swallows underscores this idea of cycles. As swallows are associated specifically with spring, and since spring is associated with renewal, the swallows are symbolic of nature’s approaching rebirth.