What does this poem mean "the apology"???

THINK me not unkind and rude

That I walk alone in grove and glen;

I go to the god of the wood

To fetch his word to men.

Tax not my sloth that I

Fold my arms beside the brook;

Each cloud that floated in the sky

Writes a letter in my book.

Chide me not, laborious band,

For the idle flowers I brought;

Every aster in my hand

Goes home loaded with a thought.

There was never mystery

But't is figured in the flowers;

Was never secret history

But birds tell it in the bowers.

One harvest from thy field

Homeward brought the oxen strong;

A second crop thine acres yield,

Which I gather in a song.
What does this poem mean "the apology"???
to me , it sounds like the person is apologizing for wandering off contemplating deep thoughts, while others are toiling away at physical labor,,,,,, he/she is saying that each thing they see/hear/touch, is bringing revelations to them,,,, it seems like the person has been doing this for a long time,,,,, threw two harvests