Yes, but let's do a breakdown of the average day in the life of a Medieval European peasant. Let's assume it's a standard 8hr day for a male serf aged 15-20 years.

Sun comes up, start the day with perhaps a half hour for breakfast, another half hour for prayer, depending on the day, then it's out to the fields for 3-4 hours work, which was dependent on the particular produce of the farm where he worked and the season. Livestock tended to, fields plowed, that sort of thing. Then an hour for evening prayer and supper, perhaps some beer with the lads at the tavern before sun down.

Another thing you're forgetting is that we measure time completely differently than they did in Medieval Europe. I'll let David Graeber, of "Bullshit Jobs" explain:

Human beings have long been acquainted with the notion of absolute, or sidereal, time by observing the heavens, where celestial events happen with exact and predictable regularity. But the skies are typically treated as the domain of perfection. Priests or monks might organize their lives around celestial time, but life on earth was typically assumed to be messier. Below the heavens, there is no absolute yardstick to apply. To give an obvious example: if there are twelve hours from dawn to dusk, there’s little point saying a place is three hours’ walk away when you don’t know the season when someone is traveling, since winter hours will be half the length of summer ones. When I lived in Madagascar, I found that rural people—who had little use for clocks—still often described distance the old-fashioned way and said that to walk to another village would take two cookings of a pot of rice. In medieval Europe, people spoke similarly of something as taking “three paternosters,” or two boilings of an egg. This sort of thing is extremely common. In places without clocks, time is measured by actions rather than action being measured by time.

What is a peasant farmer doing during the winter when nothing is growing?