Solo Lesson ... in Patience

Ask anyone who knows me, and I will probably top their list of least patient people. I will certainly be in the top 5. And yet, most of the things I do seem to require all I've got in patience (quite possibly because it takes so little to use up my patience stores).
So tonight was another "opportunity" (aka challenge, aka patience exhausting) with Willig. With the light changing, and a couple days without rain, I've been able to ride outside in the evenings after work. Today I took Willig out to lunge him (he was fine), then ride him (he was fine as we started), but as the horses went in for dinner, he got increasingly agitated. So I lunged him again (less fine, but got tired) and then rode him at the walk for what felt like an hour but was probably only half an hour, talking to him and trying to help him realize the other horses can go places, and he's still ok.
It's great for us to get the opportunity to work on this before the first show, but it's frustrating that he has so many things he needs to work on. It feels like we don't ever get to work on actual riding, just on him being calm and listening. There's always something making him agitated.
I don't know if it's because a) he's young (still 6), b) had a sheltered life, c) his personality, or d) a little bit of all of the above. After living in a stall his whole life, turn out is a pretty big deal, and I think he's doing really well. It's just hard with Mr. Sedate Mercury who was the last horse in, presumably because he's such a sweet angel he doesn't cause problems.