Disappointing, but shouldn't be, ride at Lincoln Creek

Yesterday we went to the Pony Club one day at Lincoln Creek. Willig was being naughty in the cross ties before we left, and I fortunately had a little pad of extra time for lunging - although he didn't act up too much on the lunge line, he did have a few head tosses to get out.
I felt like our dressage test was lackluster, although we did ok - 33.3, with about a point to 2nd place and another point to 1st place.
Tom watched him next to the trailer while I walked the course, which I felt pretty good about. The hopeful jumps were small and not very intimidating (I knew he'd be scared going over the dike, but since there was no way off, he'd have no choice but to go forward). The beginner novice fences, on the other hand, were huge and intimidating.
I got on him a bit early for cross-country, hoping to go early (only by 4 minutes, as it turns out), and watched the hopeful juniors slaughter their position over the warm up fences, then refuse several times, and more than a handful of falls.
Yet, I was still totally calm and confident - the thing upsetting him most was the flies (Lincoln Creek always seems to have particularly persistent ones), and he was kicking at his belly. He spooked once when someone came out of a port-a-potty, but nothing major. We did the warm up fences a couple of times each direction - he wasn't overjumping, was paying attention - everything was smooth.
Approaching each fence, he did a bit of the "snake" move, but I rode him confident and forward, using more leg instead of brakes, and he jumped the first three just fine. But when he went to the fourth - a trot over a ditch (solid ground - we didn't attempt the down bank since he was already being snaky) past a flat bed truck with lots of people standing on it - he started popping his head up looking for things to be scared of. Frustratingly, he could see those people during the whole warm up and couldn't have cared less about them.
Fence 5 was a little A frame that looked just like the A frames at Caber Farms two weeks ago that he rocked out at. He refused it. Ran out to the left, abruptly, right when he should have taken off, and accentuating the fact that I was jumping ahead. So we circled around, came back at it - this time me kicking HARD - and he refused it again. And by that point I was pissed, so we came at it a third time, me kicking him HARD!! and he jumped it, but with his head boinged up at the guy standing on the small hill recording our refusals - where we had to turn right onto the dike and also jump a very small log. Which we awkardly stepped over - continued our hesitant, leapy, jerky trot along the dike (by this time he was working himself into full blown panic and my normal solution at home to that is to stand around and let him cooooooollll down), went down the small hill, made a left turn to blue barrels (which we have at home), which he ... refused. Went back over them - me beating his neck with the crop and kicking furiously - another trot over a ditch which he could hardly stand by this point - to an A frame with brush on it, which he refused by halting in front of.
By this point, I was sure Shannon would kick me out of the barn, so I beat him over it from a standstill, and to add insult to injury, he was perfectly capable of jumping it from a halt.
Then we roared over to a big log, another damn ditch, and a modestly challenging A frame to complete the course.
Although a few people were eliminated in our division - presumably from falling since our refusal points were still more than a fall with no refusals (although how that works - do you get 20 points for the refusal and 65 for the fall?) - we still came in 5th place, after being 3rd in dressage.
Show jumping was equally calm over the warm up fences - no over jumping - and then the mad charging at the fences and trying to spook at everything that isn't a fence. Although he's jumping them just gorgeous, I hate the mad charging feeling.
What was so frustrating about this was I felt like I rode him the same as at Caber, where we had a spectacular ride, only I was more calm and not anxious, thanks to that spectacular ride. After Caber, I thought next year we will definitely be ready to go Beginner Novice. After yesterday, I thought (like sad Eeyore), "No, we're appropriately at Hopeful."
Other than being "calm", I don't know how I rode differently. Because I was less nervous, I had a clearer head to think of Shannon's instructions and follow them (forward, confident, help him) to each fence. So why didn't it work? I have no idea.
I was frustrated and disappointed. But then again, at this time last year, I had given up and wasn't even trying to show anything, let alone Hopeful, and I KNOW that if I just haul him around and build up his confidence he might be quite nice (still a bit wacky, but rideable), but it's just taking so. long.