AF Derby - We suck at jumping

Ok, the good points first:
Our dressage test was our best yet. Willig got a 34.8, with a few 8s and only 2 5s, but best of all, I understood the judge's comments and agreed with them completely and remembered each of the errors. So I think we have a lot of potential to improve even more and be consistently 8s in dressage.
Even after three days there, and all the activity, Willig was, on our stall "block" the calmest horse there. He munched his hay, licked his ball, and was like "Oh? We're going out? Ok, then." Tom commented on his sweet, calm disposition.
It was all a ruse, but I'll get there.
I decided to warm up a bit early for jumping - not so much for a lot of riding (Willig was pretty tired after the second day of the clinic and his back was a little sore), but to let him look around and see the sights and not be nervous about all the activity. So I mixed in riding in the arena with walking around the hopeful jumpers and down to the xc course start and back up and around. He was fine until a loose horse went tearing past, then it took me quite a while to get him relaxed again.
Things were a bit late, which messed up my system, and the beginner novice warmup got really crowded, so we didn't end up doing as many warm up fences as I wanted, but he had done really well over the hopeful fences a few times - really nice jumping, smooth, calm, and obedient.
So we went down to xc, and one rider went before me, and I had been visualizing a perfect course and breathing and being calm and thinking about how well we were going to ride it, just like Jonathan had been working with us the two days before - nice forward, assertive seat, looking from point to point, nice straight approaches to jumps, and circling if I needed to collect him between the jumps.
We lurched over jump one.
He refused jump two.
We circled before jump three because he was being a spaz.
He stopped for the water and wouldn't trot through it.
He refused jump five (water counted as four) - apparently because two people were standing one jump over watching.
He also refused jump six.
He lurched onto the bank (jump seven).
And then I pulled him off the course, after we hurky jerky went down the gentle hill.
Because jumps 1-7 were the jumps I wasn't nervous about at all. They were the easy ones - tiny - and all jumps we had done several times, extremely successfully, during the clinic the last two days.
Jumps 8-14, however, were going to be a challenge, and I had been visualizing how excellent we would do after building confidence on jumps 1-7.
Now, in my entire riding life spanning about 20 years now, albeit with gaps, I have NEVER, EVER pulled a horse off a course because I thought he couldn't do it and might hurt me.
And I am extremely, extremely disappointed and discouraged and frustrated, and mostly, I have no idea why. I have zero idea what went wrong or how to fix it.
Next post addresses this issue.