How do you solve a problem like Willig?

(To the tune of "How do you solve a problem like Maria?")
After our terrible experience on the xc course at the derby, I am feeling pretty low. As best I can tell, trying to be objective, Willig has gotten WORSE at jumping, not better. Although it took me months and months to jump him the first time after I bought him, last summer we went to two derbies (both intro) and a clinic with Jonathan. There were no refusals, and he wasn't refusing at home.
Then he got a few months off over the winter with the problems at the prior barn, and this year my goal was to move him up to starting to experience Beginning Novice, since the height is no problem (he's 17 hands!), and he needs to see the different types of jumps. I think we've been on a steady decline since February.
I don't really know what to do about this. I think we're a bit stuck - I'm afraid he's going to refuse, and therefore, he refuses. Then I gather all my confidence and ride him assertively to a fence, but he's learned to refuse that fence. And the refusing started before the fall, but that big fall off really, really shook my confidence in him. It doesn't help that he randomly tries to buck me off now. (He did it again today - working circles (I WANTED to be working stretchy trot circles; instead we worked rhythym instead of racing around circles), and out of nowhere, a buck. Now, it was half-assed, and I gave him a good pop in the mouth, a yell, and a smack, and he didn't try again, but still - what the hell?)

So, the problems are:
- He's an insecure horse, and I'm not riding him confidently, and that's my fault, not his.
- He's also an inexperienced horse, and he's going to be wary of fences that look new and different - which is most of them so far in his life.
- He's also a jerky face, and so on a course, when there's a lot to look at, he's going to look at it, and I just have to be able to ride him through that - getting him straight and looking at the jumps as far away as possible.
- I'm scared of the big, solid jumps. This is new - being scared. And I'm scared because he's not a reliable jumper. And the only way he's going to get to be a reliable jumper is if I keep jumping him.

My plan of attack is this:
- Work at home on:
- Bigger fences (2'9" to 2'11")
- XC fences and "spreads"
- Courses (12 fences in a row)
- Everything at the canter
- Drop back down to Intro for the next couple of shows to build his confidence (and mine)
- Haul him to some of the recognized shows for the day for him to get the show feel without the nerves and expectations

And my goals are this:
- Ride him BN at a show with no refusals by this fall
- If I can't, send him for a month of training with Jonathan in the fall or spring
- And then if I still can't ride him, sell him and get a different horse

But I'm really perplexed why he would have been so completely unfocused at the derby when he did so well at the clinic. The obvious answer is my nerves, but I KNOW how to ride a show, and even if I'm nervous, I know what to do and was trying to do it. I just got ZERO response from him.
I don't want to sell him. I also don't want to screw him up. I've really never had a horse like this that is this much of a challenge. Although I noticed today, riding Mercury is SO easy. I think Willig is making me a better rider, but I'm so insecure about everything with him. I feel like I am just ruining both of us.
Here's our one fence (the first one) at the derby - note I am, horrors, jumping ahead. Argh.