Tina,
You are further along in your homeschooling journey than I am. My dd is almost 6 and my ds is 3.5, so I haven't gotten to the facts drills yet (and I likely won't do that). But I feel I have to chime in here. I keep coming back to your post and rereading it. Here's why:
I am your child (so to speak). I grew up in the ps system back in the 70s-80s. We were drilled on our "facts" and I never learned them! Never! Even to this day, I don't think I could pass a timed math facts test. I was always at or behind grade level because I couldn't pass/finish the timed drills. If I had all the time in the world, I could do the math, but I couldn't do it quickly. I hated math. I didn't want to do it at all! When I was in the 6th grade, I had a teacher that quit focusing on the "book" and did math the "real world" way. In fact, I don't think we opened the book all year. We had races at the board. He would call out a problem and we would race in groups and at our desks to finish the problem first. He would hand out golf tees to the winners. I didn't win a lot of tees, 3 in fact, and I still have them. That year was a turning point for me. I learned that I knew the why and how of math, but I didn't have speed. Not so much a big deal, but it was my first "A" in math. When I went to middle school, I tested into the advanced group, and started pre-algebra in 7th grade. I progressed along that track and completed calculus my senior year. I then went to college and pursued engineering. I transferred to another state, so I had to retake some of the math courses because there wasn't a direct equivalent, so I ended up with enough math hours to have a minor in math. It wasn't until I was working as an engineer that I learned some of my multiplication facts. I could do them, but they weren't memorized. I am living proof that you can do math without memorized facts. I do admit that things would have been "quicker" if I had memorized my facts, but I could still "do" the math.
Math is such a sensitive topic. Everyone has their opinion about which curriculum is best. In all honesty, I haven't done anything personally but Singapore, but I have helped my neice with Horizons. I love Singapore because it teaches the concepts. I like the way it introduces math. I think it provides a great base for higher math.
We, as moms, want our kids to develop a love of learning. I think we need to ask ourselves why we think we need to do certian things. Is it because it's what we did? Is it because that's what everyone else is doing? We need to be about what is best for our family. We need to seek guidance from our Maker.
I don't think timed drills are a necessary life skill. (Have you ever been timed when you balanced your checkbook?
) Math is a necessary life skill and I think it's important not to turn our children against math. It is one of the things they will do forever. Relax it will all work itself out. Give it some time.
This is just my experience and opinion. I don't mean to offend or come across as harsh. I just want you to know that one can become "good" at math without having the "facts" memorized.
I will be praying for you and your son.
However, we know that we really don't want that! 
We do addition on Mon, subtraction on Tues, multiplication on Wed, and division on Thurs. They have gotten their facts down cold using this site, and they never seem to get tired of it. 
