Yes, you should use a backer. If you look through some of the posts in this forum, you'll see several conversations about downrigger boards, as well as some photos of others handywork. See what you can find in the archive, then fire away with the questions. There's pics from me and others that shows ways of maintaining the family boat look, while accomodating the downriggers.
One thing to keep in mind is the amount of torque that there is against the boat. You've got a downrigger with a base of about 6" square with an arm hanging out 4' with a 10# - 12# lead ball being pulled through the water. Thats a good amount of load on the base itself, now multiply that by 4 and you've got a lot of torque from your downrigger board being applied to the boat. Backerboard and fender washers are pretty much a must.
One thing to keep in mind is the amount of torque that there is against the boat. You've got a downrigger with a base of about 6" square with an arm hanging out 4' with a 10# - 12# lead ball being pulled through the water. Thats a good amount of load on the base itself, now multiply that by 4 and you've got a lot of torque from your downrigger board being applied to the boat. Backerboard and fender washers are pretty much a must.