Here's a different idea. Get your hands on a solid alum rod about 2 1/2 -3" in diameter and a couple of feet long. Then use it as a drift and place one end on the knock off and hit the other with whatever you like.
A few things here. One, is that you no longer need to "reach" into the wheel with your swing therefor reducing the possibility of hitting the wheel with the hammer. Two, is that you can stand more upright as you make your swing. Lastly you can use a nice steel hammer. They are cheap, smaller for the same weight, and don't get deformed in use. 2-3 pounds is enough weight for the hammer.