Tim
I would use 6061, .063 (16ga.) for your panels. You can bend 6061 in T-6 hardness, just don't try a minumum radius on this stuff. Rule of thumb is the bend radius on harder materials should be 1 1/2 times the thickness, just set the blade back about 3/32 on your brake. In answer to your material removed from the vehicle feeling harder this is entirely normal. Most supply houses will not stock 6061 T-0 because it age hardens even when sitting in warehouse stock. I was told I needed to buy 1000 lbs. to get it in T-0, I am assuming this was mill direct. I would say that the ageing of the material was inevitable.
Most bending jobs I would use 5052 or 3003 but these alloys are not as strong as the 6061, and they cannot be hardened. If you go carefully with your bends and fit up your panels well you should have a great looking result.
You can polish this material also...really messy job!
Phil