Could be that your firing pin set screw is too tight or that the firing pin is mushroomed. Cleaning/polishing the firing pin might fix it, too.
Valerie is the P-32 gunsmith at K-T. she may even talk you thru a repair.
Check out the Kel-Tec Owners Group for all you wanted to know about the P-32 and more. Real nice site:
http://www.ktog.org/vbulletin/