So you've tried to avoid them by shaving in the direction of hair growth, right? This helps to train the hair to grow out straight & not curl back into the skin. Also, always make sure the skin is wet to help soften the hairs to make shaving easier.
However if you're still getting them, i have learnt that salicylic acid, which is a beta hydroxide acid, is good for exfoliation because it penetrates into the skin to dissolve the hair. For this I use a toner that I stopped using on my face, however as it contains alcohol I think I will switch from that as it makes my underarms fairly dry after using it, which I then moisturize after wards.
You can also buy serums that contain BHA's for after shaving.
Hope this helps some!
Source: http://www.skinbiology.com/ingrownhairs.html