For the ones that do get you. I have used lice shampoo on the welts that persist after my shower. Just dab a bit on the affected spots and allow to dry (3-5 min max) , fans help. Then just re-apply on the ones that continue to itch after a few hours. It dosen't always work, sometimes they get too deep too quick, but if it does work the next day it will not itch at all. Beats two weeks of misery.!. You can also squeeze them out but it will leave a bruise, and is not recommended. I have also had limited success with dryer sheets, rubbing them over areas they tend to get me.
I don't know about how to get them from your compost, maybe try planting those super hot peppers and mixing them into your pile.