In this case, the person you need to practice Compassion for most is yourself and learn how to balance the Compassion with another Unselfish Emotion, Equanimity. You need to balance your Compassion with Equanimity.
How do we do this? We have to understand what belongs to us and what belongs to them. We have to understand that we are the owner of our Kamma and they are the owner of their own Kamma. Just as we normally do a Compassion/Lovingkindness meditation where we focus on the difficulties of other people, trying to feel compassion for them and Lovingkindness for them, we can also do this another type of meditation for these people. We can instead reflect, "Each of us is the owner of our own Kamma." In this way, we can develop understanding of what belongs to us and what belongs to them. Then, if we can keep more balance, this helps us to reflect on a deeper level so that we can then get to compassion for them. And we may not be so affected by them, because we're not taking their hindrances personally.
In the Satipattana Sutta, the Buddha asks us to see the hindrances internally and externally. So If we can practice seeing these hindrances - especially things like sense desire, aversion, restlessness and worry, doubt - if we can see these things inside ourselves, if we can learn how to note them, if we can learn how to open to them and feel compassion for ourselves, then we can see below the personality level of somebody else, and see the hindrances that cause them to act the way they do like that.
We can see the suffering they are having because we've seen the suffering in ourselves from our own practice. We understand suffering and it's true cause. Then, instead of seeing them as an angry person who is hurting us, we can see, "There's is anger," and with that comes our understanding of how anger feels in ourselves, because we've observed it in ourselves - but only if we've practicing practiced with awareness and wisdom to understand that this thought brings this result. Cause and Effect.
So if we develop the practice of seeing the hindrances in ourselves and feeling more Compassion for ourselves, then it's much easier to see the hindrances in somebody else. We can then look below the personality level and see the suffering that they are having. But normally, it's difficult to get to Compassion for someone else because, just like it's suffering in ourselves, it's suffering when it comes to us.
We need to be able to see deeper in order to not take their suffering onto ourselves. Reflecting, "Each of us is the owner of our own Kamma," is very helpful. That helps us develop more equanimity so we can then contemplate on a deeper level. We having have a saying, "To care and not to care." When we understand this saying deeply, we are able to understand what we're responsible for and what they're responsible for.