When someone speaks harshly or acts cruelly toward us, how we respond depends primarily on their disposition. The Bhagavad-gita (16.04) mentions two distinct dispositions: godly and ungodly. In real-life, most people manifest  a blend of godliness and ungodliness. So, when analyzing any particular person, we need to consider their overall behavior over a period of time. 

Both the godly and the ungodly have vices, but their attitude toward their vices differs drastically. The godly are sometimes overpowered by their vices; the ungodly are oftentimes powered by their vices. 

Consider the vice of anger. The godly know that succumbing to anger is bad. When they occasionally speak or act angrily, it’s because they are temporarily overpowered by anger. Soon, they repent, apologize and make amends. Wanting to prevent relapses, they strive earnestly to purify themselves. They are on the path of reform and deserve to be forgiven. 

In contrast, the ungodly delight in their anger — they think that their anger gives them power. Far from resisting anger, they give themselves to it (16.12). They get a perverse pleasure in making others cower in fear and in exploiting, abusing and wounding others. After such abusive behavior, if they are required to keep up appearances, they may nominally apologize. But even while apologizing, they rationalize. As they feel no remorse internally, they are unlikely to reform externally. Their anger-driven aggression will not just recur but also worsen. Forgiving them, in the sense of letting them go scot-free, is foolish and can even be fatal — they may degrade to murderous levels (16.14). They won’t check themselves unless warned strongly about the consequences awaiting them. If such warnings don’t deter them, then they need to be unflinchingly brought to justice.

One-sentence summary:

Those who are occasionally overpowered by their vices can be forgiven, but those who are regularly powered by their vices need to be punished.