How to get (event) when a player stop shielding?

I want when a player hold right click in any sword a shield go to their off-hand and when releases the right click the shield will be take off.

Code:

@EventHandler
private void event(PlayerInteractEvent e)
{
    if(e.getItem() != null && e.getPlayer().getInventory().getItemInOffHand().getType() == Material.AIR && (e.getAction() == Action.RIGHT_CLICK_AIR || e.getAction() == Action.RIGHT_CLICK_BLOCK))
    {
        switch(e.getItem().getType())
        {
            case WOODEN_SWORD:
            case STONE_SWORD:
            case IRON_SWORD:
            case GOLDEN_SWORD:
            case DIAMOND_SWORD:
                e.getPlayer().getInventory().setItemInOffHand(new ItemStack(Material.SHIELD));
            default:
                break;
        }
    }
}

The shield is being set, but how I'll take off the shield if (I think) there isn't an event when he stops shielding or verifying if player is holding/releasing right clicking?

Player#isBlocking() won't help because it have a delay of 6 ticks.

Using 1.14.4.