Remove the add-in when it is no longer used in your organization, for example, when the Cloud App Security license expires.
According to Microsoft, it can take up to 24 hours for the removal of an add-in to apply at your users' side.