What is Attenuated Alert?
Attenuated alerts are a mechanism designed to prevent a flood of alerts from a single source, such as a device or a specific metric, within a short timeframe. This helps to reduce alert noise and improve the signal-to-noise ratio.
key Benefits of Attenuated Alerts
- Reducing Alert Fatigue: Attenuated alerts help prevent an overwhelming number of alerts from a single source, making it easier for operations teams to focus on critical issues.
- Improving Alert Signal-to-Noise Ratio: By suppressing repetitive alerts, attenuated alerts help to highlight truly important events and reduce the number of false positives.
- Preventing Gateway Overload: Limiting the number of alerts sent to the cloud helps to prevent the gateway from being overloaded and ensures that critical alerts are always delivered.
How the Attenuated Alerts Work?
The following are the current behaviors of Attenuated Alerts, which apply only to the gateway:
- Throttling Limit: The gateway has a default limit of 4 alerts per 10 minutes that are sent to the OpsRamp Cloud. Any alerts beyond the 4th are held back and not sent until the 10-minute reset period ends. This applies to all alert severities, including Critical, Warning, or OK.
- Resumption: After the 10-minute reset period, the gateway resumes sending last saved alerts with the subject prefixed as “Attenuated Alert:” to indicate that the alert was previously suppressed.
Configuration
The alert throttling behavior can be configured in the vprobe.conf file that is located at /opt/gateway/vprobe/conf/vprobe.conf:
Alert.throttling.reset.freq: Specifies the reset interval (default: 10 minutes).Alert.throttling.limit: Specifies the maximum number of alerts allowed within the reset interval (default: 4 alerts).
After adjusting these parameters, you need to restart the vprobe service to optimize the alert throttling behavior according to your specific monitoring requirements.