Introduction

Syslog events are messages generated by devices, applications, or systems, which are then captured and sent to a syslog server for storage, monitoring, and analysis. These events provide valuable information about system operation and status, essential for troubleshooting and security analysis.

Prerequisites

The gateway server must be installed in the managed environment.

Add a syslog configuration profile

The Syslog Monitor Configuration functionality is used to filter messages and generate alerts from the messages. For example, to stop receiving alerts for mail system messages, the system can be “trained” to not monitor those messages.

  1. Select Setup > Monitoring > Syslog Monitoring Configuration.
  2. Click the Configuration Profiles tab.
  3. Click Create New.
  4. Enter the following information:
PropertyDescription
Partner(required) The partner scope is prepopulated.
Client(required) Client scope. Prepopulated if already selected.
Management profilesList of management profiles configured in Resources > Management Profiles
Configuration Name(required) Name of this configuration profile.
DescriptionDescription of this profile.
  1. Click Next.
  1. In the Global Filters page, enter the following configuration profile properties:
PropertyDescription
Severity(required) Message severity level drop-down list from RFC 5424.
Facility(required) Type of message to monitor drop-down list from RFC 5424.
Resource Filter 1
  • IP Filter Range: (required) IP address range of servers to monitor for syslog messages.
  • Enter asterisk (*) to receive messages from all devices, although, this is not recommended because of the heavy traffic load imposed on the gateway.
  • Rule Name: Previously defined rule dropdown list selection. The Action and Tag columns are populated from the rule properties.
  • Click the + symbol to add another previously defined rule to the profile. Rules are executed in the order in which they occur in the profile.
  • Click the delete icon to delete a rule.
  1. (optional) To define a new rule and add it to the resource filter, click the + New Rule button, enter the rule properties as described in Add a syslog monitoring rule, and click Save.
  2. Click Save to apply the profile configuration and list the defined configurations.
  1. Click + Add to define another profile.

Add a syslog monitoring rule

  1. Select Setup > Monitoring > Syslog Monitoring Configuration.
  2. Click the Rules tab.
  3. Click Create New.
  4. Enter the following information:
PropertyDescription
Scope(required) Scope to which the rule is applied:
  • Partner Rule: Applies to partner.
  • Client-specific Rule: Applies to specified client, only.
Client(required) If Client-specific Rule scope is selected, select from drop-down client list.
Name(required) Rule name.
Action(required) Action to apply to messages that match this rule:
  • INCLUDE: Send matching data in generated alert message.
  • EXCLUDE: Send all but matching data in generated alert message.
RegEx Pattern(required) Regex pattern to apply for matching messages. Matching groups can be used as parameters, such as ${1}, in generating alert messages.
Metric Name(required) User-defined metric name, which can be specified using regex.
Component(required) User-defined component name.
Alert Subject(required) User-defined alert subject.
Alert DescriptionAlert description.
Alert Severity(required) Alert severity level:
  • Critical
  • Warning
  • Info
  • Ok
TagsUser-defined tag name.
  1. Click Submit.

View and edit configuration profile

  1. View the defined configuration profiles in the Setup > Monitoring > Syslog Monitoring Configuration > Configuration Profiles tab:
  1. Click the configuration profile name to see the detailed profile information.
  2. (optional) You can change profile properties as needed. When complete, click Next.
  3. (optional) Change existing rule properties or add new rules as described in Add a syslog monitoring rule.
  4. Click Save.

View rules

  1. View rule details in the Setup > Monitoring > Syslog Monitoring Configuration > Rules tab.
  1. Change existing rule properties or add new rules as described in Add a syslog monitoring rule.
  2. Click Save.

Search profiles and rules

The following Syslog Monitoring Configuration search options are available.

Search profiles and rules

Use the configuration name and rule name to find a configuration profile and rules using.

To search for specific criteria, use the Advanced search option.

To search profiles using specific criteria:

  1. Click Advanced.

  2. In ADVANCED SEARCH, enter :

    • Client
    • Configuration Name

  3. Click Search.

The Configuration Profile screen displays search results.

To search rules using specific criteria:

  1. Click Advanced.

  2. In ADVANCED SEARCH, enter the following information:

    • Client
    • Action
    • Tags

  3. Click Search.

The Rules screen displays search results.

Delete profiles and rules

Use the Remove option to delete existing configuration profiles and rules.

To delete a Syslog configuration from a single gateway management profile, remove the managed profile from the respective Syslog configuration.

How to process the received syslog event message

If any syslog event is received to the gateway, it first checks the event severity against the criteria specified in the first profile that is sorted alphabetically. If the severity matches, it then proceeds to check the facility. If the facility also matches, it proceeds to evaluate the rules within that profile. If a rule is matched, the process stops; otherwise, it moves on to the next profile and repeats the same sequence of checks.

Examples

  1. Let us consider three configuration profiles, P1, P2, and P3, and event E1 is received to the gateway, the gateway first checks the event against the P1 profile. If event E1 matches the severity specified in profile P1, then the gateway checks the facility in profile P1. If it matches, then it proceeds to check the rules one by one in the P1 profile. If any of the rules in P1 do not match, the gateway moves on to profile P2 and checks the severity specified in the profile P2, it matches then checks the facility. If the facility matches the selected facility of the profile P2 then check the rules in the profile P2. If any of the rules matches, the alert will be generated, and it will not check other profiles.
  2. If Event E2 is received to the gateway, it checks against profile P1. If the severity of the event matches, then check the facility in profile P2. If it matches, then check the rules in the profile P1. If any of the rules does not match in profile P1 then move to profile P2. If it checks the severity of the profile P2 it matches, checks the facility it is also matched then checks the rules. If the rules do not match, then move to the profile P3. If the severity of the profile P3 does not match, the gateway will drop the event.
  3. If Event E3 is received to the gateway, it checks the severity in profile P1. If it matches, then it proceeds to check the facility in profile P1 and checks the rules in P1 one by one. If any rule matches in any profile, an alert is generated.
  4. If Event E4 is received to the gateway, it first verifies the severity in profile P1. If it does not match, the gateway will drop the event.
  5. If Event E5 is received to the gateway, it initially verifies the severity in profile P1. If it matches, then it proceeds to check the facility in profile P1. If the facility does not match, the gateway will drop the event.

Troubleshooting Steps

To identify if the gateway is receiving syslog messages from an external syslog server.

  1. Login to the gateway as root and execute gcli

  2. Enable below flags
    flag add syslog.log on 40 loglevel set com.vistara.gateway.syslog.SyslogEventProcessor DEBUG 180

  3. Execute tail -100f /var/log/app/vprobe.log

  4. If the events are reaching the gateway but the gateway is skipping them, you can find the following messages in the log:

    ERROR 02-Mar-24 16:16:25,776 SyslogEventProcessor#96: Skipped:Severities. IP: 10.212.0.7, severity: 5, Profile: ABC Test syslog profile ERROR 01-Mar-24 16:16:25,777 SyslogEventProcessor#56: Time : 0 s

    If you need alerts on these events create/update the configuration profile accordingly.

  5. To verify whether the syslog events are reaching the gateway or not, capture packets at the gateway as the root user.

    tcpdump -i any "udp port 514" -s 3000 -w /tmp/syslog.pcap

    If no packets are captured, or if the required packets are not captured, it is necessary to check configurations in the syslog server.

Additional Information

  • Syslog events can be categorized into 8 types of events based on severity: Emergency, Alert, Critical, Error, Warning, Notice, Informational, and Debug. OpsRamp can generate Critical, Warning, OK, and Info alerts. These alerts are generated based on rules defined in the OpsRamp UI.

  • If you want an OK alert, they must define a specific rule for it. If the OK alert needs to be appended to an existing critical/warning alert, the OpsRamp tool allows it, but it requires the Metric name, component name, and resource to remain the same; otherwise, it generates a separate alert.