Expanding Network Monitoring with Syslog and Zabbix Data

Part 5 of Our Network Monitoring Series

Our SNMP-based monitoring system is up and running, collecting vital metrics on CPU, memory, and network activity. In this next step, we’ll enhance our insights by integrating additional data sources: syslog data for a richer view of system logs and events, and the Zabbix API for real-time alert data. This added context lets us detect security events, track system errors, and act on important alerts—all from within our centralized monitoring setup.

In this post, we’ll walk through connecting to these new data sources and merging them into our existing system.

Step 1: Collecting Syslog Data for Enhanced Event Monitoring

Syslog provides valuable information about server activity, such as logins, errors, warnings, and other system events. By including syslog data, we gain a comprehensive view of network activity that goes beyond performance metrics.

Setting Up a Remote Syslog Server:

Configure your servers to send syslog data to a centralized logging server. You can use tools like rsyslog or syslog-ng to direct all logs to a remote syslog server, allowing us to process them centrally.

Storing Syslog Data in InfluxDB:

To store syslog data in InfluxDB, use a tool like Telegraf with a syslog input plugin. Telegraf can capture syslog messages and send them directly to InfluxDB for storage.
Example Telegraf configuration


Copied![[inputs.syslog]]
  address = "udp://:6514"  # Syslog listener
[[outputs.influxdb]]
  urls = ["http://localhost:8086"]
  token = "your-influxdb-token"
  organization = "your-org"
  bucket = "syslog_data"
[[inputs.syslog]]
  address = "udp://:6514"  # Syslog listener
[[outputs.influxdb]]
  urls = ["http://localhost:8086"]
  token = "your-influxdb-token"
  organization = "your-org"
  bucket = "syslog_data"

Creating Syslog Dashboards in Grafana:

With syslog data stored in InfluxDB, you can create dashboards in Grafana to visualize log data, including login attempts, error messages, and system warnings.
Set alerts for specific syslog messages or high-severity events, such as failed login attempts, which may indicate security threats.

Step 2: Integrating Zabbix API for Real-Time Alerts

Zabbix offers robust monitoring and alerting tools, and its API allows us to integrate its alerts directly into our existing setup. Here’s how we can use the Zabbix API to enrich our system:

Connecting to the Zabbix API:

Use the Zabbix API to retrieve important alerts and status data. The Zabbix API provides endpoints to get current alerts, problems, and events in real-time.
Example code to connect and retrieve data from the Zabbix API:


Copied!import requests

ZABBIX_URL = "http://zabbix_server/api_jsonrpc.php"
ZABBIX_USER = "your_username"
ZABBIX_PASSWORD = "your_password"

# Authenticate with Zabbix
def get_zabbix_token():
    payload = {
        "jsonrpc": "2.0",
        "method": "user.login",
        "params": {
            "user": ZABBIX_USER,
            "password": ZABBIX_PASSWORD
        },
        "id": 1
    }
    response = requests.post(ZABBIX_URL, json=payload)
    return response.json()["result"]

# Retrieve alerts or problems
def get_zabbix_alerts(token):
    payload = {
        "jsonrpc": "2.0",
        "method": "problem.get",
        "params": {
            "output": "extend",
            "sortfield": ["eventid"],
            "sortorder": "DESC",
            "recent": "true"
        },
        "auth": token,
        "id": 1
    }
    response = requests.post(ZABBIX_URL, json=payload)
    return response.json()["result"]

token = get_zabbix_token()
alerts = get_zabbix_alerts(token)
print(alerts)
import requests

ZABBIX_URL = "http://zabbix_server/api_jsonrpc.php"
ZABBIX_USER = "your_username"
ZABBIX_PASSWORD = "your_password"

# Authenticate with Zabbix
def get_zabbix_token():
    payload = {
        "jsonrpc": "2.0",
        "method": "user.login",
        "params": {
            "user": ZABBIX_USER,
            "password": ZABBIX_PASSWORD
        },
        "id": 1
    }
    response = requests.post(ZABBIX_URL, json=payload)
    return response.json()["result"]

# Retrieve alerts or problems
def get_zabbix_alerts(token):
    payload = {
        "jsonrpc": "2.0",
        "method": "problem.get",
        "params": {
            "output": "extend",
            "sortfield": ["eventid"],
            "sortorder": "DESC",
            "recent": "true"
        },
        "auth": token,
        "id": 1
    }
    response = requests.post(ZABBIX_URL, json=payload)
    return response.json()["result"]

token = get_zabbix_token()
alerts = get_zabbix_alerts(token)
print(alerts)

Storing Zabbix Alerts in InfluxDB:

Once retrieved, store the Zabbix alerts in InfluxDB by adapting the SNMP InfluxDB script. You can structure Zabbix alerts as a separate measurement, allowing Grafana to query and visualize these alerts.
Alerts can include key details such as severity, affected host, and time of the event.

Creating Alert Panels in Grafana:

With Zabbix alerts now in InfluxDB, set up panels in Grafana to visualize and filter alerts by severity, affected host, or issue type.
Configure alert thresholds and notifications for critical issues, ensuring your team receives timely updates.

Step 3: Merging and Analyzing Data from Multiple Sources

By combining SNMP, syslog, and Zabbix data, we create a comprehensive monitoring solution with rich, actionable insights.

Creating Combined Dashboards:

Use Grafana to merge SNMP metrics, syslog logs, and Zabbix alerts into a single dashboard. This unified view helps to track both performance metrics and critical events in one place.

Setting Up Cross-Source Alerts:

Create custom alerts that trigger when multiple sources report issues. For example, if high CPU usage (SNMP) coincides with repeated login failures (syslog) and a Zabbix alert, this could indicate a targeted attack or security issue.
Cross-source alerts provide more context for identifying and resolving complex incidents.

Conclusion: Building a Proactive and Comprehensive Network Monitoring System

By adding syslog data and Zabbix alerts to our SNMP-based monitoring, we gain a well-rounded view of network activity and health. Our system now not only tracks key performance metrics but also provides real-time event and alert data, creating a more holistic approach to network monitoring.

In future posts, we’ll explore further enhancements, such as machine learning models for anomaly detection, enabling predictive insights, and automating responses to alerts.