Storing Network Monitoring Data in InfluxDB

Part 2 of Our Network Monitoring Series

In our previous post, we built a Python script that scans a subnet for SNMP-enabled servers and identifies those that are actively responding. Now that we know which servers are SNMP-accessible, we’ll take the next step: collecting key metrics from these servers and storing them in a time-series database, InfluxDB.

InfluxDB is designed specifically for storing time-series data like network performance metrics. It provides a powerful way to analyze trends over time and visualize data in real-time. Let’s dive into setting up InfluxDB, connecting it to our SNMP scanner, and configuring it to store network metrics.

Step 1: Setting Up InfluxDB

Before we can store data, we need to install and configure InfluxDB. Follow these steps to get InfluxDB up and running.

1. Install InfluxDB:

• For Ubuntu/Debian

Copied!sudo apt update
sudo apt install -y influxdb2
sudo apt update
sudo apt install -y influxdb2

For CentOS/RHEL

Copied!sudo yum install -y influxdb2
sudo yum install -y influxdb2

2. Start and Enable InfluxDB:

Copied!sudo systemctl start influxdb
sudo systemctl enable influxdb
sudo systemctl start influxdb
sudo systemctl enable influxdb

3. Set Up InfluxDB:

• Access the setup page at http://localhost:8086 and configure your username, password, organization, and bucket.
• Alternatively, you can use the CLI:

Copied!influx setup --username <your-username> --password <your-password> --org <your-org> --bucket <your-bucket> --retention 30d</your-bucket></your-org></your-password></your-username>
influx setup --username <your-username> --password <your-password> --org <your-org> --bucket <your-bucket> --retention 30d

Step 2: Configuring the Python Script to Write Data to InfluxDB

We’ll modify our Python script to connect to InfluxDB and store SNMP metrics. In this step, we’ll collect metrics such as CPU usage, memory, and network traffic from our responsive servers.

Configuration in config.py:

In config.py, define the InfluxDB connection parameters:

Copied!# config.py

INFLUXDB_URL = 'http://localhost:8086'
INFLUXDB_TOKEN = 'your-influxdb-token'
INFLUXDB_ORG = 'your-org'
INFLUXDB_BUCKET = 'network_metrics'
# config.py

INFLUXDB_URL = 'http://localhost:8086'
INFLUXDB_TOKEN = 'your-influxdb-token'
INFLUXDB_ORG = 'your-org'
INFLUXDB_BUCKET = 'network_metrics'

Modifying the Python Script:

Here’s the main part of our Python script that collects data from SNMP and writes it to InfluxDB. We’re using the pysnmp library to gather SNMP data and influxdb_client to store it.

Copied!import sys
import time
import importlib
from pysnmp.hlapi import *
from pysnmp.proto.rfc1905 import NoSuchInstance
from influxdb_client import InfluxDBClient, Point, WritePrecision, WriteOptions
from config import INFLUXDB_URL, INFLUXDB_TOKEN, INFLUXDB_ORG, INFLUXDB_BUCKET
from oids import OIDS as DEFAULT_OIDS

# Functie om server-specifieke OID-configuratie in te laden
def load_server_specific_oids(snmp_host):
    try:
        # Vervang punten door underscores om een geldig module naam te maken
        config_module_name = f"{snmp_host.replace('.', '_')}_config"
        server_config = importlib.import_module(config_module_name)
        return {**DEFAULT_OIDS, **server_config.OIDS}  # Combineer de OIDs
    except ModuleNotFoundError:
        print(f"No specific configuration found for {snmp_host}, using default OIDs.")
        return DEFAULT_OIDS

# Functie om SNMP-data op te halen
def get_snmp_data(oid, snmp_host, community):
    iterator = getCmd(
        SnmpEngine(),
        CommunityData(community),
        UdpTransportTarget((snmp_host, 161)),
        ContextData(),
        ObjectType(ObjectIdentity(oid))
    )

    errorIndication, errorStatus, errorIndex, varBinds = next(iterator)

    if errorIndication:
        print(f"SNMP error: {errorIndication}")
        return None
    elif errorStatus:
        print(f"SNMP error at {errorIndex} {errorStatus.prettyPrint()}")
        return None
    else:
        for varBind in varBinds:
            if isinstance(varBind[1], NoSuchInstance):
                print(f"NoSuchInstance for OID {oid}")
                return None
            try:
                return float(varBind[1])
            except ValueError:
                return varBind[1]

# Functie om data naar InfluxDB te schrijven, inclusief snmp_host als tag
def write_to_influxdb(measurement, value, snmp_host):
    with InfluxDBClient(url=INFLUXDB_URL, token=INFLUXDB_TOKEN, org=INFLUXDB_ORG) as client:
        write_api = client.write_api(write_options=WriteOptions(batch_size=1))
        point = (Point(measurement)
                 .tag("snmp_host", snmp_host)  # Voeg de SNMP-host toe als tag
                 .field("value", value)
                 .time(int(time.time() * 1_000_000_000), WritePrecision.NS))
        write_api.write(INFLUXDB_BUCKET, INFLUXDB_ORG, point)

# Hoofdloop om periodiek data te verzamelen en op te slaan
def main():
    if len(sys.argv)  <community>")
        sys.exit(1)

    snmp_host = sys.argv[1]
    community = sys.argv[2]

    oids = load_server_specific_oids(snmp_host)

    while True:
        print(f"Collecting SNMP data for {snmp_host}...")
        for measurement, oid in oids.items():
            value = get_snmp_data(oid, snmp_host, community)
            if value is not None:
                write_to_influxdb(measurement, value, snmp_host)
                print(f"{measurement}: {value}")
            else:
                print(f"Failed to collect data for {measurement}")
        sys.exit()
        time.sleep(60)

if __name__ == "__main__":
    main()</community>
import sys
import time
import importlib
from pysnmp.hlapi import *
from pysnmp.proto.rfc1905 import NoSuchInstance
from influxdb_client import InfluxDBClient, Point, WritePrecision, WriteOptions
from config import INFLUXDB_URL, INFLUXDB_TOKEN, INFLUXDB_ORG, INFLUXDB_BUCKET
from oids import OIDS as DEFAULT_OIDS

# Functie om server-specifieke OID-configuratie in te laden
def load_server_specific_oids(snmp_host):
    try:
        # Vervang punten door underscores om een geldig module naam te maken
        config_module_name = f"{snmp_host.replace('.', '_')}_config"
        server_config = importlib.import_module(config_module_name)
        return {**DEFAULT_OIDS, **server_config.OIDS}  # Combineer de OIDs
    except ModuleNotFoundError:
        print(f"No specific configuration found for {snmp_host}, using default OIDs.")
        return DEFAULT_OIDS

# Functie om SNMP-data op te halen
def get_snmp_data(oid, snmp_host, community):
    iterator = getCmd(
        SnmpEngine(),
        CommunityData(community),
        UdpTransportTarget((snmp_host, 161)),
        ContextData(),
        ObjectType(ObjectIdentity(oid))
    )

    errorIndication, errorStatus, errorIndex, varBinds = next(iterator)

    if errorIndication:
        print(f"SNMP error: {errorIndication}")
        return None
    elif errorStatus:
        print(f"SNMP error at {errorIndex} {errorStatus.prettyPrint()}")
        return None
    else:
        for varBind in varBinds:
            if isinstance(varBind[1], NoSuchInstance):
                print(f"NoSuchInstance for OID {oid}")
                return None
            try:
                return float(varBind[1])
            except ValueError:
                return varBind[1]

# Functie om data naar InfluxDB te schrijven, inclusief snmp_host als tag
def write_to_influxdb(measurement, value, snmp_host):
    with InfluxDBClient(url=INFLUXDB_URL, token=INFLUXDB_TOKEN, org=INFLUXDB_ORG) as client:
        write_api = client.write_api(write_options=WriteOptions(batch_size=1))
        point = (Point(measurement)
                 .tag("snmp_host", snmp_host)  # Voeg de SNMP-host toe als tag
                 .field("value", value)
                 .time(int(time.time() * 1_000_000_000), WritePrecision.NS))
        write_api.write(INFLUXDB_BUCKET, INFLUXDB_ORG, point)

# Hoofdloop om periodiek data te verzamelen en op te slaan
def main():
    if len(sys.argv) < 3:
        print("Usage: python store_snmp.py <snmp_host> <community>")
        sys.exit(1)

    snmp_host = sys.argv[1]
    community = sys.argv[2]

    oids = load_server_specific_oids(snmp_host)

    while True:
        print(f"Collecting SNMP data for {snmp_host}...")
        for measurement, oid in oids.items():
            value = get_snmp_data(oid, snmp_host, community)
            if value is not None:
                write_to_influxdb(measurement, value, snmp_host)
                print(f"{measurement}: {value}")
            else:
                print(f"Failed to collect data for {measurement}")
        sys.exit()
        time.sleep(60)

if __name__ == "__main__":
    main()

Configure OID’s to monitor in oids.py

Copied!OIDS = {
    'cpu_load': '1.3.6.1.4.1.2021.10.1.3.1',
    'memory_usage': '1.3.6.1.4.1.2021.4.6.0',
    'network_in': '1.3.6.1.2.1.2.2.1.10.2',
    'network_out': '1.3.6.1.2.1.2.2.1.16.2'
}
OIDS = {
    'cpu_load': '1.3.6.1.4.1.2021.10.1.3.1',
    'memory_usage': '1.3.6.1.4.1.2021.4.6.0',
    'network_in': '1.3.6.1.2.1.2.2.1.10.2',
    'network_out': '1.3.6.1.2.1.2.2.1.16.2'
}

With this setup, your script will now gather metrics from the servers identified in the first post and write them to InfluxDB.

Step 3: Collecting SNMP Metrics to Store in InfluxDB

To collect useful network metrics, we’ll target OIDs (Object Identifiers) for CPU usage, memory, and network I/O. Here’s a list of recommended OIDs:

• CPU Load: .1.3.6.1.4.1.2021.10.1.3.1
• Memory Usage: .1.3.6.1.4.1.2021.4.6.0
• Disk Usage: .1.3.6.1.4.1.2021.9.1.9.1
• Network Input: .1.3.6.1.2.1.2.2.1.10
• Network Output: .1.3.6.1.2.1.2.2.1.16

What’s Next?

With InfluxDB storing these critical metrics, we’re ready to visualize the data. In the next post, we’ll explore setting up Grafana to create dashboards that provide real-time insights into our network’s health and performance.

Stay tuned for Part 3, where we’ll turn data into actionable insights with Grafana visualizations!

This setup will serve as the foundation for tracking our network’s state over time. By continuously collecting and storing SNMP metrics, we’ll be able to spot trends, anticipate issues, and maintain the health of our network infrastructure.