Generating bandwidth snapshots with Saltstack + Iperf + PyGraphviz

Background

A primer on how to generate bandwidth rates between different datacenters, regions or hosts

Unless you have dedicated network tools and resources, its hard to grasp bandwidth across your hosts and regions. This is a challenge for sysadmins to grasp what their bandwidth usage and throughput rates are at any given time across different network providers.

this primer shows how to generate a visual snapshot of bandwidth rates between all your co-locations, datacenters, or hosts

this setup uses Saltstack to deploy iperf related configs to each host you want to use as a testing station, and also how to run and generate a visual chart by collecting iperf data from across all your test hosts

end result looks something like this, I ran this across 12 different hosts, each located in separate datacenters and regions (some are physical servers in Equinix datacenters, some are EC2 instances in different Amazon regions)

shows bandwidth between datacenters and AWS regions (in mbit/sec)

All servers in this example are running Centos or Rocky Linux and have Saltstack agents installed.

Snapshot Lifecycle

to generate this graph,

1. Agent IPerf Setup
deploy iperf related python scripts to your testing hosts, each host will try to iperf to all other hosts on your list, generate bw usage and store it in a JSON file,

ie, send this to JSON file

host1 > host2 = 20mbits
host1 > host3 = 2mbits
host1 > host4 = 130mbits
etc

2. Salt Runner
This entire iperf check process is kicked off by Salt Master using a Salt Runner that will cycle through a list of hosts, make each host generate bw json snapshot

Salt Runners are special python modules that run on Salt Master and can utilize the master’s backbone to pull information out of each agent

Once each host is done generating their JSONs, the Salt Master runner collects JSON data from each host and generates a visual diagram using Py bindings for GraphViz (great library for generating charts and diagrams)

Salt Master then adds a label showing both visual bw rates and tabular data showing these same rates below the graph to help read the data (sometimes its hard to read the rates on chart if theres a lot of connections)

All arrows pointing the bandwidth test are colored from red to green, red being the lowest bw, green being the best/highest

1. Agent setup

on each agent, deploy a snapshot.py script into /opt/pyperf
Create a virtualenv and install required py pkgs

root@agent1> cd /opt
python3 -m venv pyperf
./pyperf/bin/pip install dictor iperf3
root@agent1> vim /opt/pyperf/snapshot.py

add the following iperf snapshot generator script

Once called, this script will open up a config file and scan through all iperf targets, ie, which targets you want this agent to run b/w checks against

it will generate an output JSON file with a timestamp and name of each host plus its bandwidth, if file already exists, it will create a new timestamp with new data

config file looks like this,

root@agent1> cat /opt/pyperf/config.json
{
  "port": 5201,
  "targets": [
    "ec2uk1",
    "ec2mum1",
    "ec2us1",
    "dc123",
  ],
  "timespan": 60
}

These are hostnames for other servers in other regions, so you can get an idea of how much b/w you have from this Agent1 host to all other regions to where you have connectivity, here the timespan to run the iperf is 60 seconds

the output of each snapshot looks like this

cat /opt/pyperf/output/snapshot.json | jq
{
  "20220812_0119": {
    "ec2uk1": 2,
    "ec2mum1": 3,
    "ec2us1": 3,
    "dc123": 162,
  },
  "20220812_0256": {
    "ec2uk1": 15,
    "ec2mum1": 9,
    "ec2us1": 4,
    "dc123": 170
  },

here you can see there are 2 snapshots, for each host target it shows the bandwidth speed in mbits/sec

2. Salt Runner

Now configure your Salt Master to generate an iperf snapshot on each host and give you the final chart showing all b/w counts

go to your salt runner directory and add this script (path should be /srv/saltstack/state/_runners, but check your salt master config for runner path)

root@master> vim /srv/saltstack/state/_runners/pyperf.py

copy the Salt Runner from this gist

Once your Runner is ready, run it to generate bandwidth across your hosts,

root@master> salt-run pyperf.diagram

this will iperf each minion/agent in the list sequentially, then read the results in the JSON output file on each agent

It will then generate a GraphViz diagram showing you the bandwidth rates between all datacenters and print the readings in the label below the chart.

The chart is saved into /tmp/bw_$timestamp.png on your Salt Master.