Saltstack Beacons and Reactors

2 min readAug 8, 2019

Salt is a great python-based framework on which to build out your infrastructure.

This article shows how to use Salt’s Beacons and Reactors modules to enable an automatic, self-healing infrastructure.

This example is very simple.

We have a file, for example /etc/hosts

if anyone makes an out-of-band change to this file, we want Salt to jump in and revert it back to your configured state instantly, without waiting for the next highstate run.

The process works like this,

A minion has a Beacon defined to send an Event to the salt master, whenever the file /etc/hosts is modified or deleted.

It will also monitor the file change every 5 secons (interval), and will not monitor when you run a Salt highstate or state.apply (to avoid duplicate efforts)

/etc/salt/master.d/reactor.confreactor:
  - salt/beacon/*/inotify//etc/hosts:
      - /srv/salt/reactor/revert_etc_hosts.sls------------------------------------------------
/srv/salt/reactor/revert_etc_hosts.slsrevert_etc_hosts:
  local.state.sls:
    - tgt: {{ data['id'] }}
    - arg: 
      - formula.etc_hosts------------------------------------------------
/etc/salt/minion.d/beacons.confbeacons:
  inotify:
    - files: 
       /etc/hosts:
         mask: ['modify', 'delete_self']
    - interval: 5
    - disable_during_state_run: True

** make sure to restart both Master and Minion after making changes to these files

** in order to use I-notify, install this pip package on your salt minions

pip install pyinotify

Salt master is always listening for events from its minions ( you can monitor events on the saltmaster by running this; )

root@saltmaster> salt-run state.event pretty=True

When master gets an event that matches its Reactor config, it reacts by applying the state defined in reactor.conf, in this case, it will apply “revert_etc_hosts.sls”

In this example, it will apply the regular Salt formula for /etc/hosts (/srv/salt/formula/etc_hosts)

ie,

/etc/hosts:
  file.managed:
    - source: salt://{{ slspath }}/files/etc_hosts.j2
    - template: jinja
    - user: root
    - group: root
    - mode: 644

Once an out-of-band change is made to /etc/hosts on the minion, master detects it and reverts the file back to its configuration state.

root@saltmaster> salt-run state.event pretty=Truesalt/beacon/min1/inotify//etc/hosts {
    "_stamp": "2019-08-06T09:59:30.442720", 
    "change": "IN_DELETE_SELF", 
    "id": "min1", 
    "path": "/etc/hosts"
}
salt/beacon/min1/inotify//etc/hosts {
    "_stamp": "2019-08-06T09:59:30.444459", 
    "change": "IN_IGNORED", 
    "id": "min1", 
    "path": "/etc/hosts"
}
20190806055930621236 {
    "_stamp": "2019-08-06T09:59:30.622472", 
    "minions": [
        "min1"
    ]
}
salt/job/20190806055930621236/new {
    "_stamp": "2019-08-06T09:59:30.622801", 
    "arg": [
        "formula.etc_hosts"
    ], 
    "fun": "state.sls", 
    "jid": "20190806055930621236", 
    "minions": [
        "min1"
    ], 
    "missing": [], 
    "tgt": "min1", 
    "tgt_type": "glob", 
    "user": "root"
}
salt/job/20190806055930673132/ret/min1 {
    "_stamp": "2019-08-06T09:59:30.731112", 
    "cmd": "_return", 
    "fun": "state.sls", 
    "fun_args": [
        "formula.etc_hosts"
    ], 
    "id": "min1", 
    "jid": "20190806055930673132", 
    "out": "highstate", 
    "retcode": 1, 
    "return": [
        "The function \"state.sls\" is running as PID 3031 and was started at 2019, Aug 06 05:59:30.621236 with jid 20190806055930621236"
    ], 
    "success": true
}

Additional resources on Reactors + Beacons:

https://www.linode.com/docs/applications/configuration-management/monitoring-salt-minions-with-beacons/

Saltstack Beacons and Reactors

Written by Mike R