instant Rsync service — isync

August 2022 — Update:

I have now ported this code to a compiled binary, same concept but the rsync is now done via a Crystal-based binary called “poni”, it uses iNotify C binding to detect changes and start rsync, see more here:

GitHub - perfecto25/poni: file sync based on inotify events

— -

suppose you have 2 servers (server A, server B), each having their own local filesystems, with no shared NAS or SAN mounts

you need to monitor a set of files and directories on A and instantly sync any changes to B

the usual way to do this is to create a cron job that runs every few minutes and rsyncs A > B, but the minimum crontab interval you can configure is 1 minute, anything less, lets say 30 second interval — doesnt work with cron

What’s needed is an intelligent syncing service that can poll every few seconds for file changes, and sync once a change is detected, unlike a brute force rsync via cron.

After trying several applications, including lsyncd, I could not get any to work properly.

For example lsyncd started syncing correctly, but then stopped working and would only sync again on a service restart. Additionally configuring it was a not straight forward, it has many options and is very picky with its unique config syntax

After looking around for options, I decided to use a simple python script built on Linux kernel module INOTIFY — which alerts on file or directory changes, and a great Python module called Watchdog

Watchdog uses Inotify bindings to create actions based on file changes.

If a file is created, modified, deleted, moved, etc — Watchdog can run a function or perform an action in reaction to this event.

ISYNC

isync is a simple python module that uses Watchdog and a simple configuration file to create a watcher

paths = [
  '/home/USER/dir1',     
  '/home/USER/dir2',     
  '/opt/testfile'
] 
# remote host to rsync to
remote = 'serverB' 
# polling interval in seconds
interval = 20

config.py

The paths are files or directories that you want to monitor

Remote is the server you want to rsync the changed files to

Interval is how often you want the service to cycle and check for updates

If you open isync.py, you can see the actions you can run for every event. For example, ON_MODIFY, it will rsync the source to the target (and log to a file)

def on_modified(self, event):        
    remote_path = os.path.dirname(event.src_path)
    logger.info(f"MODIFY  rsync -azpP {event.src_path} {remote}:{remote_path}")        
     os.system(f"rsync -azpP {event.src_path} {remote}:{remote_path}")

You can configure specific actions for each event type.

Installation

1. clone the repository
2. create virtualenv
3. install requirements into virtual env
4. open isync.py and adjust the shebang to point to your virtual env

!#/path/to/your/virtual/env

To setup isync as a service,

1. edit isync.service and change the User section to whatever user will run the rsync process (make sure this user can SSH to the other remote host)
2. copy the isync.service to /etc/systemd/system/
3. sudo systemctl daemon-reload
4. sudo systemctl start isync.service