UniFTP — automated FTP framework

FTP or rather sFTP (via SSH) is a reliable and time-tested method to send and receive files between 2 parties

At my company, whenever someone wants to FTP something to a client, or pull a file from a client, they write a quick bash script and are done.

If they want to connect using a SSH keypair, they just pass a -o IdentityFile parameter to the private key,

if they want to use a password, they have to use LFTP

#!/bin/bash
user="vincentvega"
password="royalewithcheese"
lftp sftp://$user:$password@sftp.host.com -e "put file1;bye"

This creates a myriad of problems, primary one being that there are a ton of bash scripts for every scenario.

There are many cases where the user may need to encrypt a file they are sending, or append a prefix to the name, or connecting via password or some other method, all this requires adjusting your bash script to process all these requirements.

The UniFTP framework abstracts all this via a YAML file.

This framework makes FTP transfers easy to automate without touching Bash.

—

Example

for every connection, you specify the actions you want to take,

lets say we want to connect to a client Citibank, we generate a config file

/opt/uniftp > ./ftp.py -c citibank -g

this generates a sample YAML config in your path/clients/citibank (ie, /opt/uniftp/clients/citibank)

vim clients/citibank/config.yaml

now we specify we want to connect to the client using a password

we can name the connection name (or environment), lets call this connection “Prod”

prod:
  action: push
  host: ftp.citibank.com
  username: joesmith
  auth_type: password
  password: abracadabra
  remote_path: /uploads
  file_prefix: myfile_

here, we are Pushing a file to the client

we are connecting using “joesmith” username with password “abracadabra”

we want to upload our file on the remote server to the “uploads” folder, and also prefix our file with “myfile_<file>”

To send a file to the client we run

/opt/uniftp > ./ftp.py -c citibank -e prod -f /tmp/testfile.txt

this will push testfile.txt (renamed as myfile_testfile.txt) to Citibank FTP host, “uploads” folder

—

We can also use SSH key for authentication

prod:
  action: push
  username: joesmith
  auth_type: key
  privkey: id_rsa

UniFTP will search for a private key in clients/citibank/sshkeys folder,

if you want to use an external SSH key, just provide a direct path to the key

privkey: /home/joesmith/.ssh/id_rsa

Pull Files

you can also pull files from remote server to your local path, just specify action as “pull” and provide “local_path” where to store the file locally,

prod:
  action: pull
  remote_path: /downloads
  local_path: /tmp

This will pull files from remote host “downloads” folder to your local /tmp

Encryption

UniFTP can also encrypt your outgoing files using either GPG or 7zip encryption. See README on Github repo for more details.

—

Because UniFTP creates a known_hosts file whenever it connects to a remote host, it is suggested to create a dedicated service account on your host that will run this process

adduser uniftp

and run this process as another user, ie

joe > sudo -u uniftp /opt/uniftp/ftp.py -c citibank -e prod -f /tmp/file1

This will use “uniftp” user’s settings to connect to various remote hosts

See the full Github README for all UniFTP options:

https://github.com/perfecto25/uniftp