Configuration

Overview

This page explains how to write configuration files for rubackup in general. It does not provide a list of all possible options. Please refer to the module specific pages for these details.

The configuration is stored in yaml files, hence you really have to understand the YAML format first. Basically this is an easy way to represent structured data in text file in a way which is similar to JSON. Data in a yaml file correspond to hashes and lists once they have been loaded in ruby, hence they are easy to read and use without any extra processing required.

The configuration can be stored either in a single monolithic yaml file, or it can be split in multiple yaml files. This can be useful when the configuration is quite large as one separate configuration file can be created for each project. It also allows to share the configuration between multiple systems when part of the configuration are common on multiple systems. It is recommended to use different yaml files for the global section, the scheduling, and the various backup entries. When you use multiple yaml files this does not affect the way the configuration works. All files will be loaded and merged in memory and behave just like a single file. Hence you can put any type of section in any yaml file as long as they are valid yaml files.

By default rubackup is going to read the following files: /etc/rubackup.d/*.yaml. This can be overridden using command line:

rubackup.rb --config="/etc/somewhere/one-file-configuration.yaml"
rubackup.rb --config="/etc/somewhere/multiple-files/*.yaml"

There are multiple sections: the global section is mandatory and applies to everything. It defines for example which day should be used to make weekly backups. There are also resource sections. They define resources such as a list of S3 buckets or notification media which can be used to send an email at the end of the run. There are also backup entries which define what to do for a particular backup.

Minimal example

Here is a minimal configuration example which shows how to manage two type of backups which correspond to the document root and mysql database of a website. This configuration file can be put in the default directory in a file such as /etc/rubackup.d/rubackup.yaml so it gets picked up without having to pass any command line option to refer to it.

---
global:
    day_of_week: Mon
    day_of_month: 1
schedules:
    myschedule_long_term:
        daily: 7
        weekly: 4
        monthly: 12
    myschedule_short_term:
        daily: 7
        weekly: 4
        monthly: 3
entries:
    website1-tar:
        enabled: true
        backup_type: ModuleBackupTarball
        backup_opts:
            includes:
                - /var/www/website1
                - /etc/httpd/conf.d/website1.conf
            excludes:
                - '.git'
                - '.svn'
            command_opts:
                - '--sparse'
            compress_prog: gzip
            compress_opts:
                - '-6'
        backup_schedule: myschedule_short_term
        bakfile_dir: /backup/backup-web
        bakfile_basename: backup-website1-www
        bakfile_owner: root
        bakfile_group: root
        bakfile_mode: 0600
    website1-sql:
        enabled: true
        backup_type: ModuleBackupMysqldp
        backup_opts:
            dbhost: 127.0.0.1
            dbuser: website1
            dbname: website1
            dbpass: 'SqlPassForWebSite1'
            compress_prog: xz
            compress_opts:
                - '-4'
                - '--memlimit-compress=80MiB'
        backup_schedule: myschedule_long_term
        bakfile_dir: /backup/backup-web
        bakfile_basename: backup-website1-sql
        bakfile_owner: root
        bakfile_group: root
        bakfile_mode: 0600

This minimal example shows there are three top level mandatory sections:

  • global: Global options for rubackup
  • schedules: Defines one or more schedules which are creation and retention policies
  • entries: List of each backup to create with specific options for these backups

Global options

There are not many global options. The ones used here say monthly backups are created every Monday and monthly backups are created the first of each month.

Schedules

You can have a single schedule which you reuse for all entries or you can have multiple schedules. You must give a name to each schedules, in this case they are myschedule_long_term and myschedule_short_term. For example in the myschedule_long_term schedule backups will be created everyday and then it will keep the last 7 days, 4 weeks (the last 4 mondays) and the last 12 months (first of the month for the last 12 months). Other backups will be deleted.

Entries

The entries are the biggest section as it lists all the backups to manage and all their specific options. Entries have options in common. For example all entries must have a backup_schedule as rubackup needs to know what the creation / retention policy is for all backups. The enabled option can be set to false to temporarily disable a backup without having to delete its configuration. Most backups (all backups that generate a local file) require bakfile_dir to specify where the backup will created, bakfile_basename to specify the name to give to the backup files (the date and extensions will be added automatically to the basename), and the ownerships and permissions. Each backup entry has a backup_opts section where the options depend on the module type being used.

You can see the backup module which produces tar archives requires a list of files and directories to be included and excluded in the archive. The includes option must be a list even when there is just one item, hence the lines must start with a dash. The excludes option is optional and is also a list. The command_opts option is also not mandatory and it can be used to provide extra arguments that must be passed to the tar command line.

The mysql backup module requires different options which are the database connection details. The dbhost option is mandatory as 127.0.0.1 will be used by default if it is not specified. The password is also optional as the mysqldump command won’t need a password if it is already in ${HOME}/.my.cnf.

These two modules support a compress_prog option which you can use to set the compression program to run in the pipeline (can be lzop, lz4, gzip, bzip2, xz). The xz program will be used by default if it is not specified. They also both support compress_opts which is not mandatory and which can be used to provide extra command arguments to the compression program. This can be used to change the compression speed/performance.

Results

The previous configuration would produce the following type of backup files. The date in the YYYYMMDD format will automatically be added. It is important as this date in the file name is what rubackup uses to determine when backups need to be deleted. Also a checksum file is automatically created and deleted when backups expire.

/backup/backup-web/backup-website1-www-20141201.tar.gz
/backup/backup-web/backup-website1-www-20141201.tar.gz.sha256
/backup/backup-web/backup-website1-www-20141202.tar.gz
/backup/backup-web/backup-website1-www-20141202.tar.gz.sha256
/backup/backup-web/backup-website1-sql-20141201.sql.xz
/backup/backup-web/backup-website1-sql-20141201.sql.xz.sha256
/backup/backup-web/backup-website1-sql-20141202.sql.xz
/backup/backup-web/backup-website1-sql-20141202.sql.xz.sha256

Global section

Here is an example which shows all the options currently supported in the global section of the configuration:

---
global:
    day_of_week: Sun
    day_of_month: 1
    notify_type: ModuleNotifyAwsSes
    notify_opts:
        ses_media: my_ses_media
    path_extra:
        - /sbin
        - /usr/sbin
        - /usr/local/sbin
aws_ses_medias:
    my_ses_media:
        awsregion: us-west-2
        mailsrc: noreply@company.com
        maildst: administrator@company.com
        accesskey: access_key_ses
aws_access_keys:
    access_key_ses:
        public: 'Your_AWS_Access_Key_Here_PUBLIC'
        secret: 'Your_AWS_Access_Key_Here_SECRET'

The path_extra option allows you to provide a list of directories where commands required to perform backups are located. You can use it when directories are not in your PATH environment variable. The notify_type and notify_opts options let you use a notification module which also has its own configuration and may refer to resources such as AWS Access Keys.

Multiple configuration files

Here is an example which shows how to use multiple configuration files with rubackup. This is more flexible and useful in environments with many modules and backup entries especially when part of the configuration is identical on multiple systems. By default rubackup will attempt to load all yaml files in /etc/rubackup.d/ hence it is recommended to use this default configuration directory.

The first file is /etc/rubackup.d/010-global.yaml and it contains the global section:

---
global:
    day_of_week: Sat
    day_of_month: 1
    path_extra:
        - /sbin
        - /usr/sbin

Here is another file: /etc/rubackup.d/020-schedules.yaml to define the schedules:

---
schedules:
    my_sched_tarballs:
        daily: 7
        weekly: 4
        monthly: 2
    my_sched_databases:
        daily: 30
        weekly: 52
        monthly: 36

Now we can have one file such as /etc/rubackup.d/110-website1.yaml for two backup entries related to a particular project.

---
entries:
    website1-tar:
        enabled: true
        backup_type: ModuleBackupTarball
        backup_opts:
            includes:
                - /var/www/website1
                - /etc/httpd/conf.d/website1.conf
        backup_schedule: my_sched_tarballs
        bakfile_dir: /backup/backup-web
        bakfile_basename: backup-website1-www
        bakfile_owner: root
        bakfile_group: root
        bakfile_mode: 0600
    website1-sql:
        enabled: true
        backup_type: ModuleBackupMysqldp
        backup_opts:
            dbhost: 127.0.0.1
            dbuser: website1
            dbname: website1
            dbpass: 'SqlPassForWebSite1'
        backup_schedule: my_sched_databases
        bakfile_dir: /backup/backup-web
        bakfile_basename: backup-website1-sql
        bakfile_owner: root
        bakfile_group: root
        bakfile_mode: 0600

And we can have another file such as /etc/rubackup.d/120-website2.yaml for another project.

---
entries:
    website2-tar:
        enabled: true
        backup_type: ModuleBackupTarball
        backup_opts:
            includes:
                - /var/www/website2
                - /etc/httpd/conf.d/website2.conf
        backup_schedule: my_sched_tarballs
        bakfile_dir: /backup/backup-web
        bakfile_basename: backup-website2-www
        bakfile_owner: root
        bakfile_group: root
        bakfile_mode: 0600
    website2-sql:
        enabled: true
        backup_type: ModuleBackupMysqldp
        backup_opts:
            dbhost: 127.0.0.1
            dbuser: website2
            dbname: website2
            dbpass: 'SqlPassForWebSite2'
        backup_schedule: my_sched_databases
        bakfile_dir: /backup/backup-web
        bakfile_basename: backup-website2-sql
        bakfile_owner: root
        bakfile_group: root
        bakfile_mode: 0600