Restoring Backups

Table of Contents
Intro
Prerequisites
Non-GreenArrow Files
Restoration Procedure
Final Testing

Intro

This document describes how to restore a GreenArrow installation from backups. Please contact GreenArrow technical support if you would rather have GreenArrow restore your GreenArrow installation.

Prerequisites

The restoration procedure described on this page has a number of prerequisites:

A good set of backups created by GreenArrow’s unmanaged backup script.

Each time a backup is taken, the results get written to the backup directory’s greenarrow-backup.log file. The backup directory is /var/backup by default.

If the backup completed without any detected errors, then the last line of the log will read:
```
The backup is finished. No errors were detected.
```
If any errors were detected, then the last line of the log will read:
```
Error(s) were detected. See above.
```
The backup script will output the following text immediately after each error that it detects:
```
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! WARNING !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
There was an error running backups. Please review the above output for
details and contact GreenArrow technical support if you have any questions.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! WARNING !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
```
An experienced Linux systems administrator who can execute the restoration procedure.

The backups must be of GreenArrow Engine 4.1.144-0 or later.

Here’s a command you can run to check what version was backed up to the /var/backup/ directory:

grep greenarrow-engine /var/backup/greenarrow-packages.txt | grep -v greenarrow-engine-common || echo "I was unable to determine what version of GreenArrow Engine is installed."

Alternatively, if you haven’t run GreenArrow’s unmanaged backup script yet, you can check what version is currently installed by running:

rpm -q greenarrow-engine 2> /dev/null || dpkg -l greenarrow-engine 2> /dev/null || echo "GreenArrow Engine is either not installed, or is an old release which GreenArrow technical supported is needed to restore."

If you’re running GreenArrow Studio, you must be on a version that’s recent enough to work with this process. If the /var/sendstudio/ directory exists, that means that you’re running an older release whose restoration procedure is not covered by this document. If this directory is missing, then you’re on a recent enough version.
The server that you’re restoring to must run a Linux distribution from the same family as the server where backups were taken from. In other words, if your backup is from Red Hat based distribution, like CentOS, then the restore must be made to a Red Hat Based distribution. If your backup is from a Debian-based distribution, such as Ubuntu, then the restore must be to a Debian-based distribution.
The server should have enough disk space to hold the restored data. If the specs are identical to or greater than the server that the backup was taken on, and you didn’t have any disk space issues prior to needing the restore, then you’re covered. Otherwise, you can get an approximation of how much disk space will be needed to restore GreenArrow’s backups in two steps:
1. cd into the directory that your backups are stored in, and run the gzip -l greenarrow-files.tar.gz command to estimate how much space will be required for non-PostgreSQL files.
  
  The uncompressed column’s value will show approximately how much disk space, in bytes, will be used by these files.
2. cd into the directory that your backups are stored in, and run the /var/hvmail/postgres/default/bin/pg_restore greenarrow.dump | wc -c command to estimate how much disk space, in bytes, will be required for PostgreSQL.
The figures that are provided by the above commands do not account for a number of things, including GreenArrow’s packages, the disk space required by PostgreSQL’s indexes, and deferred email, so you should allow for significantly more disk space usage than they indicate you need. You might allow for double the disk space, for example.

GreenArrow technical support can perform a restore using just the files created by our unmanaged backup script, so if you’re missing any of this document’s other prerequisites, contact us for details.

Non-GreenArrow Files

GreenArrow’s unmanaged backup script backs up the /etc/ and /var/spool/cron/ directories to the operating-system-files.tar.gz file. These directories are not normally needed to perform a GreenArrow restore, but we found that these backups are sometimes useful. You’re welcome to use this file, but this document doesn’t go into details on how to do so.

Restoration Procedure

Some of the steps below are Linux distribution specific. For the sake of brevity we refer to Red Hat based Linux distributions, such as CentOS, Red Hat Enterprise Linux or Scientific Linux as “CentOS”, and Debian-based distributions, such as Ubuntu and Debian to “Debian”.

Here’s how to restore GreenArrow from backups:

Log in to the server that you wish to restore to as the root user using the bash shell.
Install packages that are used by the restore process:

On CentOS systems, run:
```
yum install ruby
```
On Debian systems, run:
```
apt-get install ruby wget
```
Copy the files created by GreenArrow’s unmanaged backup script to a location that’s accessible to the operating system, such as /var/backup/, then cd into that directory.
View the greenarrow-dereferenced-symlinks.txt file.

If the file is empty, then move onto the next step.

If the file lists any symlinks, then decide whether you wish to re-create them. The symlinks that are listed in this file do not exist on a default GreenArrow installation but were present at the time that the backup was taken. If you do not re-create the symlinks, then they will be dereferenced later when we run tar to restore them. tar will restore the backed up files either way. The decision of whether to create these symlinks just impacts where the data that the symlinks pointed to gets written.
Check the postgresql-tablespaces.txt file for any tablespaces which specify a location.

Here’s an example of what this file might look like. In this example, the tablespace named “data2” is the only one which has a location (/var/hvmail/postgres/default/data2) specified:
```
                     List of tablespaces
    Name    |   Owner    |              Location
------------+------------+------------------------------------
 data2      | greenarrow | /var/hvmail/postgres/default/data2
 pg_default | postgres   |
 pg_global  | postgres   |
(3 rows)
```
If there are no tablespaces which specify a location, then you can move onto the next step.

If there are any tablespaces which specify a location, then re-create their directories, and set their owner equal to the hvpostgres user and group, with 0700 permissions. For example:
```
mkdir /var/hvmail/postgres/default/data2
chown hvpostgres:hvpostgres /var/hvmail/postgres/default/data2
chmod 0700 /var/hvmail/postgres/default/data2
```
/var/hvmail/postgres/default is normally a symlink. If it needs created during this step, then you can check on where it pointed to by viewing the postgresql-symlink.txt file in your backup directory. If the file is missing, then /var/hvmail/postgres/default was not a symlink.
Record your repository access key. This will be needed for the next step:
```
cat greenarrow-repository-access-key.txt
```
Complete the Prerequisites section of our Installation Guide. If you get to the “Install GreenArrow” step in the Installation Guide, you’ve gone too far.
Install GreenArrow’s packages:
1. cd into the directory that contains the file created by GreenArrow’s unmanaged backup script.
2. Run the hvmail_reinstall_packages command, and follow the prompt(s):
```
./hvmail_reinstall_packages greenarrow-packages.txt
```
hvmail_reinstall_packages is idempotent. If it encounters an error that you’re able to correct, then it’s safe to run it again.
GreenArrow packages make a number of changes to environmental settings. To activate the changes, either log out, then back into the server or run the following commands. If you’re logged into the server in multiple windows, then you’ll need to complete this step in each window:
```
. /etc/profile.d/greenarrow-engine.sh
. /etc/profile.d/greenarrow-ruby.sh
. /etc/profile.d/greenarrow-studio.sh
. /etc/profile.d/greenarrow-passenger.sh
```
Complete the Tune the operating system section of our Installation Guide.
Put the system into a blank state ready for a restore:
```
/var/hvmail/libexec/hvmail_clear_data_for_restore
```

Restore GreenArrow’s data files:

tar xzfp greenarrow-files.tar.gz -C /
mv /var/hvmail/postgres/default{,.restored-config-files}

If the server that you’re restoring to has a different amount of RAM than the server that backups were taken on, then:
1. Tune the /var/hvmail/control/postgres.conf file as described in the documentation on tuning PostgreSQL memory usage. PostgreSQL is not running yet, so skip the steps that involve restarting and testing PostgreSQL.
2. Adjust the size of GreenArrow’s RAM and Bounce queues as described in the documentation on changing the amount of RAM allocated to mail queues. The GreenArrow queues are not started yet, so skip the hvmail_init restart step which restarts the queues.
If you previously tuned PostgreSQL’s memory settings and this required updating the kernel.shmmax and kernel.shmall sysctl parameters on the backed up system, also update those parameters on the new system or re-tune PostgreSQL. See the documentation on tuning PostgreSQL memory usage.
Restore PostgreSQL. Depending on the size of the backup, and the speed of your hardware, this step could take anywhere from minutes to hours to complete:
```
hvmail_ini_postgres --restore-from-backup .
```
If you receive an unable to connect to the database error, then this may mean that not enough memory could be allocated. If this happens to you, then try tuning sysctl as described in the documentation on tuning PostgreSQL memory usage.
Prevent Studio from resuming sends automatically when we start GreenArrow:
```
touch /service/hvmail-studio-worker/down
```
Start GreenArrow:
```
service greenarrow start
```

Configure the /etc/resolv.conf file so that the only nameserver line is set to 127.0.0.1. For example:

echo "nameserver 127.0.0.1" > /etc/resolv.conf

Also, verify that different nameservers aren’t configured elsewhere:

On CentOS systems, run:

find /etc/sysconfig/network-scripts/ifcfg-* /etc/sysconfig/network -type f | xargs grep 'DNS=' | grep -v 'PEERDNS=' | grep -v 127.0.0.1

On Debian systems, run:

grep dns-nameservers /etc/network/interfaces

If you use GreenArrow Studio, then go to Studio’s web interface, and cancel any campaigns that you do not wish to be sent. When making this decision, think about which campaigns may have finished sending after the backup that you just restored was taken. The idea here is to avoid sending any campaigns twice.
If you’re not using GreenArrow Studio’s autoresponders then skip this step.

If you use GreenArrow Studio’s autoresponders, then decide what to do with the autoresponders which were in the queue to be sent before the GreenArrow server went offline. These are autoresponders which may have been sent already, but you don’t know for certain. You have two options:

The first option is to let the autoresponders get sent out after the restore, potentially as duplicates of what was previously sent. If you select this option, then you can move onto the next step.

The second option is to NOT allow the autoresponders to be sent out. This is the safer option, in terms of not annoying subscribers with duplicated emails, but the downside is that it may cause some subscribers not to receive any copies of autoresponder(s) which they were scheduled to be sent.

If you choose the second option, then you’ll need to delete some rows from PostgreSQL’s s_autoresponder_reactions table. Here’s how to do that:
1. Login to PostgreSQL:
```
/var/hvmail/postgres/default/bin/psql -U greenarrow
```
1. Use a SELECT TIMESTAMP query to enter the time and date that the event that caused the need for a restore occurred in UTC, and verify that PostgreSQL understands what you entered. For example:
```
SELECT TIMESTAMP 'January 1, 2050 12:00 PM UTC';
```
  The above command did not make any changes. Its purpose is to verify that you have the correct time and date syntax.
2. Begin a transaction:
```
BEGIN;
```
3. Delete all rows in the s_autoresponder_reactions table that have their delayed_until value set to less than or equal to the time and date that the event that caused the need for a restore occurred in UTC. For example, if your GreenArrow server went offline on January 1, 2050, at 12:00 PM UTC, you’d run:
```
DELETE FROM s_autoresponder_reactions WHERE delayed_until <= 'January 1, 2050 12:00 PM UTC';
```
4. If the steps above completed without error, commit the transaction. This applies changes:
```
COMMIT;
```
  If there were any issues the steps above, rollback the transaction. This will undo any changes that were made to PostgreSQL since the BEGIN; query was run:
```
ROLLBACK;
```
5. Log out of PostgreSQL:
```
\q
```
Check if the /var/hvmail/control/UPGRADE_BLOCKER.txt file exists:
```
cat /var/hvmail/control/UPGRADE_BLOCKER.txt
```
This file is only present on GreenArrow installations where non-standard steps are required to perform updates of GreenArrow’s software.

If the file is missing, then you can move onto the next step.

If the file is present, then contact GreenArrow technical support to have whatever extra steps are needed completed.

If you use GreenArrow Studio, then resume its sends:

rm -f /service/hvmail-studio-worker/down
svc -u /service/hvmail-studio-worker

Final Testing

Verify that some common misconfigurations are not present. Each test gets a simple pass/fail result:
```
hvmail_check_config
```
Verify that all services are running normally. Any services with an abnormal state will be shown in red:
```
service greenarrow status
```
The hvmail-qmail-smtpd3 service is down by default. All other services should be reported as up unless you shut them down intentionally.
Send yourself a test email (replacing [email protected] with your actual email address):
```
date | /var/hvmail/bin/mailsubj "GreenArrow Test Message" [email protected]
```

Congratulations, you’re finished restoring GreenArrow! As a next step, we recommend following up with GreenArrow technical support to request that we review the restored GreenArrow installation for issues.