Server Migrations

Table of Contents
Intro
Prerequisites
Non-GreenArrow Files and Services
Migration Procedure
Post-Migration Configuration
Final Testing
Disable Old Server

Intro

This document describes how to migrate a GreenArrow installation from one server to another. Please contact GreenArrow technical support if you would rather have GreenArrow migrate your GreenArrow installation.

Prerequisites

The migration procedure described on this page has some prerequisites:

An experienced Linux systems administrator who can execute the migration procedure.

The server that you’re migrating from must be running GreenArrow Engine 4.1.165-0 or later.

Here’s a command you can run to check what version is installed:

rpm -q greenarrow-engine 2> /dev/null || dpkg -l greenarrow-engine 2> /dev/null || echo "GreenArrow Engine is either not installed, or is an old release which GreenArrow technical supported is needed to migrate."

If you’re running GreenArrow Studio, you must be on a version that’s recent enough to work with this process. If the /var/sendstudio/ directory exists, that means that you’re running an older release whose migration procedure is not covered by this document. If this directory is missing, then you’re on a recent enough version.
The server that you’re migrating to must run a Linux distribution from the same family and hardware platform as the server that you’re migrating from. Here are two examples:
1. If your existing server runs CentOS 7 x86_64, then the migration must be made to a Red Hat x86_64 based distribution, such as CentOS 7 x86_64 or CentOS 6 x86_64.
2. If your existing server runs Ubuntu 14.04 x86_64, then the migration must be made to a Debian x86_64 based distribution, such as Ubuntu 16.04 x86_64 or Debian 9 x86_64.
If you’re planning to migrate to an Ubuntu 16.04, Ubuntu 18.04 or Debian 9 server, then be aware that those Linux distributions do not support PostgreSQL 8.3. New GreenArrow installations use PostgreSQL 9.5, which is supported, but some older GreenArrow installations still run 8.3.

You can check which version of PostgreSQL your existing server is running by checking where the /var/hvmail/postgres/default symlink points to. If it points to /var/hvmail/postgres/9.5, then you’re running PostgreSQL 9.5. If it points to /var/hvmail/postgres/8.3, then you’re running PostgreSQL 8.3:
```
ls -l /var/hvmail/postgres/default
```
Please contact GreenArrow technical support if you’d like us to migrate you from PostgreSQL 8.3 to 9.5.

Our other supported Linux distributions do not have this limitation.
The server should have enough disk space to hold the migrated data. The vast majority of GreenArrow’s data is located within the /var/hvmail directory and the migrated data will occupy approximately the same amount of disk space after the migration as it did before.

If the specs of the destination server are identical to or greater than the server that you’re migrating from, and you didn’t have any disk space issues before the migration, then you’re covered.
The /var/hvmail/control/UPGRADE_BLOCKER.txt file must not exist. This file is only present on GreenArrow installations where non-standard steps are required to perform updates of GreenArrow’s software. Those same non-standard steps will need to be performed towards the end of the migration. Contact GreenArrow technical support to arrange to have those steps applied.

Non-GreenArrow Files and Services

These migration instructions do not take into account any non-GreenArrow files or services that you may also wish to migrate.

Migration Procedure

Some of the steps below are Linux distribution specific. For the sake of brevity, we refer to Red Hat based Linux distributions, such as CentOS, Red Hat Enterprise Linux or Scientific Linux as “CentOS”, and Debian-based distributions, such as Ubuntu and Debian as “Debian”.

Here’s how to migrate GreenArrow to a new server. All steps should be completed on the new server unless otherwise indicated:

If you’re adding, removing or changing any IPs as part of the migration, then consider configuring their DNS records and feedback loop registrations in advance. Instructions for adding new IPs are here.
Authorize the new server’s root account to SSH into the old server as root.
Log in to the new server as the root user using the bash shell.
Install packages that are used by the migration process:

On CentOS systems, run:
```
yum install ruby
```
On Debian systems, run:
```
apt-get install ruby wget
```
Make an entry in the new server’s /etc/hosts file, pointing the oldserver hostname to an IP on the old server that has an SSH server running. For example:
```
echo "1.2.3.4 oldserver" >> /etc/hosts
```
This entry is needed because we’re going to run some commands later on which connect to oldserver.
Verify that you can SSH into the old server from the new server by running:
```
ssh oldserver
```
If a non-default port is used for ssh then you can tell the SSH client to use it by creating or updating /root/.ssh/config. For example:
```
Host oldserver
  Port 2222
```
Take a backup of the old server. A migration normally only uses some of the files that get created by a backup, but we recommend that you hold onto the entire backup directory to increase the odds of being able to recover if something goes wrong during the migration. For extra safety, you may want to copy this directory to some media that’s not attached to either server.
Copy the following files from the backup directory to the /var/migrate/ directory on the new server:
1. greenarrow-packages.txt
2. greenarrow-repository-access-key.txt
3. hvmail_reinstall_packages
4. postgresql-tablespaces.txt
Check the /var/migrate/postgresql-tablespaces.txt file for any tablespaces which specify a location.

Here’s an example of what this file might look like. In this example, the tablespace named “data2” is the only one which has a location (/media/data2) specified:
```
                     List of tablespaces
    Name    |   Owner    |              Location
------------+------------+------------------------------------
 data2      | greenarrow | /media/data2
 pg_default | postgres   |
 pg_global  | postgres   |
(3 rows)
```
If there are no tablespaces which specify a location, then you can move onto the next step.

If there are any tablespaces which specify a location, then re-create their directories on the new server. For example:
```
mkdir /media/data2
```
Also, rsync each of the directories identified above from the old server to the new server. You should both run each rsync invocation now, and record it so that it can be run again later after hvmail_server_migration has run to completion. For example:
```
rsync -a --delete oldserver:/media/data2/ /media/data2
```
View the contents of the /var/migrate/greenarrow-repository-access-key.txt file. This key is needed for the next step.
Complete the Prerequisites section of our Installation Guide. If you get to the “Install GreenArrow” step in the Installation Guide, you’ve gone too far.
Install GreenArrow’s packages on the new server:
```
/var/migrate/hvmail_reinstall_packages /var/migrate/greenarrow-packages.txt
```
If the above command encounters an error that you’re able to correct, then it’s safe to run it again.
GreenArrow packages make some changes to environmental settings. To activate the changes, either log out, then back into the server or run the following commands. If you ’re logged into the server in multiple windows, then you’ll need to complete this step in each window:
```
. /etc/profile.d/greenarrow-engine.sh
. /etc/profile.d/greenarrow-ruby.sh
. /etc/profile.d/greenarrow-studio.sh
. /etc/profile.d/greenarrow-passenger.sh
```
Run the migration script for the first time. This can safely be done while GreenArrow is still running on the old server:
```
/var/hvmail/libexec/hvmail_server_migration --no-exit-on-error
```
This migration script could take anywhere from minutes to hours to run, depending on how much data there is to transfer. It’s safe to run the script multiple times, as long as only one instance is running at any given point in time.

rsync is used for most of the data transfer, so all other things being equal, the closer together two invocations of this script are to each other, the faster the second invocation will run.

Stop GreenArrow on the old server.

service greenarrow stop
chkconfig greenarrow off 2> /dev/null || update-rc.d greenarrow disable

Run the migration script again. This will synchronize any files which changed since the last time the script was run:
```
/var/hvmail/libexec/hvmail_server_migration
```
If the migration script completes without error, then its last line of output will read No errors were detected. If the last line contains anything else, review the remaining output for issues. Once any issues have been corrected, it’s safe to re-run hvmail_server_migration.
If you identified any additional rsync invocations to run while you were inspecting postgresql-tablespaces.txt earlier, run them now.
If there are any IP addresses to move from the old server to the new one, move them now.
Import messages into the new server’s disk queue:
```
find /var/hvmail/qmail-disk/queue/old/mess -type f | /var/hvmail/libexec/hvmail_queue_merge /var/hvmail/qmail-disk/queue/old /var/hvmail/qmail-disk/queue
```
Depending on how many messages there are to import and how fast your storage is, this step could take anywhere from seconds to hours to complete. 5-20 minutes is typical. You can monitor the progress by checking how many files remain in the /var/hvmail/qmail-disk/queue/old/mess directory:
```
find /var/hvmail/qmail-disk/queue/old/mess -type f | wc -l
```
Verify that all messages were imported in the previous step. If they were, then the following command will output 0:
```
find /var/hvmail/qmail-disk/queue/old/mess -type f | wc -l
```
Delete what’s left of the /var/hvmail/qmail-disk/queue/old directory
```
rm -rf /var/hvmail/qmail-disk/queue/old
```

Start GreenArrow on the new server.

On CentOS systems, run:

systemctl start greenarrow.service 2> /dev/null || service greenarrow start
chkconfig greenarrow on

On Debian systems, run:

service greenarrow start
update-rc.d greenarrow defaults
sed -i 's/^deb /#deb /' /etc/apt/sources.list.d/greenarrow.list

Configure the /etc/resolv.conf file so that the only nameserver line is set to 127.0.0.1. For example:
```
echo "nameserver 127.0.0.1" > /etc/resolv.conf
```
Verify that different nameservers aren’t configured outside of /etc/resolv.conf. If the commands below return any output, then that suggests that there’s a configuration which should be updated:

On CentOS systems, run:
```
find /etc/sysconfig/network-scripts/ifcfg-* /etc/sysconfig/network -type f | xargs grep 'DNS=' | grep -v 'PEERDNS=' | grep -v 127.0.0.1
```
On Debian systems, run:
```
grep dns-nameservers /etc/network/interfaces
```
Check if the /var/hvmail/control/UPGRADE_BLOCKER.txt file exists:
```
cat /var/hvmail/control/UPGRADE_BLOCKER.txt
```
This file is only present on GreenArrow installations where non-standard steps are required to perform updates of GreenArrow’s software.

If the file is missing, then you can move onto the next step.

If the file is present, then contact GreenArrow technical support to have those non-standard steps applied.
Complete the Performance Tuning section of our Installation Guide.

Post-Migration Configuration

If you’re adding, removing or changing any IPs as part of the migration, then they should be added into GreenArrow’s configuration now. Instructions for adding new IPs are here.
If there are any other configuration changes desired as part of this migration, make them now.

Final Testing

Verify that some common misconfigurations are not present. Each test gets a simple pass/fail result:
```
hvmail_check_config
```
Verify that all services are running normally. Any services with an abnormal state are shown in red:
```
service greenarrow status
```
The hvmail-qmail-smtpd3 service is down by default. All other services should be reported as up unless you shut them down intentionally.
Send yourself a test email (replacing [email protected] with your actual email address):
```
date | /var/hvmail/bin/mailsubj "GreenArrow Test Message" [email protected]
```
Remove the entry that you made earlier for oldserver in the new server’s /etc/hosts file.

Congratulations, you’re finished migrating GreenArrow to the new server! As a next step, we recommend following up with GreenArrow technical support to request that we review the migrated GreenArrow installation for issues.

Continue to the next section to prevent the old server from coming back online, and attempting to send duplicate emails and/or use the same license key in two locations at the same time.

Disable Old Server

Remove GreenArrow’s licenses from the old server:

rm -f /var/hvmail/control/license_key /var/hvmail/control/bounce.boogie_studio_api_license_key

Delete the queued messages on the old server:

/usr/bin/find /var/hvmail/qmail-ram/savedqueue -type f -delete
/usr/bin/find /var/hvmail/qmail-bounce/savedqueue -type f -delete
cd /var/hvmail/qmail-disk/queue/
/usr/bin/find info -type f -delete
/usr/bin/find mess -type f -delete
/usr/bin/find remote -type f -delete
/usr/bin/find intd -type f -delete
/usr/bin/find local -type f -delete
/usr/bin/find todo -type f -delete
/usr/bin/find bounce -type f -delete