GreenArrow Documentation

Installation Guide

Intro

This document describes how to install and configure GreenArrow Engine and Studio. It’s targetted towards experienced Linux systems administrators. Please contact GreenArrow technical support if you encounter any unexpected issues with these instructions, or if would rather have GreenArrow perform your installation.

Prerequisites

Provision a server

We recommend running GreenArrow on a dedicated server or VM. It’s possible for GreenArrow to co-exist with other applications, but that may introduce conflicts which are not addressed by this document.

The server should run one of the following Linux distributions:

  1. CentOS, Red Hat Enterprise Linux or Scientific Linux version 7.x x86_64, 6.x x86_64 or 6.x x86. These are referred to collectively as “CentOS” in these instructions for the sake of brevity.
  2. Ubuntu 18.04, 16.04 or 14.04 x86_64 or Debian 9.x or 7.x x86_64. These are referred to collectively as “Debian” in these instructions for the sake of brevity.

We recommend using CentOS 7.x x86_64, because that’s the distribution that’s most commonly used by our customers, and therefore, the most widely tested and optimized against.

If you’re running GreenArrow with less than 4GB of RAM, then it will require tuning that’s outside the scope of this document. Please contact GreenArrow technical support if you’re in this situation.

Our Hardware Requirements document contains more server sizing details.

Log in as the root user using the bash shell

Install prerequisite packages

In CentOS, run:

yum -y groupinstall base

In Debian, run:

apt-get -y install curl apt-transport-https

Disable SELinux (CentOS only)

GreenArrow requires that SELinux be disabled. Here’s an example sequence of commands for disabling it in most CentOS installations. Note that this will reboot your server:

sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
sed -i 's/SELINUX=permissive/SELINUX=disabled/' /etc/selinux/config
reboot

If you’d like more details on this topic, then see RedHat’s SELinux documentation.

Configure time synchronization

GreenArrow works better with a stable system clock. We recommend using ntpd to keep your server’s time synchronized with NTP servers.

In CentOS, run:

yum -y install ntp
ntpdate pool.ntp.org
chkconfig ntpd on
service ntpd start

In Debian, run:

apt-get -y install ntp ntpdate
ntpdate pool.ntp.org
update-rc.d ntp defaults
service ntp start

Configure filesystems

We recommend that your filesystems hosting GreenArrow data (particularly the message queue) be configured in the following ways for best performance:

  • Use an XFS or Ext4 filesystem
  • Use a block size of 4096 bytes
  • Allocate one inode for every block

Here’s an example of creating an XFS filesystem that meets the above criteria. The inode recommendation is met by mkfs.xfs’s default values:

mkfs.xfs -b size=4096 -L /media/disk1 /dev/sdb1

Here’s an example of creating an Ext4 filesystem that meets the above criteria:

mkfs -t ext4 -j -b 4096 -i 4096 -L /media/disk1 /dev/sdb1

GreenArrow will install into /var/hvmail. If /var is not on the correct partition for installation, then you should create symlinks out to the directory where you’d like data to be stored. You can also place different parts of GreenArrow on different disks if you’d like to split the load between them. Here are the directories that are safe to put on their own disks:

Directory Contents
/var/hvmail All of GreenArrow’s data
/var/hvmail/qmail-disk/queue GreenArrow’s message queue
/var/hvmail/postgres GreenArrow’s internal PostgreSQL database
/var/hvmail/log GreenArrow’s log files and searchable databases of log data

For example, to create a symlink to relocate /var/hvmail to /media/disk1/hvmail run the following:

mkdir -p /media/disk1/hvmail
ln -s /media/disk1/hvmail /var/hvmail

This should be done before GreenArrow is installed.

Free up ports for GreenArrow

There are a number of ports that GreenArrow will bind to for network services. We recommend ensuring that these ports are free prior to installing GreenArrow.

TCP port 25 on all IP addresses – SMTP

By default, GreenArrow will want to listen on port 25 for incoming connections on all IPs. Most Linux systems ship with a default MTA of Postfix or Sendmail also listening on port 25.

We recommend you do one of the following:

  1. Disable the existing MTA

    If another MTA is already installed, it’s usually Sendmail, Exim or Postfix. The following command will check whether any of these are running:

    ps axf | grep -P 'sendmail|exim|postfix' | grep -v grep

    We recommend against uninstalling the other MTA since a subsequent package update could reinstall that MTA, and configure it to automatically start and listen on port 25. Instead, we recommend configuring the MTA not to start.

    Here’s an example of how to do this with the copy of Postfix that’s often installed by default in CentOS:

    service postfix stop && chkconfig postfix off

    Here’s an example of how to do this with an Exim installation running in Debian:

    service exim4 stop && update-rc.d exim4 disable

  2. Configure the existing MTA to listen on a port other than 25

    This will allow the existing MTA to continue to send mail, but allow GreenArrow to receive email.

  3. Configure GreenArrow to bind to a port/IP that does not conflict with the existing MTA

    It’s possible to have the existing MTA listening on port 25 on one IP and GreenArrow listen on port 25 on another IP. If you configure GreenArrow not to listen on port 25, then it will not be able to receive incoming emails, including bounces and spam complaints from the Internet. The port/IP GreenArrow binds its SMTP service to can be configured by updating the /var/hvmail/control/smtp.ipaddr and /var/hvmail/control/smtp.port files.

TCP port 587 on all IP addresses – SMTP submission

By default, GreenArrow will listen on TCP port 587 on all IP addresses.

If a service conflicts with this port, it’s usually another MTA such as Postfix.

Instruction on how to change GreenArrow’s default SMTP submission port can be found here.

TCP port 110 on all IP addresses – POP3

By default, GreenArrow will listen on TCP port 110 on all IP addresses.

Instruction on how to change GreenArrow’s default POP3 port can be found here.

TCP ports 80 and 443 on all IP addresses – HTTP and HTTPS

By default, GreenArrow will listen on TCP ports 80 and 443 on all IP addresses.

Instruction on how to change GreenArrow’s default HTTP and HTTPS ports can be found here.

We recommend being careful about changing these ports because click and open tracking links which use non-default ports can trigger spam filters.

TCP and UDP Port 53 on 127.0.0.1 – DNS

GreenArrow provides a recursive DNS resolver on the system. This is because GreenArrow creates a high volume of DNS queries as part of delivering a high volume of mail. Providing a built-in high-volume DNS resolver eliminates an external dependency and potential speed bottleneck.

If you have BIND or another DNS server already bound to 127.0.0.1 or all IPs, please disable it or configure it to bind to IP(s) other than 127.0.0.1. It’s okay to have another DNS server binding to port 53 of IPs other than 127.0.0.1.

Make firewall openings

The Firewall Configuration page describes GreenArrow’s requirements for incoming and outgoing connections.

Configure GreenArrow repository access

You will need your GreenArrow repository access key, which is a 16-character hexadecimal number. This is provided with your license.

Run these commands in CentOS, and follow the prompts:

curl https://git.drh.net/pub/setup_greenarrow_yum_repository > /root/setup_greenarrow_yum_repository
/bin/bash /root/setup_greenarrow_yum_repository

Run these commands in Debian, and follow the prompts:

curl https://git.drh.net/pub/setup_greenarrow_apt_repository > /root/setup_greenarrow_apt_repository
/bin/bash /root/setup_greenarrow_apt_repository

If you’re running RedHat Enterprise Linux 7.x, you’ll also need to run the following command:

sed -i 's/$releasever/7/' /etc/yum.repos.d/greenarrow.repo

If you’re performing a migration or restoring from backups, stop here, and return to the document that you sent you here.

If you’re performing a fresh install, continue on to the next step.

Install GreenArrow

The following steps install GreenArrow:

Install the packages

Now let’s install Engine and Studio. There are some shared components, so you’ll need to install both, even if you’re only using Engine.

Run these commands in CentOS:

yum -y install --enablerepo=greenarrow greenarrow-engine-common && \
yum -y install --enablerepo=greenarrow greenarrow-engine greenarrow-studio && \
chkconfig greenarrow on

Run these commands in Debian:

apt-get update && \
apt-get -y install greenarrow-engine-common && \
apt-get -y install greenarrow-engine greenarrow-studio && \
ln -sf /var/hvmail/libexec/greenarrow.init.sh /etc/init.d/greenarrow && \
update-rc.d greenarrow defaults && \
sed -i 's/^deb /#deb /' /etc/apt/sources.list.d/greenarrow.list

Update Environmental Settings

GreenArrow packages make a number of changes to environmental settings. To activate the changes, either log out, then back into the server, or run the following commands. If you’re logged into the server in multiple windows, then you’ll need to complete this step in each window:

. /etc/profile.d/greenarrow-engine.sh
. /etc/profile.d/greenarrow-ruby.sh
. /etc/profile.d/greenarrow-studio.sh
. /etc/profile.d/greenarrow-passenger.sh

Initialize the database

Run this command:

hvmail_ini_postgres

Start GreenArrow

Run this command:

service greenarrow start

Configure /etc/resolv.conf to use GreenArrow’s DNS server

  1. Configure the /etc/resolv.conf file so that the only nameserver line is set to 127.0.0.1. For example:

    nameserver 127.0.0.1

  2. Verify that different nameservers aren’t configured elsewhere:

    1. On CentOS systems:

      find /etc/sysconfig/network-scripts/ifcfg-* /etc/sysconfig/network -type f | xargs grep 'DNS=' | grep -v 'PEERDNS=' | grep -v 127.0.0.1

    2. On Debian systems:

      grep dns-nameservers /etc/network/interfaces

Required GreenArrow Engine configuration

License key

Save the license key that was provided by GreenArrow to the /var/hvmail/control/license_key file.

Create web interface accounts

Run the following command:

/var/hvmail/studio/script/post-install

The above command will take a moment to run. When it’s finished, it will output the login information for both Engine and Studio’s web interfaces.

Engine and Studio have some shared components, so even if you don’t have a Studio license, you’ll still have Studio installed. You just won’t be able to use Studio without a Studio license key.

Be sure to record the login information so that you can use it later.

If you wish to change Engine’s password or create additional Engine accounts, those procedures are documented here.

If you wish to change Studio’s password or create additional Studio accounts, either can be done within the ADMIN tab of Studio’s web interface.

Initialize hvmail_migrate

Run this command:

hvmail_migrate migrate --new-install

Login URLs

Engine’s web interface is accessed at the path /greenarrowadmin/ of GreenArrow’s HTTP or HTTPS server. For example, on a server named example.com running HTTP service on the default ports, the web interface will be accessed at:

http://example.com/greenarrowadmin/

Studio’s web interface is accessible at the path /ga/ of GreenArrow’s HTTP or HTTPS server, so using the domain example.com as above, the login URL for GreenArrow Studio’s web interface will be:

http://example.com/ga/

Disable Studio services (Engine-only customers)

Studio customers should skip this step.

If you’re not using Studio, then disable its services to conserve system resources:

svc -d /service/hvmail-{studio-worker,pure-ftpd,pure-authd-studio}
touch /service/hvmail-{studio-worker,pure-ftpd,pure-authd-studio}/down

Server name

Configure the domain name of the server. This should be a fully qualified domain name with an A record pointing to this server. The default value is the server’s hostname.

Here’s an example of how to set the name to SERVER_NAME:

hvmail_set me SERVER_NAME
hvmail_init restart

More documentation on setting the server name can be found here.

Incoming email configuration

Create at least one incoming email domain using the Server Name, along with at least one bounce mailbox, and at least one spam complaint mailbox. These will be used by GreenArrow’s bounce processor and spam complaint processor.

We recommend having at leat one incoming email domain configured (using the Server name) with root, postmaster, and abuse email accounts or email forwarders. Any additional incoming email domains can be created using these steps, or they can be configured as aliased incoming email domains if they do not need to have a unique configuration.

From address domains

Add any from address domains that you’re planning to use into GreenArrow Engine’s configuration.

IP addresses for sending

At least one IP address is required for sending. Here’s how to configure each IP:

  1. Login to the admin interface
  2. Click on Configure -> VirtualMTAs
  3. Click the Add IP Address button in the IP Addresses section of the page, add fill out the form.

Further documentation here.

Optional GreenArrow Engine configuration

If you want the operating system to use GreenArrow for sending mail, then create the following symlinks:

ln -sf /var/hvmail/bin/sendmail /usr/sbin/sendmail
ln -sf /var/hvmail/bin/sendmail /usr/lib/sendmail

Grant SSH access to GreenArrow

GreenArrow technicians may occasionally need to SSH into your server in order to fulfill support requests. See SSH Access for details on how to grant access.

GreenArrow Monitor integration

If you have a GreenArrow Monitor account and would like to be able to view Monitor stats within Engine’s web interface, two configuration files should be created which store your Monitor account’s username and password. For example, if your username is username, and your password is password, these files would be created by running the following commands:

echo username > /var/hvmail/control/monitor.username
echo password > /var/hvmail/control/monitor.password

See the Installing and Updating your GreenArrow Monitor Seedlist document for information on how to add the GreenArrow Monitor seedlist into GreenArrow Engine or Studio.

Create URL Domain(s)

GreenArrow Engine and Studio use URL domains for click and open tracking.

We recommend creating at least one URL domain using the instructions in the URL (Click and Open Tracking) Domains page.

Configure SimpleMH

  1. Create a Mail Class named “default”. This is used for SimpleMH mail which does not have a Mail Class defined. You can either:
    1. Create the Mail Class using the Web Interface to choose your own default settings.
    2. Run this command to create the default Mail Class with a general preset configuration: 
      echo "INSERT INTO eng_mailclass (name, mail_route_id, listid) VALUES ('default', (SELECT default_mail_route_id FROM mr_default), 't1')" | /var/hvmail/postgres/default/bin/psql -U greenarrow
  2. Update the following lines in the /var/hvmail/control/simplemh-config configuration file:
    1. $CLICKTHROUGH_URL - set the default URL Domain. This should be set to the Server Name.
    2. $CAMPAIGNID_PREFIX - this is the prefix used with GreenArrow Monitor. It needs to be a unique value to your server, and needs to follow the parameters listed in the Campaign ID documentation. If you’re unsure of what to use here, you can run this to get a unique ID and paste it into the config: 
      cut -c1-30 /var/hvmail/control/me
    3. $RETURN_PATH_OVERRIDE - set the default Return Path Address Domain. This should be set to the Server Name. (Note that you need to comment out the line that sets this to “Undefined”: $RETURN_PATH_OVERRIDE = undef; and uncomment the line that has the special VERP syntax.)

Tune the operating system

  1. Mount each filesystem that GreenArrow is using with the noatime option. For example, the options in /etc/fstab might look like:

    defaults,noatime

    /etc/fstab updates can be applied with the mount command. For example, to apply changes to the / filesystem, run:

    mount -o remount /

  2. Disable disk quotas on each filesystem that GreenArrow is using. This is typically the default configuration on a new Linux installation.
  3. Add /var/hvmail, as well as any additional filesystem paths that GreenArrow is using to the list of paths to exclude from locate’s database by updating the PRUNEPATHS variable in the /etc/updatedb.conf file.

Tune GreenArrow Engine

Change the amount of RAM allocated to mail queues

GreenArrow Engine uses three mail queues. Two of these - the ram-queue, and the bounce-queue are stored in RAM. The amount of RAM that’s used for each of these queues is configurable.

If you’d like to do a deep dive into this topic, then our GreenArrow Concepts page discusses the queues in detail, and our General Settings page provides details on how to configure them. That said, here are my recommendations, which will provide you with a solid configuration the vast majority of the time:

  1. If you’re installing GreenArrow on a server that has at least 4GB of RAM, but less than 16GB, keep the default setings.

  2. If your GreenArrow server has at least 16GB of RAM, then you may be able to improve performance by increasing the defaults. You can increase them by running these commands:

    hvmail_set ramdisk_size xxlarge_3300mb_12000conn
hvmail_init restart

    Running the above commands on a server that has less than 16GB of RAM could lead to swapping, and in turn, performance issues.

    If your server starts swapping after running the above commands, then you can reset GreenArrow to its default mail queue configuration by running the following commands:

    hvmail_set ramdisk_size xlarge_500mb_2000conn
hvmail_init restart

  3. If your GreenArrow server has less than 4GB of RAM, then you may benefit from reducing the defaults. Please contact GreenArrow technical support if you’re in this situation, because a server with that little RAM may need to have other areas tuned as well.

Tune GreenArrow Studio

PostgreSQL

The default GreenArrow PostgreSQL configuration is usually optimal in two situations:

  1. You’re running GreenArrow Engine, but not GreenArrow Studio.
  2. You have at least 4GB, but less than 6GB of RAM.

If either of the above describe your server, then you can skip the rest of this PostgreSQL tuning section. Otherwise, read on.

If your server has less than 4GB of RAM, then PostgreSQL’s default settings may be too aggressive. Please contact GreenArrow technical support if you’re in that situation.

For all other servers (servers that are running GreenArrow Studio and have at least 6GB of RAM), we recommend tuning the following PostgreSQL settings, based on the amount of RAM available to GreenArrow. If there are any non-GreenArrow applications running on your server, then you should subtract their estimated maximum memory usage from your server’s configuration before performing the calculations. For example, if your server has 8GB of RAM, and there’s a MySQL server running on it which will use a maximum of 2GB of RAM, then you should perform the calculations in this section as if the server has 6GB of total RAM.

The four settings to adjust are:

Name Recommend value
shared_buffers 25% of total RAM
effective_cache_size 50% of total RAM
work_mem 1% of total RAM
maintenance_work_mem 2% of total RAM

These values can be set by editing the /var/hvmail/control/postgres.conf file.

Here’s an example /var/hvmail/control/postgres.conf file from a server that has 8GB of RAM, and is running only GreenArrow Engine and Studio:

shared_buffers = 2GB
effective_cache_size = 4GB
work_mem = 82MB
maintenance_work_mem = 164MB

Once you’re finished editing /var/hvmail/control/postgres.conf, restart PostgreSQL to apply changes by running the following command. Please note that this will cause between 15 seconds and a few minutes of downtime, so plan accordingly if your server is currently in production:

hvmail_postgres_manager restart

Once PostgreSQL has been restarted, you can verify that it’s running by visiting GreenArrow Studio’s web interface or running the command below, which should return the string PostgreSQL is online.

echo "select ('PostgreSQL is online')" | /var/hvmail/postgres/default/bin/psql -U greenarrow

One of the more common reasons for PostgreSQL not staying up is not being able to allocate enough memory. If this happens, then add or edit the kernel.shmmax and kernel.shmall sysctl parameters. This can be set the /etc/sysctl.conf file if you’re using CentOS, or the /etc/sysctl.d/greenarrow.conf file if you’re using Debian. Make sure kernel.shmmax (specified in in bytes) exceeds PostgreSQL’s shared_buffers setting.

To apply sysctl changes, run:

sysctl -p

Final Testing

  1. Verify that some common misconfigurations are not present. Each test gets a simple pass/fail result:

    hvmail_check_config

  2. Verify that all services are running normally. Any services with an abnormal state will be shown in red:

    hvmail_init status

    The hvmail-qmail-smtpd3 service is down by default. All other services should be reported as up unless you shut them down intentionally.

  3. Send yourself a test email (replacing [email protected] with your actual email address):

    date | /var/hvmail/bin/mailsubj "GreenArrow Test Message" [email protected]

Congratulations, you’re finished installing GreenArrow! As a next step, we recommend following up with GreenArrow technical support to schedule a training call.

 Firewall Configuration
FAQs