GreenArrow Documentation

Troubleshooting Disk Space Issues

Overview

These instructions are meant to help you prevent and detect disk space issues, identify the cause(s), and implement corrective actions. Feel free to contact GreenArrow technical support if you’d like help with a disk space issue.

It is important that you follow these instructions carefully. If you have a low disk situation, please do not start deleting files. Read this documentation and follow all the mitigation actions that match your situation. If you feel unsure of what actions to follow, then it is a good sign that you should contact GreenArrow technical support.

Get Notifications When Free Disk Space Is Low

To be notified when your system is running low on disk space we recommend that you add one or more email addresses to the error notification addresses list. You can find instructions in the General Settings page’s “Error Notification Address” section.

By default, the error notification will only send emails when a mounted filesystem’s utilization exceeds 80%. If you wish to set a different threshold, follow the instructions in the General Settings page’s “Disk Space Depletion Notifications” section.

View Available Space

Use the df command to view available disk space. This command reports on all mounted filesystems:

df -h

The output can be seen in the following example. Note: the -h option presents values in human readable format:

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        18G  5.0G   12G  30% /
tmpfs           1.9G     0  1.9G   0% /dev/shm
/dev/loop6      100M  1.7M   98M   2% /var/hvmail/qmail-bounce/queue
/dev/loop5      396M  2.4M  394M   1% /var/hvmail/qmail-ram/queue

The filesystem mounted on / has 12GB available space. 30% of the filesystem is in use which amounts to 18GB. The RAM (/var/hvmail/qmail-bounce/queue) and bounce (/var/hvmail/qmail-ram/queue) queues are almost empty. NOTE: the RAM and bounce queues are not stored on disk.

Use the du command to identify which directories are using the most space. du reports the space taken by all files and subdirectories of the specified directory, including filesystems mounted under the path that is reported.

This command will report GreenArrow’s total disk usage as well as the usage of the three subdirectories that usually account for the majority of GreenArrow’s disk usage:

du -hs /var/hvmail /var/hvmail/log /var/hvmail/postgres /var/hvmail/qmail-disk

This is an example of the output:

1.2G	/var/hvmail
21M	/var/hvmail/log
94M	/var/hvmail/postgres
35M	/var/hvmail/qmail-disk

In this example, the total space used by GreenArrow is 1.2GB, while the three subdirectories are using 21MB, 94M and 35MB.

Here’s more information on the above three subdirectories:

  • Messages that are deferred or throttled are stored in /var/hvmail/qmail-disk
  • The PostgreSQL database stores its data in /var/hvmail/postgres
  • GreenArrow stores logs in /var/hvmail/log

What to Do If GreenArrow’s Queue Is Causing a Partition to Be at Risk of Filling Up

If you’ve determined that the queue is the main cause for the partition to be at risk of filling up, then the following actions may help you get the system back to normal. Make sure that you follow the order written here.

Pause Injections

Try to stop the source application from injecting more messages. This will give GreenArrow Engine time to process the queue and either deliver or bounce the messages.

Dump Messages

Sometimes a few sends take a lot of space in the queue. For example, when they have big attachments and/or are being deferred constantly. If this is the case, it is possible to dump those messages from the queue. GreenArrow will remove the messages that are already in the queue on their next delivery attempt. It will also dump new messages injected with the same SendID. Follow the instructions in the Pausing and Dumping Queues document to dump messages.

Decrease the Queue Lifetime

By default, GreenArrow will retry delivering deferred and throttled messages for up to two days. Decreasing this value will reduce the amount of disk space that’s used by these messages.

The General Settings page’s “Queue Lifetime” section describes how to check and update this value.

Contact GreenArrow Technical Support

If you followed these instructions and couldn’t solve the problem, or if you feel unsure of what to do, please contact GreenArrow’s technical support.

What to Do If PostgreSQL Is Causing a Partition to Be at Risk of Filling Up

If PostgreSQL is using a lot of space, then we recommend that you do the following:

  1. If you use GreenArrow Studio, then verify that your Data Retention Settings are at the desired levels.

  2. Contact GreenArrow’s technical support

What to Do If Logs Are Causing a Partition to Be at Risk of Filling Up

If log files represent a high percentage of the disk usage, you can:

Reduce the Space Available for Delivery Attempt Log Files

Use the following command to determine the space that is being used by the delivery attempt log files:

du -hc /var/hvmail/log/*-qmail-send/ | grep total

The above figure can be reduced by using the hvmail_set command, as described in the General Settings page’s “Amount of Logs to Store” section to set the space that GreenArrow will use for keeping logs of delivery attempts.

Move /var/hvmail/log/send-summary Files

Use the following command to determine the space that is being used by the /var/hvmail/log/send-summary files:

du -hs /var/hvmail/log/send-summary

The files in /var/hvmail/log/send-summary are used to store some of GreenArrow Engine’s web interface statistics. To move older copies of these files to another partition:

  1. Verify that you’re running a version of GreenArrow Engine (4.1.64 or later) that supports this:

    rpm -q greenarrow-engine 2> /dev/null  || dpkg -l greenarrow-engine 2> /dev/null

    If you’re running an earlier release of GreenArrow Engine, please contact GreenArrow’s technical support to request that they upgrade your installation, or move these files.

  2. Determine how much free space will be needed in the directory that you’d like to move these files to:

    du -hs /var/hvmail/log/send-summary

  3. Create the directory that you would like old /var/hvmail/log/send-summary files to be moved to.

  4. Run the hvmail_move_old_send_summary_files command, specifying the minimum age of files to be moved as the first argument, and the destination directory as the second argument. For example, to move all /var/hvmail/log/send-summary files that were last modified 7 or more days ago to /media/scratch/var-hvmail-log-send-summary, run:

    /var/hvmail/bin/hvmail_move_old_send_summary_files 7 /media/scratch/var-hvmail-log-send-summary

  5. If you’re using Unmanaged Backups, verify that you’re using the latest version of the Unmanaged Backups script, which will automatically backup files at the new location. Managed Backups will also automatically backup files at the new location.

Contact GreenArrow Technical Support

If you followed these instructions and couldn’t solve the problem, or if you feel unsure of what to do, please contact GreenArrow’s technical support.

 hvmail_init Command
High Availability Cluster