Event Processor

Table of Contents
Configuration
Configuration Examples
Event Processor Logs
Starting and Stopping the Event Processor
Legacy Event Processor

The Event Processor provides a configurable mechanism for delivering events from GreenArrow to a database or HTTP server.

GreenArrow will only deliver events that it has been configured to log. If you haven’t configured event logging yet, then we recommend using the Event Notification System page as your starting point. It will direct you back to this page after the prerequisites are met.

Configuration

Definition

The event processor is configured in the JSON file found at /var/hvmail/control/event_processor.json.

Below is the definition for this configuration JSON document. The root of the document should be an object that defines the top-level keys configuration_mode and event_destinations.

The configuration must define at least one event destination.

The last event destination must match all events (matches must be set to { "all": true }).

event_destinations
array of hashes
An array of event destination hashes. The first event destination to match the incoming event will have the event delivered to it. Subsequent event destinations will not be used.

Each entry in the event_destinations array should have the following keys:

matches
hash
The matches hash defines what events this event destination will receive. All of the specified filters must match for this event.

all boolean	All events deliver to this event destination. This must be the only filter in the matches hash if present. This filter is best used as the final entry in your configuration.
mail_class string or array of strings	Events on one of the specified mail classes match when using this filter. If combined with `event_type`, both filters must match. This filter is case-insensitive.
event_type string or array of strings	Events on of the specified event types match when using this filter. If combined with `mail_class`, both filters must match. This filter is case-insensitive.

destination
hash
The destination hash defines where events that match this event destination will be delivered. All destinations must define the type key, with the individual destination types requiring different keys based on that type value.

type
string The method of communication to use for this event destination. For HTTP POST, this value must be http_post. For database connections, this value must be custom_sql. To leave the event in the queue, this value must be leave_in_queue. To drop the event from the queue without delivering it, this value must be drop_from_queue.

If type is http_post, the event will be delivered to the destination using an HTTP POST request. The following keys must be defined for these destinations.

url
string

The URL that will receive the POSTed data.

If type is custom_sql, the event will be delivered to the destination using a database connection. The following keys must be defined for these destinations.

db_dsn string	The DSN of the target database. See the following examples. For a MySQL database: `DBI:mysql:database=event_db;host=127.0.0.1` For a PostgreSQL database: `dbi:Pg:dbname=greenarrow;host=127.0.0.1` For an MS SQL database: `dbi:ODBC:DRIVER={ms-sql};Server=1.1.1.1;port=1433;database=dbname`
db_username string (optional)	The username that will be used in the database connection authentication.
db_password string (optional)	The password that will be used in the database connection authentication.
sql string	The SQL statement that will be executed in the database connection. Use `$X` for bind variables, where `X` is a 1-based index of the column in your `bind_list` array. For example, with `bind_list` set to `[ "event_time", "email", "event_type" ]`: `INSERT INTO my_events ( time, recipient_email, event_type ) VALUES ( ?, ?, ? )`
bind_list array of strings	The list of fields that will be bound to your query. See the Event Notification Documentation for a list of what fields can be bound.
replace_non_ascii string	If any of the data contains non-ASCII bytes, those bytes will be replaced with the given string.

If type is leave_in_queue, the event will remain in the event queue indefinitely. This should only be used in two situations:

As the only rule.
As a portion of the rule set for a limited period of time. Making this one of a set of rules long term can have drastically hinder GreenArrow's performance. The performance penalty is proportionate to the number of rows that accumulate.

If type is drop_from_queue, the event will be dropped from the event queue without being delivered.

http_keep_alive
integer Enable HTTP Keep-Alive for HTTP endpoints. Set this value to the maximum number of connections. This will cause the event processor to re-use the same connection for multiple events, increasing throughput. Set to null to disable HTTP Keep-Alive.

configuration_mode
string Set this key to perl to run the event processor from the legacy configuration file /var/hvmail/control/event_processor.conf. If this value is not present or set to json, this JSON configuration file will be used.

concurrency
integer Set this to the number of concurrent event processors that should execute simultaneously. Use this to increase the throughput of the event processor. By default, concurrency is set to 1 for a single event processor. If this value is set to 0, no event processors will run.

query_limit
integer Set the maximum number of events that can be delivered by a single execution of an event processor. By default, there is no limit on the number of events. If this value is set to 0, no limit will be used.
Unless you have been instructed by GreenArrow technical support to use this option, please do not use it. (This may be needed in situations where a large backlog of events has accumulated. Attempts to process them all in a single run can degrade performance, or cause significant pressure on available memory.)

Verification

To verify your configuration file, run the following command.

# hvmail_event_processor --check-syntax
No errors found in configuration.

This will verify that your configuration appears to be valid for running. This will not tell you that any successful connection was actually made, nor that your connection settings are correct.

Here’s an example of a syntax check on a configuration file that did not declare its event_destinations array.

# hvmail_event_processor --check-syntax
There was a problem with the configuration file /var/hvmail/control/event_processor.json:
configuration must define an 'event_destinations' array

You may also run the event processor in a mode that processes the events only for a single email address. This provides a good way of testing your configuration without bringing the event processor up, leaving all other events in your queue.

hvmail_event_processor --process-by-email "[email protected]"

Reloading

The configuration file will be automatically reloaded every 10 seconds. If there is an error found in the configuration after a reload, events will not be delivered.

Configuration Examples

HTTP Post Example

Here’s an example configuration that posts all events to an HTTP URL:

{
  "configuration_mode": "json",

  "event_destinations": [
    {
      "matches": { "all": true },
      "destination": {
        "type": "http_post",
        "url": "http://example.com/event_receiver?source=ga"
      }
    }
  ]
}

MySQL Example

Here’s an example configuration that sends all events, including all columns that are present by default as of 2017-8-24 to a MySQL database:

{
  "configuration_mode": "json",

  "event_destinations": [
    {
      "matches": { "all": true },
      "destination": {
        "type": "custom_sql",
        "db_dsn": "DBI:mysql:database=greenarrow;host=127.0.0.1",
        "db_username": "greenarrow",
        "db_password": "secretpassword",
        "sql": "INSERT IGNORE INTO events ( id, event_type, event_time, email, listid, list_name, list_label, sendid, bounce_type, bounce_code, bounce_text, click_url, click_tracking_id, studio_rl_seq, studio_rl_recipid, studio_campaign_id, studio_autoresponder_id, studio_is_unique, studio_mailing_list_id, studio_subscriber_id, studio_ip, studio_rl_seq_id, studio_rl_distinct_id, engine_ip, user_agent, json_before, json_after, timestamp, channel, status, is_retry, msguid, sender, mtaid, injected_time, message, outmtaid, sendsliceid, throttleid ) VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ? )",
        "bind_list": [ "id", "event_type", "event_time", "email", "listid", "list_name", "list_label", "sendid", "bounce_type", "bounce_code", "bounce_text", "click_url", "click_tracking_id", "studio_rl_seq", "studio_rl_recipid", "studio_campaign_id", "studio_autoresponder_id", "studio_is_unique", "studio_mailing_list_id", "studio_subscriber_id", "studio_ip", "studio_rl_seq_id", "studio_rl_distinct_id", "engine_ip", "user_agent", "json_before", "json_after", "timestamp", "channel", "status", "is_retry", "msguid", "sender", "mtaid", "injected_time", "message", "outmtaid", "sendsliceid", "throttleid"]
      }
    }
  ]
}

PostgreSQL Example

Here’s an example configuration that sends all events, including all columns that are present by default as of 2017-8-24 to a PostgreSQL database:

{
  "configuration_mode": "json",

  "event_destinations": [
    {
      "matches": { "all": true },
      "destination": {
        "type": "custom_sql",
        "db_dsn": "dbi:Pg:dbname=greenarrow;host=127.0.0.1",
        "db_username": "greenarrow",
        "db_password": "secretpassword",
        "sql": "INSERT INTO events ( id, event_type, event_time, email, listid, list_name, list_label, sendid, bounce_type, bounce_code, bounce_text, click_url, click_tracking_id, studio_rl_seq, studio_rl_recipid, studio_campaign_id, studio_autoresponder_id, studio_is_unique, studio_mailing_list_id, studio_subscriber_id, studio_ip, studio_rl_seq_id, studio_rl_distinct_id, engine_ip, user_agent, json_before, json_after, timestamp, channel, status, is_retry, msguid, sender, mtaid, injected_time, message, outmtaid, sendsliceid, throttleid ) VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ? ) ON CONFLICT DO NOTHING",
        "bind_list": [ "id", "event_type", "event_time", "email", "listid", "list_name", "list_label", "sendid", "bounce_type", "bounce_code", "bounce_text", "click_url", "click_tracking_id", "studio_rl_seq", "studio_rl_recipid", "studio_campaign_id", "studio_autoresponder_id", "studio_is_unique", "studio_mailing_list_id", "studio_subscriber_id", "studio_ip", "studio_rl_seq_id", "studio_rl_distinct_id", "engine_ip", "user_agent", "json_before", "json_after", "timestamp", "channel", "status", "is_retry", "msguid", "sender", "mtaid", "injected_time", "message", "outmtaid", "sendsliceid", "throttleid"]
      }
    }
  ]
}

Microsoft SQL Server Example

Here’s an example configuration that sends all events, including all columns that are present by default as of 2017-8-24 to a Microsoft SQL Server database:

{
  "configuration_mode": "json",

  "event_destinations": [
    {
      "matches": { "all": true },
      "destination": {
        "type": "custom_sql",
        "db_dsn": "dbi:ODBC:DRIVER={ms-sql};Server=10.0.0.1;port=1433;database=greenarrow",
        "db_username": "greenarrow",
        "db_password": "secretpassword",
        "sql": "INSERT INTO events ( id, event_type, event_time, email, listid, list_name, list_label, sendid, bounce_type, bounce_code, bounce_text, click_url, click_tracking_id, studio_rl_seq, studio_rl_recipid, studio_campaign_id, studio_autoresponder_id, studio_is_unique, studio_mailing_list_id, studio_subscriber_id, studio_ip, studio_rl_seq_id, studio_rl_distinct_id, engine_ip, user_agent, json_before, json_after, timestamp, channel, status, is_retry, msguid, sender, mtaid, injected_time, message, outmtaid, sendsliceid, throttleid ) VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ? )",
        "bind_list": [ "id", "event_type", "event_time", "email", "listid", "list_name", "list_label", "sendid", "bounce_type", "bounce_code", "bounce_text", "click_url", "click_tracking_id", "studio_rl_seq", "studio_rl_recipid", "studio_campaign_id", "studio_autoresponder_id", "studio_is_unique", "studio_mailing_list_id", "studio_subscriber_id", "studio_ip", "studio_rl_seq_id", "studio_rl_distinct_id", "engine_ip", "user_agent", "json_before", "json_after", "timestamp", "channel", "status", "is_retry", "msguid", "sender", "mtaid", "injected_time", "message", "outmtaid", "sendsliceid", "throttleid"]
      }
    }
  ]
}

HTTP Post and PostgreSQL Example

Here’s an example configuration that sends the id, event_type and event_time values for studio_open events to a database table, and everything else to an HTTP URL:

{
  "configuration_mode": "json",

  "event_destinations": [
    {
      "matches": { "event_type": [ "studio_open" ] },
      "destination": {
        "type": "custom_sql",
        "db_dsn": "dbi:Pg:dbname=greenarrow;host=127.0.0.1",
        "db_username": "greenarrow",
        "db_password": "secretpassword",
        "sql": "INSERT INTO events ( id, event_type, time_int ) VALUES ( ?, ?, ? )",
        "bind_list": [ "id", "event_type", "event_time" ]
      }
    },
    {
      "matches": { "all": true },
      "destination": {
        "type": "http_post",
        "url": "http://example.com/event_receiver?source=ga"
      }
    }
  ]
}

Do Nothing Example

This is the default configuration, which leaves all events in queue:

{
  "event_destinations": [
    {
      "matches": {
        "all": true
      },
      "destination": {
        "type": "leave_in_queue"
      }
    }
  ]
}

Event Processor Logs

The event processor logs are kept in /var/hvmail/log/event-processor. Use these commands to diagnose why an event is not being delivered.

For a streaming view of the log as it happens:

tail -F /var/hvmail/log/event-processor/current | tai64nlocal

To see a particular time range of events:

logdir_select_time --start "2015-11-24 19:00" --end "2015-11-25 00:00" --dir /var/hvmail/log/event-processor | tai64nlocal

Starting and Stopping the Event Processor

To check the running state of the event processor:

# svstat /service/hvmail-event-processor/
/service/hvmail-event-processor/: up (pid 20207) 515166 seconds

To start the event processor:

svc -u /service/hvmail-event-processor

To stop the event processor:

svc -d /service/hvmail-event-processor

Legacy Event Processor

Prior to the JSON configuration, the event processor was configured using a Perl file /var/hvmail/control/event_processor.conf that would be loaded straight into the running code.

If the JSON file exists and its configuration mode is not set to perl, the legacy configuration file will not be loaded. If you need to continue to use the legacy configuration file, simply edit /var/hvmail/control/event_processor.json, setting configuration_mode to perl.

To convert an existing legacy configuration file to the new syntax, run:

hvmail_event_processor --convert-legacy

This does the following:

Generate a new configuration file /var/hvmail/control/event_processor.json based on the data contained in legacy file /var/hvmail/control/event_processor.conf.
Renames the legacy file /var/hvmail/control/event_processor.conf to /var/hvmail/control/event_processor.conf.old.