Event stores

Zentral events are persisted and forwarded by event stores. Multiple stores can be used at the same time: each event is offered to every store, and each store decides whether to keep it, based on its event filters.

Each store that can write events gets its own queue and its own store worker. One store – the admin console store – is also used to fetch the events displayed in the Zentral UI.

Available backends

Options key is the object that carries the options of the backend, alongside the common store options . Terraform indicates whether the zentral_store resource covers the backend; its nested attribute is the options key without the _kwargs suffix. Admin console indicates whether the backend can fetch events back for display in the Zentral UI. Some backends – Datadog, Elasticsearch, OpenSearch and Splunk – can also display links to the events in their own UI; those require extra options, listed in the section of each backend.

The DATADOG, PANTHER, SNOWFLAKE, SPLUNK and SUMO_LOGIC backends are part of the Zentral Pro Edition.

Where stores are configured

Stores are database objects, not a section of the base.json configuration file. There are three ways to manage them:

• the API, at /api/stores/ (see API for the authentication options),
• the Zentral Terraform provider , with the zentral_store resource, which is built on that API. It covers four of the backends – see the backend overview ,
• provisioning, from the Zentral configuration, for stores that must be deployed with the instance.

The Zentral UI at /stores/ is read-only: it lists the configured stores and displays their settings, with the secrets redacted. The stores.view_store permission is required.

A store created by provisioning carries a provisioning_uid and can only be changed by updating the configuration: the API refuses to update or delete it, and its backend and backend options are omitted from the API representation. Stores created through the API are called custom stores, and their number is limited – see max_custom_store_count .

Configuration section options

The event store app itself is configured under the apps → zentral.core.stores key.

max_custom_store_count

OPTIONAL

An integer, 3 by default. The maximum number of stores that can be created through the API. Stores created by provisioning are not counted.

provisioning

OPTIONAL

A dictionary with a single stores key, itself a dictionary of store specifications. The key of each specification is its provisioning UID – a stable identifier of your choosing, used to match a specification with the store it created. Renaming the key on an existing store creates a second store.

{
    …
    "apps": {
        "zentral.core.stores": {
            "max_custom_store_count": 5,
            "provisioning": {
                "stores": {
                    "main-store": {
                        "name": "ClickHouse",
                        "admin_console": true,
                        "backend": "CLICKHOUSE",
                        "clickhouse_kwargs": {
                            "host": "clickhouse.example.com",
                            "password": "{{ env:CLICKHOUSE_PASSWORD }}"
                        }
                    }
                }
            }
        }
    }
}

The standard configuration substitutions are available for every value: "{{ env:ENV_VAR_NAME }}" to read an environment variable, "{{ file:FILE_PATH }}" to read a file, and "{{ secret:NAME_OF_THE_SECRET }}" to read an AWS or GCP secret. Use them for the secrets – they are encrypted with the configured secret engine when the store is saved.

Common store options

These options apply to every store, whatever the backend, and are used both by the API and by provisioning.

event_filters

An object with two optional keys, excluded_event_filters and included_event_filters. In Terraform, this is the event_filters attribute of zentral_store. Each is a list of filters, and each filter is an object with optional tags, event_type and routing_key attributes, whose values are non-empty lists of strings.

{
  "event_filters": {
    "excluded_event_filters": [
      {"tags": ["munki", "santa"]},
      {"event_type": ["osquery_result"], "routing_key": ["important"]},
      {"event_type": ["zentral_login", "zentral_logout"]}
    ]
  }
}

With these filters, the following events are excluded:

• munki or santa tagged events
• osquery_result events with the important routing_key value
• zentral_login or zentral_logout events

Boolean combinations: arrays/lists → OR, dictionaries/objects → AND.

• Within excluded_event_filters – or included_event_filters – the different filters are combined using the OR operator.
• Within each filter, the different attributes must all match (AND).
• For each filter attribute, at least one value must match (OR).

The excluded_event_filters take precedence over the included_event_filters. If an event is a match for the excluded_event_filters, the included_event_filters are not evaluated, and the event is excluded.

If neither excluded_event_filters nor included_event_filters is set, all events are included in the store.

Backend options

One section per backend, listing what goes in its <backend>_kwargs object.

ClickHouse

The events table, a few aggregation tables and their materialized views are created by Zentral when the store worker starts. The events table is partitioned by day, and the events expire after ttl_days. The aggregation tables – used for the event type and tag histograms, and for the machine heartbeats – have their own fixed retention of 31 days.

Example

{
    "name": "ClickHouse",
    "admin_console": true,
    "backend": "CLICKHOUSE",
    "clickhouse_kwargs": {
        "host": "clickhouse.example.com",
        "username": "zentral",
        "password": "{{ env:CLICKHOUSE_PASSWORD }}",
        "database": "zentral",
        "ttl_days": 400,
        "batch_size": 500
    }
}

Datadog

Events are sent to the Datadog log intake. Reading the events back – required for the admin console store – uses the Datadog log API, and needs an application_key in addition to the api_key.

Links to the events in the Datadog UI are always available.

Example

{
    "name": "Datadog",
    "admin_console": true,
    "backend": "DATADOG",
    "datadog_kwargs": {
        "site": "datadoghq.eu",
        "api_key": "{{ env:DATADOG_API_KEY }}",
        "application_key": "{{ env:DATADOG_APPLICATION_KEY }}",
        "service": "zentral.example.com"
    }
}

Elasticsearch

Multiple indices

Instead of a single index, a list of indices can be configured, so that different events land in different indices – to give them a different retention or a different storage class, for example.

Each entry is an object with a name, a priority and, optionally, included_event_filters and excluded_event_filters – the same filter syntax as event_filters . For each event, the matching index with the highest priority wins.

• All the names must be different, and all the priorities must be different.
• The index with the lowest priority is the default index. It cannot be filtered, so that no event is left without an index.
• read_index is required, and should point to an alias covering every index you want to search from the Zentral UI.

Example

{
    "name": "Elasticsearch",
    "admin_console": true,
    "backend": "ELASTICSEARCH",
    "elasticsearch_kwargs": {
        "hosts": ["https://elasticsearch.example.com:9200"],
        "username": "zentral",
        "password": "{{ env:ELASTICSEARCH_PASSWORD }}",
        "batch_size": 100,
        "indices": [
            {"name": "zentral-osquery", "priority": 10,
             "included_event_filters": [{"tags": ["osquery"]}]},
            {"name": "zentral-other", "priority": 1}
        ],
        "read_index": "zentral-all",
        "kibana_discover_url": "https://kibana.example.com/app/discover",
        "kibana_index_pattern_uuid": "00000000-0000-0000-0000-000000000000"
    }
}

HTTP

Terraform resource: zentral_store , with an http attribute.

Events are POSTed one by one, as JSON, to an arbitrary endpoint.

Example

{
    "name": "ServiceNow",
    "backend": "HTTP",
    "event_filters": {
        "included_event_filters": [
            {"event_type": ["add_machine", "add_machine_os_version", "remove_machine_os_version"]}
        ]
    },
    "http_kwargs": {
        "endpoint_url": "https://acme.service-now.com/api/now/import/zentral_events",
        "username": "Zentral",
        "password": "{{ env:SERVICE_NOW_API_PASSWORD }}",
        "concurrency": 5
    }
}

Kinesis

Terraform resource: zentral_store , with a kinesis attribute.

This store is capable of batch operation. The maximum batch_size is 500. See the kinesis:PutRecords documentation for more details.

AWS authentication and authorization

When operating in AWS, it is recommended to use a role attached to the EC2 instance or to the container to authenticate the calls to the Kinesis API, and to leave aws_access_key_id and aws_secret_access_key empty.

Example of an IAM policy to allow Zentral to write to the Kinesis stream:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowKinesisPut",
            "Action": [
                "kinesis:PutRecords",
                "kinesis:PutRecord"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:kinesis:<AWS_REGION>:<AWS_ACCOUNT_ID>:stream/<KINESIS_STREAM_NAME>"
        }
    ]
}

The PutRecord action can be omitted if the store is configured for batch operations, i.e. with a batch_size greater than 1.

Example

{
    "name": "Kinesis",
    "backend": "KINESIS",
    "kinesis_kwargs": {
        "stream": "name_of_the_stream",
        "region_name": "us-east-1",
        "serialization_format": "firehose_v1",
        "assume_role_arn": "arn:aws:iam::<ACCOUNT_ID>:role/<NAME_OF_THE_ROLE>",
        "batch_size": 500
    }
}

OpenSearch

Use this store with a managed AWS OpenSearch domain. It takes all the Elasticsearch options – including multiple indices – plus aws_auth, to sign the API calls with the standard AWS signatures.

It is recommended to use a role attached to the EC2 instance or to the container, and to leave aws_auth.aws_access_key_id and aws_auth.aws_secret_access_key empty: when they are, the credentials are resolved with the default boto3 mechanisms .

Use kibana_discover_url and kibana_index_pattern_uuid for the OpenSearch Dashboards, for example https://example-00000000000000000000000000.us-east-1.es.amazonaws.com/_dashboards.

Example

{
    "name": "OpenSearch",
    "admin_console": true,
    "backend": "OPENSEARCH",
    "opensearch_kwargs": {
        "hosts": ["https://example-00000000000000000000000000.us-east-1.es.amazonaws.com"],
        "index": "zentral-events",
        "batch_size": 100,
        "kibana_discover_url": "https://example-00000000000000000000000000.us-east-1.es.amazonaws.com/_dashboards",
        "kibana_index_pattern_uuid": "00000000-0000-0000-0000-000000000000",
        "aws_auth": {"region_name": "us-east-1"}
    }
}

Panther

Terraform resource: zentral_store , with a panther attribute.

Zentral can send events to a Panther HTTP log source, with Bearer authentication. A custom schema must be configured in Panther – use schema.yml from the Zentral repository.

Example

{
    "name": "Panther",
    "backend": "PANTHER",
    "panther_kwargs": {
        "endpoint_url": "https://logs.example.runpanther.net/http/00000000-0000-0000-0000-000000000000",
        "bearer_token": "{{ env:PANTHER_BEARER_TOKEN }}",
        "batch_size": 100
    }
}

S3 Parquet

Events are batched and written to an S3 bucket as Parquet files, for querying with Athena, Snowflake, DuckDB, … Each file is written to <prefix>/<YYYY>/<MM>/<DD>/<writer id>/<batch index>.parquet, where the writer ID is unique to each store worker run.

This backend only operates in batch mode: batch_size cannot be lower than 100. The max_batch_age_seconds option bounds how long a partial batch is held before being flushed.

As with the other AWS backends, it is recommended to use a role attached to the EC2 instance or to the container, and to leave aws_access_key_id and aws_secret_access_key empty. The role must be allowed to perform the s3:PutObject action on the bucket.

Example

{
    "name": "S3 Parquet",
    "backend": "S3_PARQUET",
    "s3_parquet_kwargs": {
        "bucket": "acme-zentral-events",
        "prefix": "events/",
        "region_name": "us-east-1",
        "batch_size": 50000,
        "max_batch_age_seconds": 600
    }
}

Snowflake

The Snowflake backend is read-only: it has no store worker, and can only be used as the admin console store. To get the events into Snowflake, set up a pipeline with the Kinesis or S3 Parquet backend – with Kinesis Firehose or Snowpipe, for example.

Example

{
    "name": "Snowflake",
    "admin_console": true,
    "backend": "SNOWFLAKE",
    "snowflake_kwargs": {
        "account": "acme",
        "user": "ZENTRAL",
        "password": "{{ env:SNOWFLAKE_PASSWORD }}",
        "database": "ZENTRAL",
        "schema": "ZENTRAL",
        "role": "ZENTRAL",
        "warehouse": "DEFAULTWH"
    }
}

Splunk

Terraform resource: zentral_store , with a splunk attribute.

Events are written with the Splunk HTTP Event Collector. Two optional extras are available:

• set search_app_url to display links to the events in the Splunk UI,
• set both search_url and search_token to let Zentral fetch the events back, i.e. to use the store as the admin console store.

Example

{
    "name": "Splunk",
    "admin_console": true,
    "backend": "SPLUNK",
    "splunk_kwargs": {
        "hec_url": "https://splunk.example.com:8088",
        "hec_token": "{{ env:SPLUNK_HEC_TOKEN }}",
        "hec_extra_headers": [
            {"name": "CF-Access-Client-Id", "value": "123"},
            {"name": "CF-Access-Client-Secret", "value": "{{ env:SPLUNK_HEC_CF_ACCESS_CLIENT_SECRET }}"}
        ],
        "hec_request_timeout": 30,
        "hec_index": "zentral",
        "hec_source": "zentral.example.com",
        "batch_size": 100,
        "serial_number_field": "serial_number",
        "computer_name_as_host_sources": ["santa", "osquery"],
        "search_app_url": "https://splunk.example.com/en-US/app/search/search",
        "search_url": "https://splunk.example.com:8089",
        "search_token": "{{ env:SPLUNK_SEARCH_TOKEN }}",
        "search_extra_headers": [
            {"name": "CF-Access-Client-Id", "value": "456"},
            {"name": "CF-Access-Client-Secret", "value": "{{ env:SPLUNK_SEARCH_CF_ACCESS_CLIENT_SECRET }}"}
        ],
        "search_source": "zentral.example.com",
        "verify_tls": true
    }
}

Sumo Logic

Events are sent to a Sumo Logic HTTP collector.

Example

{
    "name": "Sumo Logic",
    "backend": "SUMO_LOGIC",
    "sumo_logic_kwargs": {
        "collector_url": "{{ env:SUMO_LOGIC_COLLECTOR_URL }}",
        "batch_size": 100
    }
}