ElastAlert
ElastAlert is a simple framework for alerting on anomalies, spikes, or other patterns of interest from data in Elasticsearch.
It works by combining Elasticsearch with two types of components, rule types and alerts. Elasticsearch is periodically queried and the data is passed to the rule type, which determines when a match is found. When a match occurs, it is given to one or more alerts, which take action based on the match.
This is configured by a set of rules, each of which defines a query, a rule type, and a set of alerts.
Several rule types with common monitoring paradigms are included with ElastAlert:
- "Match where there are X events in Y time" (frequencytype)
- "Match when the rate of events increases or decreases" (spiketype)
- "Match when there are less than X events in Y time" (flatlinetype)
- "Match when a certain field matches a blacklist/whitelist" (blacklistandwhitelisttype)
- "Match on any event matching a given filter" (anytype)
- "Match when a field has two different values within some time" (changetype)
In addition to this basic usage, there are many other features that make alerts more useful:
- Alerts link to Kibana dashboards
- Aggregate counts for arbitrary fields
- Combine alerts into periodic reports
- Separate alerts by using a unique key field
- Intercept and enhance match data
Rule Types
- 'flatline' -- when there are less than X events in Y time
- 'blacklist/whitelist' -- when a certain field matches 'blacklist' or 'whitelist'
- 'any' -- when an event that matches a given filter happens
- 'change' -- when a field has two different values within a specified period of time
Types
-
spike
-
frequency
-
flatline
-
new_term
-
change
Common Configuration Example
-
Required settings
- es_host
- es_port
- index
- name
- type
- alert
-
Optional settings
-
import
-
use_ssl
-
verify_certs
-
client_cert
-
client_key
-
ca_certs
-
es_username
-
es_password
-
es_url_prefix
-
es_send_get_body_as
-
use_strftime_index
-
search_extra_index
-
aggregation
aggregation: schedule: '2 4 ** mon, fri'
aggregation: hours: 2
-
aggregate_by_match_time
-
realert
-
exponential_realert
-
buffer_time
-
query_delay
-
owner
-
priority
-
category
-
max_query_size
-
filter
-
include
-
top_count_keys
-
top_count_number
-
raw_count_keys
-
description
-
generate_kibana_link
-
kibana_url
-
use_kibana_dashboard
-
use_kibana4_dashboard
-
kibana4_start_timedelta
-
kibana4_end_timedelta
-
use_local_time
-
match_enhancements
-
run_enhancements_first
-
query_key
-
aggregation_key
-
summary_table_fields
-
timestamp_type
-
timestamp_format
-
timestamp_format_expr
-
_source_enabled
-
scan_entire_timeframe
-
https://elastalert.readthedocs.io/en/latest/ruletypes.html#common-configuration-options
https://elastalert.readthedocs.io/en/latest/ruletypes.html
https://elastalert.readthedocs.io/en/latest/ruletypes.html#rule-types
Example
deadman_slack: |-
---
name: Deadman Switch Slack
type: frequency
index: containers-*
num_events: 3
timeframe:
minutes: 3
filter:
- term:
message: "deadmanslack"
alert:
- "slack"
slack:
slack_webhook_url: dummy
deadman_pagerduty: |-
---
name: Deadman Switch PagerDuty
type: frequency
index: containers-*
num_events: 3
timeframe:
minutes: 3
filter:
- term:
message: "deadmanpd"
alert:
- "pagerduty"
pagerduty:
pagerduty_service_key: dummy
pagerduty_client_name: Elastalert Deadman Switch
References
https://elastalert.readthedocs.io/en/latest/index.html
https://engineeringblog.yelp.com/2015/10/elastalert-alerting-at-scale-with-elasticsearch.html
https://github.com/bitsensor/elastalert