Centralized Logging with OpenSearch
Centralized Logging with OpenSearch helps organizations collect, ingest, and visualize log data from various sources using Amazon OpenSearch Service. This AWS Solution provides a web-based console, which you can use to create log ingestion pipelines with a few clicks. Log ingestion pipelines include log collection agent deployment, log enrichment without writing codes, buffer layer creation, and OpenSearch index configuration. After logs are stored in OpenSearch Service, the solution automatically generates ready-to-use dashboards for analyzing AWS service logs and application logs in different formats (for example, Nginx, JSON, and Spring Boot). In combination with other AWS services, this solution provides you with a turnkey environment to begin logging and monitoring your AWS applications.
Concepts
Log Analytics Engine
A log analytics engine is a sophisticated tool designed to process, analyze, and visualize vast amounts of log data from diverse systems, applications, and devices. Our solution primarily uses the Amazon OpenSearch Service as the default log analytics engine, complemented by a Light Engine specifically optimized for structured, infrequent logs.
OpenSearch Engine
The OpenSearch Engine in this solution refers to the Amazon OpenSearch Service, a distributed, community-driven, Apache 2.0-licensed, 100% open source search and analytics suite used for a broad set of use cases like real-time application monitoring, log analytics, and website search.
Light Engine
The Light Engine is a serverless log analytics engine that uses AWS services like Athena, Glue, Lambda, and Step Functions. Designed to analyze structured and infrequent logs, it offers up to a 90% cost reduction compared to the OpenSearch Engine.
For mission-critical, high-performance, or essential workloads, opt for OpenSearch. Conversely, use Light Engine to save costs when handling secondary, less critical workloads that do not require the same level of performance.
| OpenSearch Engine | Light Engine | |
|---|---|---|
| Ingest latency | Seconds ~ Minutes (depends on buffer layer type) | 5 minutes preceding |
| Query latency | Milliseconds ~ Seconds | Seconds ~ Minutes |
| Text search | Full-text | Fuzzy (the "Like" syntax of SQL) |
| Query language | DSL (Domain Specific Language) | SQL |
| Dashboards | OpenSearch Dashboards | Grafana |
| Alarm | Provided by OpenSearch Dashboards | Provided by Grafana |
| Access control | Field level | Table level |
| Sharing | Export data | Parquet files in Amazon S3 |
| Operational effort | High | Low |
| Schema | Semistructured | Structured |
| Cost | High | Low |
| Pricing model | Fixed; OpenSearch node types and counts, storage | On demand |
Light Engine - Centralized Logging with OpenSearch
Log Analytics Pipeline
A Log Analytics Pipeline, or Log Pipeline, represents the entire data flow from the source to the log analytics engines. It typically encompasses the stages of shipping, buffering, processing, filtering, enriching, and storing logs. Centralized Logging with OpenSearch supports two types of Log Analytics Pipelines: the Service Log Pipeline, tailored for ingesting logs generated by AWS Services, and the Application Log Pipeline, designed for ingesting logs from custom applications.
Log Source
A Log Config defines the metadata of your logs, specifying the log type, format, sample logs, filters, and the schema needed to map raw log data into the structured format used by the log analytics engine. A Log Source refers to the location where logs are generated or stored. Centralized Logging with OpenSearch supports ingesting logs from diverse sources, encompassing both application logs and logs from AWS services. For supported AWS service logs, refer to AWS Service Logs. For supported application logs, refer to Application Logs.
Log Agent
A log agent is a program that reads logs from one location and sends them to another location (for example, OpenSearch). Currently, Centralized Logging with OpenSearch only supports the Fluent Bit 1.9 log agent, which is installed automatically. The Fluent Bit agent has a dependency of OpenSSL 1.1. To learn how to install OpenSSL on Linux instances, refer to OpenSSL installation. To find the platforms supported by Fluent Bit, see Supported Platforms.
Log Config
A Log Config defines the metadata of your logs, specifying the log type, format, sample logs, filters, and the schema needed to map raw log data into the structured format used by the log analytics engine.
Log Buffer
Log Buffer is a buffer layer between the Log Agent and OpenSearch clusters. The agent uploads logs into the buffer layer before being processed and delivered into the log analytics engine. A buffer layer is a way to protect the log analytics engine from being overwhelmed. For AWS service logs, a log buffer is automatically configured if needed. For Application logs, this solution provides the following types of buffer layers.
- Amazon S3. Use this option if you can bear minutes-level latency for log ingestion. The log agent periodically uploads logs to an Amazon S3 bucket. The frequency of data delivery to Amazon S3 is determined by Buffer size (default value is 50 MiB) and Buffer interval (default value is 60 seconds) values that you configured when creating the application log analytics pipelines. The condition satisfied first starts data delivery to Amazon S3.
- Amazon Kinesis Data Streams. Use this option if you need real-time log ingestion. The log agent uploads logs to Amazon Kinesis Data Stream in seconds. The frequency of data delivery to Kinesis Data Streams is determined by Buffer size (10 MiB) and Buffer interval (5 seconds). The condition satisfied first triggers data delivery to Kinesis Data Streams.
Log Buffer is optional when creating an application log analytics pipeline. For all types of application logs, you can use this solution to ingest logs without any buffer layers. However, we only recommend this option when you have small log volume, and you are confident that the logs will not exceed the thresholds at the OpenSearch side.
Instance Group
An Instance Group represents a group of EC2 instances, which enables the solution to associate a Log Config with multiple EC2 instances quickly. Centralized Logging with OpenSearch uses Systems Manager Agent (SSM Agent) to install/configure Fluent Bit agent, and sends log data to Kinesis Data Streams. Instance Group is one of the supported Log Sources in this solution.
Main Account
An AWS account where the Centralized Logging with OpenSearch console is deployed. The Log Analytics Engines must also reside in the same account.
Member Account
Another AWS account from which you want to ingest AWS Service logs or application logs. Logs are sent from Member Accounts to Main Accounts, where they are analyzed using resources in the Main Account.
Access Proxy
An Access Proxy serves as an intermediary for accessing Amazon OpenSearch Service domains from the internet securely. By default, an Amazon OpenSearch Service domain within a VPC is not accessible from the internet. The Centralized Logging with OpenSearch solution implements a Nginx-based proxy stack architecture to enable internet access to OpenSearch Dashboards. This allows users to interact conveniently with the dashboards from anywhere with internet connectivity.
Centralized Logging with OpenSearch | Implementations | AWS Solutions