Logging and Observability with ELK Stack

Table of Contents

1. Introduction to the ELK Stack
   • What is Elasticsearch?
   • What is Logstash?
   • What is Kibana?
2. Setting Up Centralized Logging for Applications
   • Installing Elasticsearch and Logstash
   • Configuring Logstash
   • Setting Up Application Logging
3. Analyzing Logs and Creating Visualizations with Kibana
   • Importing Logs into Kibana
   • Using Kibana for Log Analysis
   • Creating Dashboards and Visualizations
4. Best Practices for Logging and Observability
5. Conclusion

Introduction to the ELK Stack

The ELK Stack refers to three open-source tools — Elasticsearch, Logstash, and Kibana — that together form a powerful logging and observability platform used for searching, analyzing, and visualizing machine-generated data in real-time.

What is Elasticsearch?

Elasticsearch is a distributed search and analytics engine. It is designed for fast search and retrieval of large volumes of data. Elasticsearch provides full-text search capabilities, real-time analytics, and scalability, making it ideal for storing and querying logs.

• Main features: Full-text search, real-time data retrieval, distributed design for scalability, and powerful aggregation capabilities.
• Use cases: Logs, metrics, and application data storage and analysis.

What is Logstash?

Logstash is a powerful data collection and transformation pipeline tool. It is responsible for gathering logs from various sources, processing the logs (such as parsing, filtering, and transforming), and then sending them to Elasticsearch for storage.

• Main features: Data collection, transformation, and sending data to Elasticsearch.
• Use cases: Centralized logging, data ingestion from multiple sources, filtering, and data enrichment.

What is Kibana?

Kibana is a data visualization tool for Elasticsearch. It allows users to search, view, and interact with the data stored in Elasticsearch through a user-friendly interface. Kibana provides powerful visualization capabilities, including dashboards, charts, and graphs, for analyzing logs and metrics.

• Main features: Interactive dashboards, real-time log analysis, visualizations, and alerts.
• Use cases: Log exploration, data visualization, and troubleshooting.

Together, these three tools form the ELK Stack, which provides a comprehensive solution for logging, monitoring, and observability in a modern DevOps environment.

Setting Up Centralized Logging for Applications

In this section, we’ll walk through setting up centralized logging with the ELK Stack, covering the installation and configuration of Elasticsearch, Logstash, and Logstash input filters.

Installing Elasticsearch and Logstash

1. Installing Elasticsearch: Elasticsearch can be installed from the official Elasticsearch website. For Linux, use the following commands: bashCopyEditwget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.10.2-linux-x86_64.tar.gz tar -xvzf elasticsearch-7.10.2-linux-x86_64.tar.gz cd elasticsearch-7.10.2 ./bin/elasticsearch
2. Installing Logstash: You can download Logstash from the Logstash website. For Linux, you can use: bashCopyEditwget https://artifacts.elastic.co/downloads/logstash/logstash-7.10.2-linux-x86_64.tar.gz tar -xvzf logstash-7.10.2-linux-x86_64.tar.gz cd logstash-7.10.2

Configuring Logstash

Logstash is used to parse and process log data before sending it to Elasticsearch. Let’s configure Logstash to accept logs from a file and send them to Elasticsearch.

1. Logstash Configuration File: Create a logstash.conf file to define input, filter, and output settings: plaintextCopyEditinput { file { path => "/var/log/myapp/*.log" start_position => "beginning" } } filter { # Here, you can apply log parsing, such as grok, date, and others. grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } } output { elasticsearch { hosts => ["http://localhost:9200"] index => "myapp-logs-%{+YYYY.MM.dd}" } }
   • Input: The file input plugin is used to collect log data from the specified directory (/var/log/myapp/*.log).
   • Filter: We use the grok filter to parse Apache log format. You can customize this based on your application’s log format.
   • Output: The logs are sent to an Elasticsearch instance running locally, with the logs being indexed daily.
2. Running Logstash: After creating the configuration file, run Logstash with the following command: bashCopyEdit./bin/logstash -f logstash.conf

This will start the Logstash service, which will monitor the log file and send the parsed logs to Elasticsearch.

Analyzing Logs and Creating Visualizations with Kibana

Once logs are ingested into Elasticsearch, we can use Kibana to visualize and analyze the logs.

Importing Logs into Kibana

1. Access Kibana: Once you have Kibana installed, access it at http://localhost:5601 (default port).
2. Configure an Index Pattern:
   • In Kibana, go to Management → Kibana Index Patterns.
   • Create a new index pattern for the logs coming from Logstash. For example, if your Logstash configuration specifies an index name like myapp-logs-*, create a pattern for myapp-logs-*.
   • Select the timestamp field (e.g., @timestamp) that will be used for time-based searches.

Using Kibana for Log Analysis

1. Search Logs:
   • Use the Discover tab in Kibana to search logs. You can filter logs based on fields (e.g., status codes, error messages) or time range.
   • Kibana allows you to run queries using Elasticsearch’s query language, making it easy to analyze and find issues in your logs.
2. Filter Logs:
   • Use the filters at the top of the Discover tab to filter logs by attributes such as status, method, or ip.

Creating Dashboards and Visualizations

1. Create a Dashboard:
   • Go to the Dashboard tab in Kibana and click Create New Dashboard.
   • Add visualizations such as time series graphs, pie charts, or tables to represent various metrics like the number of requests, error rates, or response times.
2. Visualization Types:
   • Use Visualize to create custom visualizations. You can use line charts for trends, pie charts for distribution, or bar charts for aggregations.
   • Example: Create a visualization to show the number of 500 errors in the last 24 hours.
3. Save and Share Dashboards:
   • After creating the necessary visualizations, save your dashboard. You can also share it with your team or integrate it into your monitoring system for real-time insights.

Best Practices for Logging and Observability

1. Centralized Logging:
   • Collect logs from all applications and services into a centralized logging system (like ELK). This simplifies monitoring and troubleshooting.
2. Structured Logging:
   • Use structured logging (e.g., JSON format) to make it easier to parse and analyze logs. Structured logs allow you to filter and query logs based on fields (e.g., user_id, status_code, error_message).
3. Use of Contextual Data:
   • Include useful contextual data in logs, such as request IDs, user session information, and error codes. This helps in tracing logs across distributed systems.
4. Retention and Index Management:
   • Implement proper log retention policies to avoid running out of disk space. Set up Elasticsearch index lifecycle management (ILM) to automatically manage the retention and rollover of indices.
5. Alerting:
   • Use Kibana’s alerting features to set up notifications for specific events, such as when error rates spike or when logs contain certain keywords.

Conclusion

In this module, we’ve explored how to set up centralized logging using the ELK Stack (Elasticsearch, Logstash, and Kibana). We covered installing and configuring Elasticsearch and Logstash for collecting and parsing logs, and then visualizing and analyzing them with Kibana. With these tools, you can monitor your applications more effectively, gain insights from logs, and quickly detect issues or anomalies in your system.