> For the complete documentation index, see [llms.txt](https://www.unh4ck.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://www.unh4ck.com/building-an-open-siem-from-scratch/1.-installing-elastic-stack.md).

# 1. Introduction to Elastic Stack

## What is Elastic Stack?

Elastic stack is a set of tools that can be used for collecting, ingesting, and searching data. In a context of SIEM, Elastic stack can be a powerful solution for monitoring purposes.

![](/files/-MOIh8PcxAlyv2FdMetK)

Formerly called ELK stack, now contains four major components:

* **Elasticsearch**: The database where your events will be indexed, stored, and queried for search.
* **Kibana**: The user interface to interact (*i.e. searching, creating visualizations and dashboards, and managing your stack*) with Elasticsearch.
* **Logstash**: My favorite tool of the stack, is equivalent of what you may call in a SIEM architecture a Data Collector or Processor. Although Logstash works in an ETL workflow and doesn't store any data, it supports many destinations where it can send data.
* **Beats**: Agents and events collectors. Elastic has a set of agents that support different datasets from different systems like Windows, Linux, Metrics, Uptime, Cloud, ...etc.

## Elasticsearch terminology?

> Elasticsearch is a distributed, open-source search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured. (<https://www.elastic.co/what-is/elasticsearch>)

### **Quick notes about Elasticsearch for a SIEM use case**

* Elasticsearch is fast, faster than most SIEMs you will see out there. One reason for its search speed that Elasticsearch is a **Schema-on-Write** kind of databases which means the **schema**; the fields, structure, and mappings of data are all defined first for the specific purpose that database will serve.
* In this [article ](https://www.elastic.co/blog/schema-on-write-vs-schema-on-read)Elastic refers to the way it stores logs as both Schema-on-Write and Schema-on-read they call it **minimal schema (***which is by default Elasticsearch stores data with minimal fields like **@timestamp**, **message** and some metadata like **tags** and **host*****)** but, in my point of view, in a logging use case it's best used with a schema-on-write approach since you would want to map and parse your IPs as IP data type and port numbers as long instead of adding a lot of [scripted fields](https://www.elastic.co/guide/en/kibana/current/scripted-fields.html) with some `painless`scripts at search time that may affect the performance of your cluster.
* This approach comes with pros and cons:
  * **Pros**:&#x20;
    * Increase in search time
    * Good for event-centric correlations
  * **Cons**:&#x20;
    * Writing data to disk could be affected
    * Time-base correlations would need more work at ingestion time.
* The logging use case need appropriate sizing of your Elasticsearch nodes for both data indexing and searching as well as a proper understanding of your data.

### Index

Indexing is the method by which search engines organize data for quick retrieval. The resulting structure is called, an index. An analogy of an index could be made to a table in a relational database (not exactly accurate but you got the idea).

An index is a collection of documents or a logical grouping of documents that often have a similar structure and is used to store and read documents.

### **Document**

Which are our log events in a SIEM context. A document can be similar to a '*row in a table in a relational database.'*

### Type

A type is a schema definition of a document in an index, a mapping, which mean the structure of the fields in a document. [Starting Elasticsearch version 7.0.0 an index can only have one type](https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html)

### Shard:

A shard is a collection of documents. It is similar to a smaller data partition. An index is stored in multiple distributed shards. There are two types of shards, **Primary** shards and their copies which called **Replicas**. Elasticsearch facilitates resiliency to node failover thanks to its horizontal scalability and cluster design. So, replicating your data across multiple nodes makes it high-available and resilient to single-node failover.

### Cluster

A cluster is a set on nodes that each runs an instance of Elasticsearch. There are distinct roles that can be defined for each node ([see more](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#modules-node)):

* **Master node**
* **Ingest node**
* **Data node (hot, warm, and cold)**
* **Machine Learning node**
* &#x20;**Remote-eligible node**
* **Transform node**

The bellow diagram can put all these Elasticsearch data structure concepts together:

![Elasticsearch Data Structure](/files/-MOI_NrRsCS-sGwk06vH)

### Inverted Indices

Searching data in Elasticsearch uses **inverted indices** which means that when we look for data we don't interact with JSON documents stored in our database but with inverted indices.

Using an inverted index is a lot like finding a book page that contains a certain keyword by scanning the index on the end of the book instead of scanning each page from start to finish. Think of when you start typing a phone number in your cell phone and the contact names start showing up according to what you are typing.