a. Installing and configuring Elasticsearch

Guide to install Elastic Stack

For simplicity reasons I will be installing Elastic Stack as an All-in-One server that has all three instances, Elasticsearch, Kibana, and Logstash on a Ubuntu 18.04 LTS server with 8GB of RAM.

Sizing and configuring an Elastic Stack cluster with multiple nodes for logging use cases can be challenging and might be the subject of a separate blog.

Modify Hosts File

Firstly, I will add the name of my single-node cluster in /etc/hosts to use later in my elasticsearch configuration file.

192.168.20.222  elk_allinone
127.0.0.1 localhost

Disabling SWAP

Considering the impact of swap on the performance and stability of the nodes, it is recommended to disable it permanently by modifying the/etc/fstab file.

$ sudo swapon –show
$ sudo swapoff -v /swapfile
$ sudo vi /etc/fstab

Then comment any ligne that contains the word swap in the/etc/fstab then delete swapfile

$ sudo rm /swapfile

Download and install Debian packages manually

I am going to work here with version 7.8.0 of elastic stack since, later, I will be installing opendistro's plugins which are at the time of writing this blog was not compatible with the latest version of Elastic Stack. See here for more information about version compatibilities.

$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.8.0-amd64.deb
$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.8.0-amd64.deb.sha512
$ shasum -a 512 -c elasticsearch-7.8.0-amd64.deb.sha512 
$ sudo dpkg -i elasticsearch-7.8.0-amd64.deb

Modify Heap Size

As a rule of thumb, it is recommended to dedicate half of the available RAM to Elasticsearch. In our case we have 8GB so we will modify jvm.optionsaccordingly.

$ sudo vi /etc/elasticsearch/jvm.options

-Xms4g
-Xmx4g

Modify Configuration File

Next step is modifying elasticsearch.ymlconfiguration file to fit my single-node cluster requirements.

// Back up your configuration file
$ sudo cp elasticsearch.yml elasticsearch.yml.backup
$ sudo vi /etc/elasticsearch/elasticsearch.yml

# ======================== Elasticsearch Configuration ===================#
cluster.name: cyb3rsn0rlax
node.name: elk_allinone
node.master: true
node.ingest: true
node.data: true
node.ml: false
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: [_local_, _site_]
http.port: 9200
cluster.initial_master_nodes: ["elk_allinone"]

I have a single-node cluster so obviously it will be my data node for storing data, my master node and my ingest node if I am willing to use Elasticsearch pipelines.

Modifying System Configuration

It is recommended to modify system configuration to allow the user running Elasticsearch to access more resources than allowed by default. This part depends on the system you're using and the method you used to install Elasticsearch, since we are using Ubuntu and the Debian package manual installation the following method would make the necessary changes.

$ sudo systemctl edit elasticsearch

Add the following lines then reload and start Elasticsearch :

[Service]
LimitMEMLOCK=infinity

$ sudo systemctl daemon-reload
$ sudo systemctl start elasticsearch 

Lets begin!

Configure elasticsearch service for automatic startup

$ sudo /bin/systemctl daemon-reload
$ sudo /bin/systemctl enable elasticsearch.service

Check its status and curl it

$ sudo systemctl status elasticsearch
$ sudo curl localhost:9200

{
  "name" : "elk_allinone",
  "cluster_name" : "cyb3rsn0rlax",
  "cluster_uuid" : "-wuP6BX2QCiB5CSHKSw2GQ",
  "version" : {
    "number" : "7.8.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f",
    "build_date" : "2020-11-26T06:34:37.794943Z",
    "build_snapshot" : false,
    "lucene_version" : "8.4.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}