Elastic Stack (Elasticsearch, Logstash and Kibana) : Setup Kibana with Elasticsearch 6.0 (Document management using DevConsole tool)

The combination of Elasticsearch, Logstash, and Kibana, referred to as the "Elastic Stack. Elasticsearch (ELS) is a distributed search and analytics engine which runs on Apache Lucene (The indexing and search library for high performance, full text search engine). Logstash provides an input stream to Elastic for storage and search. Kibana is an open source data visualization plugin for Elasticsearch. Kibana accesses the data for visualizations (dashboards).

Setting up Elasticsearch and Kibana :

1. Download elasticsearch and kibana (I have downloaded mac version: elasticsearch-6.4.0.tar.gz and kibana-6.4.0-darwin-x86_64.tar.gz)
2. Unzip both and place at convenient location.

➜  ~ tar -xzf elasticsearch-6.4.0.tar.gz /Users/n0r0082/elasticsearch
➜  ~ tar -xzf kibana-6.4.0-darwin-x86_64.tar.gz /Users/n0r0082/Kibana

3. Start elasticsearch, run below command from elasticsearch home directory. 

   ➜  elasticsearch-6.1.1 bin/elasticsearch
   [2018-09-07T18:53:11,112][INFO ][o.e.n.Node               ] [] initializing ...
   [2018-09-07T18:53:11,196][INFO ][o.e.e.NodeEnvironment    ] [gnUvrnB] using [1] data paths, mounts [[/ (/dev/disk1s1)]], net usable_space [17.5gb], net total_space [233.5gb], types [apfs]
   [2018-09-07T18:53:11,196][INFO ][o.e.e.NodeEnvironment    ] [gnUvrnB] heap size [989.8mb], compressed ordinary object pointers [true]
   [2018-09-07T18:53:11,199][INFO ][o.e.n.Node               ] node name [gnUvrnB] derived from node ID [gnUvrnBNQual_2Jo9YD-ag]; set [node.name] to override
   [2018-09-07T18:53:11,199][INFO ][o.e.n.Node               ] version[6.1.1], pid[17236], build[bd92e7f/2017-12-17T20:23:25.338Z], OS[Mac OS X/10.13.4/x86_64], 
   JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_144/25.144-b01]
   ......
   [2018-09-07T18:53:46,762][INFO ][o.e.h.n.Netty4HttpServerTransport] [gnUvrnB] publish_address {127.0.0.1:9200}, bound_addresses {[fe80::1]:9200}, {[::1]:9200}, {127.0.0.1:9200}
   Note: Optionally we can add cluster_name and node_name while starting elsticsearch.
   bin/elasticsearch -Ecluster.name=devinline_es -Enode.name=devnode

4. Open terminal and run below command to verify that elasticsearch is running.

   ➜  ~ curl http://localhost:9200/
   {
     "name" : "gnUvrnB",
     "cluster_name" : "elasticsearch",
     "cluster_uuid" : "sxb87eolQuCVw4H4FhnbEQ",
     "version" : {
       "number" : "6.4.0",
       "build_hash" : "595516e",
       "build_flavor" : "default",
       "build_type" : "tar",
       "build_date" : "2018-08-17T23:18:47.308994Z",
       "build_snapshot" : false,
       "lucene_version" : "7.4.0",
       "minimum_wire_compatibility_version" : "5.6.0",
       "minimum_index_compatibility_version" : "5.0.0"
     },
     "tagline" : "You Know, for Search"
   }

5. Open config/kibana.yml and Set elasticsearch.url to point at your Elasticsearch instance. Here address of Elasticsearch instance is http://localhost:9200/
6. Start Kibana, run below command  from kibna home directory. 

   ➜  kibana-6.4.0-darwin-x86_64 bin/kibana                   
     log   [14:54:23.795] [info][status][plugin:kibana@6.4.0] Status changed from uninitialized to green - Ready
     log   [14:54:23.837] [info][status][plugin:elasticsearch@6.4.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
     log   [14:54:23.840] [info][status][plugin:xpack_main@6.4.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
     log   [14:54:23.844] [info][status][plugin:searchprofiler@6.4.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
     log   [14:54:29.992] [warning][admin][elasticsearch] Unable to revive connection: http://localhost:9200/
     log   [14:54:29.993] [warning][admin][elasticsearch] No living connections
     log   [14:54:32.658] [info][status][plugin:elasticsearch@6.4.0] Status changed from red to green - Ready
     log   [14:54:32.732] [info][license][xpack] Imported license information from Elasticsearch for the [data] cluster: mode: basic | status: active
   

7. Open Kibana WebUI with http://localhost:5601/ and run DevTools -> Run Default GET API & validate output. 
   

An analogy of Elasticsearch with RelationalDatabase:

Document management with Kibana Dev Tool Console: 

Create Index/Indexing document: Using  PUT command new index is created.
Syntax for creating index : PUT /<INDEX>/<TYPE>/<ID>. Below command create document with Id = 1 and index name = courses and type  = classrooom
Note:- By creating very first document, Index is automatically created. Index consist of two major components - "mappings" and "settings".

PUT /courses/classroom/1
{
    "name": "Capital Markets 350",
    "room": "E3",
    "professor": {
        "name": "Thomas Baszo",
        "department": "finance",
        "facutly_type": "part-time",
        "email": "baszot@onuni.com"
        },
    "students_enrolled": 13,
    "course_publish_date": "2016-01-11",
    "course_description": "This is an advanced course teaching crucial topics 
     related to raising capital and bonds, 
    shares and other long-term equity and debt financial instrucments"
}

Show response

{
  "_index": "courses",
  "_type": "classroom",
  "_id": "11",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "_seq_no": 1,
  "_primary_term": 1
}

Note: CURL command can be obtained from console (On right corner click on key icon beside play button). Refer this post for more details - Elasticsearch operations using CURL

GET/Retrieve document:  Using GET command we can retrieve document details.
Syntax for retrieving document : GET /<INDEX>/<TYPE>/<ID> . Below command retrieve document with Id = 1

GET /courses/classroom/11

Show response

{
  "_index": "courses",
  "_type": "classroom",
  "_id": "11",
  "_version": 1,
  "found": true,
  "_source": {
    "name": "Capital Markets 350",
    "room": "E3",
    "professor": {
      "name": "Thomas Baszo",
      "department": "finance",
      "facutly_type": "part-time",
      "email": "baszot@onuni.com"
    },
    "students_enrolled": 13,
    "course_publish_date": "2016-01-11",
    "course_description": "This is an advanced course teaching crucial topics related to raising capital and bonds, shares and other long-term equity and debt financial instrucments"
  }
}

GET Index Details: Using GET command we can get Index components - "mappings" and "settings" details. By default, elastic search created 5 shards which we can be controlled by creating Index structure before creating very first document. Below response shows structure of index - type of each fields of documents.

GET /courses

Show response

{
  "courses": {
    "aliases": {},
    "mappings": {
      "classroom": {
        "properties": {
          "course_description": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "course_publish_date": {
            "type": "date"
          },
          "name": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "professor": {
            "properties": {
              "department": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "email": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "facutly_type": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "name": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "room": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "students_enrolled": {
            "type": "long"
          }
        }
      }
    },
    "settings": {
      "index": {
        "creation_date": "1536385523497",
        "number_of_shards": "5",
        "number_of_replicas": "1",
        "uuid": "06GGK-uEQ3e8C2IpHZZOTQ",
        "version": {
          "created": "6040099"
        },
        "provided_name": "courses"
      }
    }
  }
}

Update document: Using POST command document can be updated. On success we get response as "result": "updated" and "_version" is incremented.

POST /courses/classroom/11
{
    "name": "Capital Markets 350",
    "room": "E5",
    "professor": {
        "name": "Thomas Baszo",
        "department": "finance",
        "facutly_type": "part-time",
        "email": "baszot@onuni.com"
        },
    "students_enrolled": 13,
    "course_publish_date": "2016-01-11",
    "course_description": "This is an advanced course teaching crucial topics related to raising capital and bonds, shares and other long-term equity and debt financial instrucments"
}

Show response

{
  "_index": "courses",
  "_type": "classroom",
  "_id": "11",
  "_version": 2,
  "result": "updated",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "_seq_no": 2,
  "_primary_term": 1
}

GET documents using search query(Request body search) : Using _search with GET command we can pass specific term for search. Here is complete list of options that can be used for request body search.

GET _search
{
    "query" : {
        "term" : { "professor.department": "finance" 
        }
    }
}

Delete document:

DELETE /courses/classroom/11

Show response

{
  "_index": "courses",
  "_type": "classroom",
  "_id": "11",
  "_version": 3,
  "result": "deleted",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "_seq_no": 3,
  "_primary_term": 1
}

Invalid operations in elastic Version 6 :

Create multiple type in an index: In older version of elasticsearch multiple type in a given index was allowed but in version 6, one type is tightly coupled with index. Blow command try to create employee type under courses index and it fails.

Note: In Elastic version7, type will also be removed.

PUT /courses/employee/1
{
    "name": "Capital Markets 350",
    "phone": "9861246780",
    "address": {
        "Address1": "Windsor prime corner",
        "apartment": "S4",
        "place": "Bangalore"
        },
    "department": "DST",
    "manager": "Siva Kuamr"
}

Show response

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Rejecting mapping update to [courses] as the final mapping would have more than 1 type: [classroom, employee]"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "Rejecting mapping update to [courses] as the final mapping would have more than 1 type: [classroom, employee]"
  },
  "status": 400
}

Related post: How to create inverted index and how inverted index is stored in segments of Shards.