Executing Search Requests Using Elasticsearch Query DSL: Search using query param and request body

In previous posts we witnessed CRUD Operations & _bulk API uses in Elasticsearch(ELS). Once documents are placed in Indices, we can perform search operation and retrieve results. There are two ways to execute search request for documents in ELS - Search using query param and Search using request body.

Prerequisite:
1. Make sure Elasticsearch is up and running(at least single node).
2. cURL is installed or some web browser is in place to send HTTP request.
3. Setup documents in Indices - Refer _bulk API for setup JSON document in indices so that query can be executed. Download cleaned JSON file with 1000 documents (for customers) & Download JSON for Products index and execute following command to setup indexed document in bulk.

curl -H "Content-Type: application/x-ndjson" -XPOST 'localhost:9200/customers/vendors/_bulk?pretty&refresh' --data-binary @"customers.json"

curl -H "Content-Type: application/x-ndjson" -XPOST 'localhost:9200/products/shoes/_bulk?pretty&refresh' --data-binary @"shoes.json"

4. Validate that 1000 document is added at index customer with type name vendors and 8 documents added in index products.

Desktop curl -XGET 'localhost:9200/_cat/indices?v&pretty'                                                                                          
health status index     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   customers KSbOq8eySwGgvJdH7VfQWQ   5   1       1000            0    101.5kb        101.5kb
yellow open   employee  LU8xvoyMRwi-0o5K2JCyMg   5   1        100            0     86.1kb         86.1kb
yellow open   products  AIA9n0qFQN6suaMG6kzYMw   5   1          8            0     25.2kb         25.2kb

Once we have setup documents we are ready to execute search query against those documents. 

Search using query param:-  Search term as URL query param 

Search in a given index with query parameter:- Pass as one string as parameter and display results based on relevance of query parameter.

➜  Desktop curl -XGET 'localhost:9200/customers/_search?q=wyoming&pretty'             
{
  "took" : 4,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 18,
    "max_score" : 4.9512873,
    "hits" : [
      {
        "_index" : "customers",
        "_type" : "vendors",
        "_id" : "rX_g6mABB3_D7Pc85hRy",
        "_score" : 4.9512873,
        "_source" : {
          "name" : "Dillard Wilkerson",
          "age" : 38,
          "gender" : "male",
          "email" : "dillardwilkerson@comvex.com",
          "phone" : "+1 (935) 505-3017",
          "street" : "954 Vernon Avenue",
          "city" : "Wyoming",
          "state" : "North Carolina, 3746"
        }
      },
      {
        "_index" : "customers",
        "_type" : "vendors",
        "_id" : "CX_g6mABB3_D7Pc85hNx",
        "_score" : 4.6838965,
        "_source" : {
          "name" : "Ashlee Barber",
          "age" : 54,
          "gender" : "female",
          "email" : "ashleebarber@comvex.com",
          "phone" : "+1 (957) 494-3259",
          "street" : "705 Batchelder Street",
          "city" : "Katonah",
          "state" : "Wyoming, 9682"
        }
      },
      {
        "_index" : "customers",
        "_type" : "vendors",
        "_id" : "TX_g6mABB3_D7Pc85hVy",
        "_score" : 4.6157947,
        "_source" : {
          "name" : "House Hughes",
          "age" : 31,
          "gender" : "male",
          "email" : "househughes@comvex.com",
          "phone" : "+1 (936) 406-2766",
          "street" : "175 Ivan Court",
          "city" : "Eden",
          "state" : "Wyoming, 6312"
        }
      },
      {
       .... 
       ....
      }
      
    ]
}
}

We can execute <localhost:9200/customers/_search?q=wyoming> in browser and get similar result as above. Below is screenshot of result obtained from browser.

Note :-
- By default maximum number of results returned is 10.
- Results are returned in decreasing order of score (Score indicate most relevant doc first followed by decreasing relevance)

Search in a given index with multiple query parameter:- Pass as string as parameter and sort result based on another parameter.

➜  Desktop curl -XGET 'localhost:9200/customers/_search?q=wyoming&sort=age:asc&pretty'
{
  "took" : 5,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 18,
    "max_score" : null,
    "hits" : [
      {
        "_index" : "customers",
        "_type" : "vendors",
        "_id" : "n3_g6mABB3_D7Pc85hRy",
        "_score" : null,
        "_source" : {
          "name" : "Robin Anthony",
          "age" : 19,
          "gender" : "female",
          "email" : "robinanthony@comvex.com",
          "phone" : "+1 (806) 428-3024",
          "street" : "367 Union Avenue",
          "city" : "Verdi",
          "state" : "Wyoming, 2643"
        },
        "sort" : [
          19
        ]
      },{
        "_index" : "customers",
        "_type" : "vendors",
        "_id" : "Q3_g6mABB3_D7Pc85hNx",
        "_score" : null,
        "_source" : {
          "name" : "Henry Waters",
          "age" : 21,
          "gender" : "male",
          "email" : "henrywaters@comvex.com",
          "phone" : "+1 (888) 493-3529",
          "street" : "215 Cook Street",
          "city" : "Santel",
          "state" : "Wyoming, 3995"
        },
        "sort" : [
          21
        ]
      },
      {
      ....
      ....
      }
    ]
  }
}

Note:-
- Score is irrelevant when results are sorted based on some parameter so max score and individual score is displayed as null.
- Results are in sorted order lowest age first.

Search query with name/value pair parameters :  Query parameter can be passed name value pair and documents can be retrieved. Here query parameter is state:Connecticut.

➜  Desktop curl -XGET 'localhost:9200/customers/_search?q=state:Connecticut&pretty'

➜  Desktop curl -XGET 'localhost:9200/customers/_search?q=state:Connecticut&sort=age:asc&pretty'

Search query with from<N> and size<M> parameters : In search query parameter from and size parameters can be passed and specific documents are retrieved. In other words, we want document from at index N and result should contain M documents.

Desktop curl -XGET 'localhost:9200/customers/_search?q=state:Connecticut&from=5&size=2&pretty'       
{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 18,
    "max_score" : 5.2171135,
    "hits" : [
      {
        "_index" : "customers",
        "_type" : "vendors",
        "_id" : "43_g6mABB3_D7Pc85hRy",
        "_score" : 4.128432,
        "_source" : {
          "name" : "Fernandez Brock",
          "age" : 69,
          "gender" : "male",
          "email" : "fernandezbrock@comvex.com",
          "phone" : "+1 (951) 577-2819",
          "street" : "922 Oriental Boulevard",
          "city" : "Hilltop",
          "state" : "Connecticut, 5451"
        }
      },
      {
        "_index" : "customers",
        "_type" : "vendors",
        "_id" : "HX_g6mABB3_D7Pc85hVy",
        "_score" : 4.128432,
        "_source" : {
          "name" : "Ronda Gallegos",
          "age" : 25,
          "gender" : "female",
          "email" : "rondagallegos@comvex.com",
          "phone" : "+1 (845) 438-2445",
          "street" : "969 Montgomery Street",
          "city" : "Salix",
          "state" : "Connecticut, 3041"
        }
      }
    ]
  }
}

Above result shows that two documents are retrieved and difference of max_score & individual score of first document indicate first document is some distant document.

Search multiple indices  : Multiple indices can be searched using comma separated values passed in URL. Below screenshot shows size to total documents in scope is 1008 and only 10 items are retrieved by default.

Search using request body:-  Search term within URL request body

Retrieve all documents in given index : Below query matches all documents present in index products.

➜  Desktop curl -XGET 'localhost:9200/products/_search?pretty' -d'
{
  "query": { "match_all": {} }
}' -H 'Content-Type: application/json'
{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 8,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "products",
        "_type" : "shoes",
        "_id" : "5",
        "_score" : 1.0,
        "_source" : {
          "name" : "VeftER",
          "size" : 6,
          "color" : "White"
        }
      },
      {
        ......
      },
      {
        .....
      }
    ]
  }
}

Note:-
- Score of all documents obtained will be 1.0, as all documents matches in Products index.
- Documents are not retrieved in same order as inserted via file(shoes.json), ordering of documents is nor guaranteed.

Retrieve documents of given size : Number of documents need to be retrieved is passed as input parameter in body. Elasticsearch is stateless (i.e: ELS maintains no open cursor or session for search operation) so search results are retrieved in one go and displayed (no pagination).

➜  Desktop curl -XGET 'localhost:9200/products/_search?pretty' -d'
{
  "query": { "match_all": {} },
  "size": 2
}' -H 'Content-Type: application/json'

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 8,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "products",
        "_type" : "shoes",
        "_id" : "5",
        "_score" : 1.0,
        "_source" : {
          "name" : "VeftER",
          "size" : 6,
          "color" : "White"
        }
      },
      {
        "_index" : "products",
        "_type" : "shoes",
        "_id" : "8",
        "_score" : 1.0,
        "_source" : {
          "name" : "PFFlys",
          "size" : 7,
          "color" : "White"
        }
      }
    ]
  }
}

Search/retrieves documents using from<N> and size<M> : From parameter indicates from where documents need to be retrieved and size indicates how many documents need to be returned.

➜  Desktop curl -XGET 'localhost:9200/products/_search?pretty' -d'
{
  "query": { "match_all": {} },
  "from": 3,
  "size": 2
}' -H 'Content-Type: application/json'
{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 8,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "products",
        "_type" : "shoes",
        "_id" : "4",
        "_score" : 1.0,
        "_source" : {
          "name" : "Puma",
          "size" : 7,
          "color" : "Red"
        }
      },
      {
        "_index" : "products",
        "_type" : "shoes",
        "_id" : "6",
        "_score" : 1.0,
        "_source" : {
          "name" : "Adidas",
          "size" : 7,
          "color" : "Red"
        }
      }
    ]
  }
}

Note:- match_all() return all documents ("total" : 8 ) matched in index products and then from position 3 two documents are returned. Execute <localhost:9200/products/_search?pretty> in browser and list all documents returned then validate why these two documents returned.

Search multiple indices  : Multiple indices can be searched using comma separated values passed in URL. Total 1008 documents is in scope of results obtained and 1 is displayed as requested.

➜  Desktop curl -XGET 'localhost:9200/products,customers/_search?pretty' -d'
{
  "query": { "match_all": {} },
  "size": 1
}' -H 'Content-Type: application/json'

{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 10,
    "successful" : 10,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1008,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "customers",
        "_type" : "vendors",
        "_id" : "tn_g6mABB3_D7Pc85hFv",
        "_score" : 1.0,
        "_source" : {
          "name" : "Lou Brooks",
          "age" : 70,
          "gender" : "female",
          "email" : "loubrooks@comvex.com",
          "phone" : "+1 (975) 536-2152",
          "street" : "159 Lester Court",
          "city" : "Roland",
          "state" : "New Jersey, 4742"
        }
      }
    ]
  }
}

Summary : We can execute all possible search queries executed via request body which are run via query param. Query param options are subset of options available in the request body.