Debugging Elasticsearch queries with Kibana

When writing or debugging queries with Postgres or another relational database, it’s often helpful to use a database client like psql to view the data directly in the database. However, when using Elasticsearch we can sometimes feel lost trying to understand why a query returned some particular records, or even the document structure of the results. Kibana is a user interface for Elasticsearch that is used for data analysis and management. This article provides a brief tutorial on how to use one specific feature of Kibana, the Dev Tools Console, to more easily view your Elasticsearch data.

Kibana Installation

See the official Elasticsearch guide for installing Kibana. Once it is installed, make sure Elasticsearch is running, and then start Kibana by running kibana on the command line. Kibana will be available at http://localhost:5601.

Using Kibana Dev Tools to view Elasticsearch results

Access the Dev Tools by clicking on the hamburger menu in the upper left hand corner and going to Dev Tools in the Management section. In the left-hand panel of the console enter the following query, replacing “occupations_development” with the name of your index.

GET occupations_development/_search
{
  "query": {
    "match_all": {}
  }
}

Click the green arrow to run the query and you will see the results of the query in the panel on the right.

In the hits section, we can see in the value field that we have 1207 records stored in our local Elasticsearch database for the occupations_development index. The records themselves are available in the nested hits array, although by default Elasticsearch will just return the first 10 records. For this particular Elasticsearch index, we can see that the title, rapids_code, and onet_code fields of our occupation data are being mapped.

To see more records, we can add a size parameter to the query:

GET occupations_development/_search
{
  "query": {
    "match_all": {}
  },
  "size": 20
}

Now you can start adding more complex queries that you may be working with in your own code that you may need help with debugging. Here is a slightly more complex example that uses a bool query and the minimum_should_match parameter to return records that either match “soft” in the title or have an ONET code that matches “51”.

GET occupations_development/_search
{
  "query": {
    "bool": {
      "should": [
        {
          "match": {
            "title": {
              "query": "Soft"
            }
          }
        },
        {
          "match": {
            "onet_code": {
              "query": "51"
            }
          }
        }
      ],
      "minimum_should_match": 1
    }
  }
}

We can see that we have 440 results in our Elasticsearch index that match the conditions. The results also provide a score for each record. The first record returned has the highest score of 7.0680957, since it matches both the title and the onet_code.

Elasticsearch query display

The Kibana Dev Tools Console has a couple of nice formatting features to help with query readability. Clicking the small wrench icon allows you to auto indent the JSON structure. You can also collapse parts of your request to help focus on a particular section while you are fine-tuning your query.

Summary

Using the Kibana Dev Tools can help provide insights into your Elasticsearch query results, improve your productivity, and reduce frustration! If you use a Procfile in development, you can easily add a process for kibana so that it will be readily available anytime you find yourself working on Elasticsearch query development.