使用python操作elasticsearch

发表于 2016-12-13 更新于 2018-03-27

和你们这些少爷不同，我们光是活着就竭尽全力了。

Lately, here at Tryolabs, we started gaining interest in big data and search related platforms which are giving us excellent resources to create our complex web applications. One of them is Elasticsearch. Elastic{ON}15, the first ES conference is coming, and since nowadays we see a lot of interest in this technology, we are taking the opportunity to give an introduction and a simple example for Python developers out there that want to begin using it or give it a try. ### 1. What is Elasticsearch?

Elasticsearch is a distributed, real-time, search and analytics platform.

2. Yeah, but what IS Elasticsearch?

Good question! In the previous definition you can see all these hype-sounding tech terms (distributed, real-time, analytics), so let’s try to explain.
ES is distributed, it organizes information in clusters of nodes, so it will run in multiple servers if we intend it to.
ES is real-time, since data is indexed, we get responses to our queries super fast!
And last but not least, it does searches and analytics. The main problem we are solving with this tool is exploring our data!
A platform like ES is the foundation for any respectable search engine.

3. How does it work?

Using a restful API, Elasticsearch saves data and indexes it automatically. It assigns types to fields and that way a search can be done smartly and quickly using filters and different queries.
It’s uses JVM in order to be as fast as possible. It distributes indexes in “shards” of data. It replicates shards in different nodes, so it’s distributed and clusters can function even if not all nodes are operational. Adding nodes is super easy and that’s what makes it so scalable.
ES uses Lucene to solve searches. This is quite an advantage with comparing with, for example, Django query strings. A restful API call allows us to perform searches using json objects as parameters, making it much more flexible and giving each search parameter within the object a different weight, importance and or priority.
The final result ranks objects that comply with the search query requirements. You could even use synonyms, autocompletes, spell suggestions and correct typos. While the usual query strings provides results that follow certain logic rules, ES queries give you a ranked list of results that may fall in different criteria and its order depend on how they comply with a certain rule or filter.
ES can also provide answers for data analysis, like averages, how many unique terms and or statistics. This could be done using aggregations. To dig a little deeper in this feature check the documentation here.

4. Should I use ES?

The main point is scalability and getting results and insights very fast. In most cases using Lucene could be enough to have all you need.
It seems sometimes that these tools are designed for projects with tons of data and are distributed in order to handle tons of users. Startups dream of growing to that scenario, but may start thinking small first to build a prototype and then when the data is there, start thinking about scaling problems.
Does it make sense and pays off to be prepared to grow A LOT? Why not? Elasticsearch has no drawback and is easy to use, so it’s just a decision of using it to be prepared for the future.
I’m going to give you a quick example of a dead simple project using Elasticsearch to quickly and beautifully search for some example data. It will be quick to do, Python powered and ready to scale in case we need it to, so, best of both worlds.

5. Easy first steps with ES

For the following part it would be nice to be familiarized with concepts like Cluster, Node, Document, Index. Take a look at the official guide if you have doubts.
First things first, get ES from here.
I followed this video tutorial to get things started in just a minute. I recommend all you to check it out later.
Once you downloaded ES, it’s as simple as running bin/elasticsearch and you will have your ES cluster with one node running! You can interact with it at http://localhost:9200/
If you hit it you will get something like this:

{
  "status" : 200,
  "name" : "Delphine Courtney",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "1.4.2",
    "build_hash" : "927caff6f05403e936c20bf4529f144f0c89fd8c",
    "build_timestamp" : "2014-12-16T14:11:12Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.2"
  },
  "tagline" : "You Know, for Search"
}

view rawes_first.json hosted with ❤ by GitHub
Creating another node is as simple as:
bin/elasticsearch -Des.node.name=Node-2
It automatically detects the old node as its master and joins our cluster. By default we will be able to communicate with this new node using the 9201 port http://localhost:9201. Now we can talk with each node and receive the same data, they are supposed to be identical.

6. Let’s Pythonize this thing!

To use ES with our all time favorite language; Python, it gets easier if we install elasticsearch-py package.
pip install elasticsearch
Now we will be able to use this package to index and search data using Python.

7. Let’s add some public data to our cluster

So, I wanted to make this project a “real world example”, I really did, but after I found out there is a star wars API (http://swapi.co/), I couldn’t resist it and ended up being a fictional - ”galaxy far far away” example. The API is dead simple to use, so we will get some data from there.
I’m using an IPython Notebook to do this test, I started with the sample request to make sure we can hit the ES server.

make sure ES is up and running

1
2
3

import requests
res = requests.get('http://localhost:9200')
print(res.content)

view rawes_first.py hosted with ❤ by GitHub
Then we connect to our ES server using Python and the elasticsearch-py library:
#connect to our cluster

1 2	from elasticsearch import Elasticsearch es = Elasticsearch([{'host': 'localhost', 'port': 9200}])

view rawes_first3.py hosted with ❤ by GitHub
I added some data to test, and then deleted it. I’m skipping that part for this guide, but you can check it out in the notebook.
Now, using The Force, we connect to the Star Wars API and index some fictional people.
#let’s iterate over swapi people documents and index them

import json
r = requests.get('http://localhost:9200') 
i = 1
while r.status_code == 200:
    r = requests.get('http://swapi.co/api/people/'+ str(i))
    es.index(index='sw', doc_type='people', id=i, body=json.loads(r.content))
    i=i+1
 
print(i)

view rawes_first4.py hosted with ❤ by GitHub
Please, notice that we automatically created an index “sw” and a “doc_type” with de indexing command. We get 17 responses from swapi and index them with ES. I’m sure there are much more “people” in the swapi DB, but it seems we are getting a 404 with http://swapi.co/api/people/17. Bug report here! :-)
Anyway, to see if all worked with this few results, we try to get the document with id=5.
es.get(index=’sw’, doc_type=’people’, id=5)
view rawes_first10.py hosted with ❤ by GitHub
We will get Princess Leia:

{u'_id': u'5',
 u'_index': u'sw',
 u'_source': {u'birth_year': u'19BBY',
  u'created': u'2014-12-10T15:20:09.791000Z',
  u'edited': u'2014-12-20T21:17:50.315000Z',
  u'eye_color': u'brown',
  u'films': [u'http://swapi.co/api/films/1/',
   u'http://swapi.co/api/films/2/',
   u'http://swapi.co/api/films/3/',
   u'http://swapi.co/api/films/6/'],
  u'gender': u'female',
  u'hair_color': u'brown',
  u'height': u'150',
  u'homeworld': u'http://swapi.co/api/planets/2/',
  u'mass': u'49',
  u'name': u'Leia Organa',
  u'skin_color': u'light',
  u'species': [u'http://swapi.co/api/species/1/'],
  u'starships': [],
  u'url': u'http://swapi.co/api/people/5/',
  u'vehicles': [u'http://swapi.co/api/vehicles/30/']},
 u'_type': u'people',
 u'_version': 1,
 u'found': True}

view rawes_first5.py hosted with ❤ by GitHub
Now, let’s add more data, this time using node 2! And let’s start at the 18th person, where we stopped.

r = requests.get('http://localhost:9201')
i = 18
while r.status_code == 200:
   r = requests.get('http://swapi.co/api/people/'+ str(i))
   es.index(index='sw', doc_type='people', id=i,     body=json.loads(r.content))
   i=i+1

view rawes_first6.py hosted with ❤ by GitHub
We got the rest of the characters just fine.

8. Now, let’s try an interesting search

Where is Darth Vader? Here is our search query:
es.search(index=”sw”, body={“query”: {“match”: {‘name’:’Darth Vader’}}})
view rawes_first7.py hosted with ❤ by GitHub
This will give us both Darth Vader AND Darth Maul. Id 4 and id 44 (notice that they are in the same index, even if we use different node client call the index command). Both results have a score, although Darth Vader is much higher than Darth Maul (2.77 vs 0.60) since Vader is a exact match. Take that Darth Maul!

{u'_shards': {u'failed': 0, u'successful': 5, u'total': 5},
 u'hits': {u'hits': [{u'_id': u'4',
    u'_index': u'sw',
    u'_score': 2.7754524,
    u'_source': {u'birth_year': u'41.9BBY',
     u'created': u'2014-12-10T15:18:20.704000Z',
     u'edited': u'2014-12-20T21:17:50.313000Z',
     u'eye_color': u'yellow',
     u'films': [u'http://swapi.co/api/films/1/',
      u'http://swapi.co/api/films/2/',
      u'http://swapi.co/api/films/3/',
      u'http://swapi.co/api/films/6/'],
     u'gender': u'male',
     u'hair_color': u'none',
     u'height': u'202',
     u'homeworld': u'http://swapi.co/api/planets/1/',
     u'mass': u'136',
     u'name': u'Darth Vader',
     u'skin_color': u'white',
     u'species': [u'http://swapi.co/api/species/1/'],
     u'starships': [u'http://swapi.co/api/starships/13/'],
     u'url': u'http://swapi.co/api/people/4/',
     u'vehicles': []},
    u'_type': u'people'},
   {u'_id': u'44',
    u'_index': u'sw',
    u'_score': 0.6085256,
    u'_source': {u'birth_year': u'54BBY',
     u'created': u'2014-12-19T18:00:41.929000Z',
     u'edited': u'2014-12-20T21:17:50.403000Z',
     u'eye_color': u'yellow',
     u'films': [u'http://swapi.co/api/films/4/'],
     u'gender': u'male',
     u'hair_color': u'none',
     u'height': u'175',
     u'homeworld': u'http://swapi.co/api/planets/36/',
     u'mass': u'80',
     u'name': u'Darth Maul',
     u'skin_color': u'red',
     u'species': [u'http://swapi.co/api/species/22/'],
     u'starships': [u'http://swapi.co/api/starships/41/'],
     u'url': u'http://swapi.co/api/people/44/',
     u'vehicles': [u'http://swapi.co/api/vehicles/42/']},
    u'_type': u'people'}],
  u'max_score': 2.7754524,
  u'total': 2},
 u'timed_out': False,
 u'took': 44}

view rawes_first8.py hosted with ❤ by GitHub
So, this query will give us results if the word is contained exactly in our indexed data. What if we want to build some kind of autocomplete input where we get the names that contain the characters we are typing?
There are many ways to do that and another great number of queries. Take a look here to learn more. I picked this one to get all documents with prefix “lu” in their name field:
es.search(index="sw", body={"query": {"prefix" : { "name" : "lu" }}})
view rawes_first9.py hosted with ❤ by GitHub
We will get Luke Skywalker and Luminara Unduli, both with the same 1.0 score, since they match with the same 2 initial characters.

{u'_shards': {u'failed': 0, u'successful': 5, u'total': 5},
 u'hits': {u'hits': [{u'_id': u'1',
    u'_index': u'sw',
    u'_score': 1.0,
    u'_source': {u'birth_year': u'19BBY',
     u'created': u'2014-12-09T13:50:51.644000Z',
     u'edited': u'2014-12-20T21:17:56.891000Z',
     u'eye_color': u'blue',
     u'films': [u'http://swapi.co/api/films/1/',
      u'http://swapi.co/api/films/2/',
      u'http://swapi.co/api/films/3/',
      u'http://swapi.co/api/films/6/'],
     u'gender': u'male',
     u'hair_color': u'blond',
     u'height': u'172',
     u'homeworld': u'http://swapi.co/api/planets/1/',
     u'mass': u'77',
     u'name': u'Luke Skywalker',
     u'skin_color': u'fair',
     u'species': [u'http://swapi.co/api/species/1/'],
     u'starships': [u'http://swapi.co/api/starships/12/',
      u'http://swapi.co/api/starships/22/'],
     u'url': u'http://swapi.co/api/people/1/',
     u'vehicles': [u'http://swapi.co/api/vehicles/14/',
      u'http://swapi.co/api/vehicles/30/']},
    u'_type': u'people'},
   {u'_id': u'64',
    u'_index': u'sw',
    u'_score': 1.0,
    u'_source': {u'birth_year': u'58BBY',
     u'created': u'2014-12-20T16:45:53.668000Z',
     u'edited': u'2014-12-20T21:17:50.455000Z',
     u'eye_color': u'blue',
     u'films': [u'http://swapi.co/api/films/5/',
      u'http://swapi.co/api/films/6/'],
     u'gender': u'female',
     u'hair_color': u'black',
     u'height': u'170',
     u'homeworld': u'http://swapi.co/api/planets/51/',
     u'mass': u'56.2',
     u'name': u'Luminara Unduli',
     u'skin_color': u'yellow',
     u'species': [u'http://swapi.co/api/species/29/'],
     u'starships': [],
     u'url': u'http://swapi.co/api/people/64/',
     u'vehicles': []},
    u'_type': u'people'}],
  u'max_score': 1.0,
  u'total': 2},
 u'timed_out': False,
 u'took': 9}

view rawes_first11.py hosted with ❤ by GitHub
There are many other interesting queries we can do. If, for example, we want to get all elements similar in some way, for a related or correction search we can use something like this:
es.search(index=”sw”, body={“query”:
{“fuzzy_like_this_field” : { “name” :
{“like_text”: “jaba”, “max_query_terms”:5}}}})
view rawes_first1.py hosted with ❤ by GitHub
And we got Jabba although we had a typo in our search query. That is powerful!

{u'_shards': {u'failed': 0, u'successful': 5, u'total': 5},
 u'hits': {u'hits': [{u'_id': u'16',
    u'_index': u'sw',
    u'_score': 1.5700331,
    u'_source': {u'birth_year': u'600BBY',
     u'created': u'2014-12-10T17:11:31.638000Z',
     u'edited': u'2014-12-20T21:17:50.338000Z',
     u'eye_color': u'orange',
     u'films': [u'http://swapi.co/api/films/1/',
      u'http://swapi.co/api/films/3/',
      u'http://swapi.co/api/films/4/'],
     u'gender': u'hermaphrodite',
     u'hair_color': u'n/a',
     u'height': u'175',
     u'homeworld': u'http://swapi.co/api/planets/24/',
     u'mass': u'1,358',
     u'name': u'Jabba Desilijic Tiure',
     u'skin_color': u'green-tan, brown',
     u'species': [u'http://swapi.co/api/species/5/'],
     u'starships': [],
     u'url': u'http://swapi.co/api/people/16/',
     u'vehicles': []},
    u'_type': u'people'}],
  u'max_score': 1.5700331,
  u'total': 1},
 u'timed_out': False,
 u'took': 40}

view rawes_first2.py hosted with ❤ by GitHub

9. Next Steps

This was just a simple overview on how to set up your Elasticsearch server and start working with some data using Python. The code used here is publicly available in this IPython notebook.
We encourage you to learn more about ES and specially take a look at the Elastic stack where you will be able to see beautiful analytics and insights with Kibana and go through logs using Logstash.
In following posts we will talk about more advanced ES features and we will try to extend this simple test and use it to show a more interesting Django app powered by this data and by ES.
Hope this post was useful for developers trying to enter the ES world.
At Tryolabs we’re Elastic official partners. If you want to talk about Elasticsearch, ELK, applications and possible projects using these technologies, drop us a line to hello@tryolabs.com (or fill out this form) and we will be glad to connect!