Module 4: Analytical and NoSQL databases · Lesson 4.4 · about 30 min
Elasticsearch, and choosing a store
Inverted indexes, analysers, relevance, shards and the ELK stack; then a decision table for picking the right store, and the Module 4 mini project.
Module 4 · about 30 minutes · read first: 4.1. You’ll be able to explain how an inverted index makes text search fast, describe indexes, documents, mappings, shards and replicas, and choose between BigQuery, Druid, Bigtable, Elasticsearch, Redis and Postgres for a requirement.
1. What Elasticsearch is
Elasticsearch is a distributed search and analytics engine built on the Apache Lucene library. You send it JSON documents; it indexes every field so you can search text, filter, sort and count in milliseconds. It powers site search, log analytics and security monitoring. OpenSearch is the open-source fork, and Apache Solr is an older Lucene-based sibling.
To find “Kafka” in a 900-page book you don’t read every page; you look in the index at the back, which lists the pages. An inverted index is that, for every word in every document. Elasticsearch also “cooks” the words first (lower-casing, dropping “the”, turning “running” into “run”), so “Running Shoes” finds “run shoe”.
2. Watch an index being built
Three products go into a tiny index: doc 1 “Blue Rain Jacket”, doc 2 “Rain Boots for Kids”, doc 3 “Blue Running Shoes”.
3. Core ideas
- Index: a collection of documents (like a table). Document: one JSON object. Mapping: field types:
textfor full-text search,keywordfor exact filters and counts, numbers, dates, anddense_vectorfor embeddings. - Analysers: split and normalise text; language analysers, synonyms (“tee” = “t-shirt”), and n-grams for search-as-you-type.
- Relevance: results are scored (BM25 by default); you can boost fields (title over description) and business signals (popularity, stock).
- Shards and replicas: an index is split into primary shards spread over nodes, each with replica copies for safety and extra read capacity.
- Near real-time: new documents become searchable after a short refresh, about a second by default.
GET products/_search
{
"query": { "bool": {
"must": { "match": { "name": { "query": "rain jaket", "fuzziness": "AUTO" } } },
"filter": [ { "term": { "size": "M" } }, { "range": { "price": { "lte": 80 } } } ] } },
"aggs": { "by_brand": { "terms": { "field": "brand" } } }
}
That finds “rain jacket” despite the typo, filters by size and price, and returns brand counts for the side panel (often called facets).
How many shards? Too few and each one grows huge and slow to move or recover; too many and the cluster wastes memory managing them. Elastic’s sizing guidance suggests shards of tens of GB each; check the current docs for your version.
4. The ELK stack and logs
Elasticsearch, Logstash and Kibana (plus small Beats shippers) became the standard for central logs: every server sends its logs, Elasticsearch indexes them, Kibana searches and charts them. Data platforms use it to search pipeline logs and errors; security teams use it to spot attacks. Rivals include Splunk, Datadog, Grafana Loki and Google Cloud Logging.
It is not a system of record: keep the source of truth elsewhere and rebuild the index from it when needed. It is also not for heavy joins or big batch analytics. It now does vector and hybrid search for AI as well (Lesson 6.3).
5. Choosing a store: the decision table
| Requirement | First choice | Why |
|---|---|---|
| Analysts’ ad hoc SQL over years of data | BigQuery (or Snowflake) | Scales, any question, pay per use |
| Live dashboards for thousands of outside users, seconds fresh | Druid or Pinot (ClickHouse if simpler running matters) | Low delay with many users, streaming ingestion |
| One key’s data in milliseconds, huge volume | Bigtable (Cassandra, DynamoDB) | Fast key access, massive throughput |
| Text search, typos, facets, relevance | Elasticsearch / OpenSearch | Inverted index and relevance scoring |
| Tiny, hot data: sessions, rate limits, caches | Redis / Memorystore | In memory, microseconds |
| App transactions and modest analytics | PostgreSQL / Cloud SQL / AlloyDB | Transactions, SQL, joins, cheap at small scale |
| Global transactions at scale | Spanner | Strongly consistent, distributed SQL |
Most large platforms use several, all fed from the same lake or Kafka topics, each shaped for its own question.
- Inverted index
- A list from each word to the documents that contain it, like the index of a book.
- Analyser
- The steps that split and normalise text, used both when indexing and when searching.
- Relevance score
- How well a document matches; rare matched words count more.
- Shard / replica
- One slice of an index / a full copy of a slice on another node.
- Facet
- Counts by a field (brand, size) shown beside results to narrow a search.
- Physical meaning
- When a shopper types "rain jaket" on a retailer's site and still sees jackets, sorted with the in-stock ones first and brand counts on the side, an inverted index, an analyser, fuzzy matching and a relevance score have all run in a few milliseconds.
“Elasticsearch can be our main database, since it stores everything anyway.” It stores documents, but it is built for search, not as the system of record: it has no multi-document transactions, and mapping changes often mean rebuilding the index. Keep the truth in a database or the lake and treat the search index as a copy you can rebuild at any time.
6. Mini project for Module 4: three stores, one dataset
Brief (3 hours, all local with Docker).
- Generate one day of synthetic ad events (campaign, placement, device, user, impressions, clicks, spend) and a product catalogue of 5,000 items.
- Analytical: load the events into ClickHouse or Druid (both have single-container quick starts) and build an hourly campaign query that runs in under a second. Try it with and without rollup or a pre-aggregated table.
- Key-value: load each user’s “last 10 ads seen” into Redis (or the Bigtable emulator,
gcloud beta emulators bigtable start) with a well-designed key. - Search: index the catalogue in OpenSearch or Elasticsearch and get a fuzzy query plus brand facets working.
- README: one table comparing load time, query time and what each store cannot do.
Checklist for “done”:
- One script loads all three stores from the same generated files.
- The hourly campaign query returns in under a second, with and without rollup timed.
- A user’s last 10 ads come back with one key lookup.
- “rain jaket” finds the rain jackets in the catalogue.
Acceptance checks: totals from the analytical store match a plain count over the raw file; the README names one question each store refuses.
New words: Lucene, inverted index, analyser, document, mapping, keyword field, relevance, BM25, fuzziness, facet, shard, replica, refresh interval, ELK stack, OpenSearch.
- An inverted index maps each word to the documents containing it, which makes text search a lookup.
- Analysers cook text the same way at index and search time; fuzziness forgives typos; BM25 ranks results.
- Indexes are split into primary shards with replicas; total shards = primaries × (1 + replicas).
- Elasticsearch is a search copy, not the system of record: keep the truth elsewhere and rebuild when needed.
- Choose stores by question: BigQuery for ad hoc SQL, Druid for live dashboards, Bigtable for key lookups, Elasticsearch for search, Redis for tiny hot data, Postgres for transactions.
Check yourself
Sources
- Elastic documentation: Elasticsearch guide https://www.elastic.co/guide/index.html
- OpenSearch documentation https://opensearch.org/docs/latest/
- Manning, C. D., Raghavan, P. and Schütze, H. (2008), Introduction to Information Retrieval, Cambridge University Press, Chapters 1 and 6 https://nlp.stanford.edu/IR-book/
- Robertson, S. and Zaragoza, H. (2009), The Probabilistic Relevance Framework: BM25 and Beyond, Foundations and Trends in Information Retrieval 3(4)
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. All company data in the examples is synthetic. Tool versions and prices change: check the official docs before you rely on them.