For a long time, I thought Elasticsearch was the obvious answer.
It was fast. It scaled. Everyone I respected was using it.
So when our system started to feel slow, I added Elasticsearch without hesitation.
That decision worked… until it didn’t.
🚧 The problem I was actually solving
In production, our pain wasn’t search speed.
It was real-time correctness.
We had workflows where:
A user action needed to be reflected immediately
State changes had to be visible within milliseconds
“Eventually consistent” was not good enough to explain to customers or support teams
But at the time, I framed the problem incorrectly:
“Queries are slow → add Elasticsearch”
That framing cost us months.
🔍 What went wrong at scale
Elasticsearch did exactly what it promises:
fast, distributed search over indexed data.
What it does not promise:
Strong real-time guarantees
Immediate consistency after writes
Acting as a primary data source for transactional flows
In real production traffic, we started seeing:
Recently updated records missing from results
Edge cases where writes succeeded but reads lagged
Complex retry and refresh logic creeping into application code
Every bug was “rare”.
Together, they were constant.
Non-technical stakeholders didn’t care why it happened.
They just saw data that felt unreliable.
⚖️ The tradeoff I underestimated
Elasticsearch trades correctness now for performance at scale.
We needed:
Predictable reads after writes
🧠 The lesson that stuck with me
The mistake wasn’t using Elasticsearch.
The mistake was using it to compensate for a poorly defined problem.
Fast search ≠ real-time data retrieval
Indexing ≠ state management
Scalability ≠ correctness
Once we re-centered the architecture around:
A primary, strongly consistent data store
Clear read/write paths
Search used only where search made sense
The system became calmer.
Incidents dropped.
Engineers slept better.
🌱 How this changed how I design systems
Today, I’m much slower to introduce “powerful” infrastructure.
Before adding anything new, I ask:
What guarantee does this system give me?
What guarantee does it explicitly not give?
What failure will I be debugging six months from now?
Most production pain doesn’t come from lack of tools.
It comes from misaligned guarantees.
Closing thought
Elasticsearch is an excellent tool.
It just wasn’t the right one for a problem that required trust, not speed.
Experience has taught me this:
Architectural maturity isn’t about knowing more technologies — it’s about knowing when not to use them.
That lesson only comes from shipping, breaking things, and owning the consequences.
No comments:
Post a Comment