Enquiry Lab #02: Knowing Enough
Advertisement
An intimate 2-hour home salon on first-principles thinking, primary texts, and personal library exploration.
About This Session
After Lab #01, guests said they would love some technical topics too. So this time we look at three ideas that let big systems handle huge amounts of data, and we play each one out by hand. Each idea has a simple question behind it:
LSM trees and SSTables: how do you save data fast when there is far too much of it? You write it down quickly now, and sort and tidy it later.
HyperLogLog: how do you count millions of different visitors with a tiny memory? You keep a clever guess instead of a list.
Merkle trees, rolling hashes and content-defined chunking: how do two computers find the small difference between huge copies? They cut the data into pieces at points chosen by the content itself, so an insertion only disturbs nearby pieces, then compare a few fingerprints instead of everything.
The format is the same as before: 2 hours, up to 5 people and no screens. No technical background is needed.
How the evening runs (2 hours)
0:00 to 0:10 Arrival and a quick question to break the ice
0:10 to 0:30 Quiet reading of short excerpts from the original papers
0:30 to 1:00 Three small hands-on games, one for each idea
1:00 to 1:25 Open discussion: what these ideas say about how we remember, count and compare things in our own lives
1:25 to 2:00 Free reading: explore our library of 250+ books
Good to know
The games
1. The messy desk (LSM tree and SSTable). Guests call out words, and the host writes each one on a sticky note in their hand. When the hand holds five notes, sort them in alphabetical order and file them as one card: that card is an SSTable. To look up a word, check the hand first, then the cards from newest to oldest. When there are three cards, merge them into one sorted card by walking through them side by side. To delete a word, write "deleted" on a new note.
Discover: writing is fast because nothing is sorted at write time.
Discover: reading gets slower as cards pile up, which is why merging matters.
2. Coin-flip counting (HyperLogLog). Each person flips a coin until it lands heads and writes down how many tails came first. Repeat about ten times. The group keeps only the longest run of tails anyone got. If the longest run is k tails, the number of flips was roughly 2 to the power k. Then split into two groups, estimate in each, and average the results to see the guess get steadier.
Discover: you can estimate a count without remembering a single item.
Discover: one lucky run can throw the guess off, so real systems average many small groups.
3. Spot the difference (rolling hash, chunking and Merkle tree). This game has two parts.
First, cut a sentence into chunks. Use A = 1, B = 2 and so on. Slide a window three letters wide along the sentence. Each time it moves, take away the letter that leaves and add the one that enters; this is a rolling hash. Wherever the window's total divides evenly by 7, cut the sentence there. Now add a word at the start and cut the new sentence the same way.
Discover: the cuts after the new word land in the same places, so only the first chunk changes. Cutting every fixed number of letters would have shifted every chunk.
Discover: the rolling update means each move costs one subtraction and one addition.
Then, find the changed chunk. Write each chunk's total on a card, giving two rows of cards, old and new. Add the cards in pairs, then add those totals in pairs, until each row has one total at the top. If the top totals differ, compare the two halves, and keep going down only the half that differs.
Discover: with eight chunks, the change is found in three steps instead of eight.
Discover: one top total tells you whether two huge copies match at all.
Discussion
LSM trees write first and tidy later. Where in your own life do you do the same, and what does it cost you?
HyperLogLog gives a good guess instead of an exact count. When is a guess good enough?
A Merkle tree trusts a single total at the top. What do we lose when we trust a summary instead of looking at everything?
How does each idea compare with Percy's warning about trusting the package instead of the thing itself?
Free-reading shelf
Designing Data-Intensive Applications, Martin Kleppmann
Database Internals, Alex Petrov
The Information, James Gleick
Code, Charles Petzold
The Message in the Bottle, Walker Percy
Up to 5 people. No technical background needed; the games explain everything. No screens: phones go in a basket at the door. Readings, pencils and tea are provided.
HSR Layout, Bengaluru. Exact address shared with approved guests.
About This Session
After Lab #01, guests said they would love some technical topics too. So this time we look at three ideas that let big systems handle huge amounts of data, and we play each one out by hand. Each idea has a simple question behind it:
LSM trees and SSTables: how do you save data fast when there is far too much of it? You write it down quickly now, and sort and tidy it later.
HyperLogLog: how do you count millions of different visitors with a tiny memory? You keep a clever guess instead of a list.
Merkle trees, rolling hashes and content-defined chunking: how do two computers find the small difference between huge copies? They cut the data into pieces at points chosen by the content itself, so an insertion only disturbs nearby pieces, then compare a few fingerprints instead of everything.
The format is the same as before: 2 hours, up to 5 people and no screens. No technical background is needed.
How the evening runs (2 hours)
0:00 to 0:10 Arrival and a quick question to break the ice
0:10 to 0:30 Quiet reading of short excerpts from the original papers
0:30 to 1:00 Three small hands-on games, one for each idea
1:00 to 1:25 Open discussion: what these ideas say about how we remember, count and compare things in our own lives
1:25 to 2:00 Free reading: explore our library of 250+ books
Good to know
The games
1. The messy desk (LSM tree and SSTable). Guests call out words, and the host writes each one on a sticky note in their hand. When the hand holds five notes, sort them in alphabetical order and file them as one card: that card is an SSTable. To look up a word, check the hand first, then the cards from newest to oldest. When there are three cards, merge them into one sorted card by walking through them side by side. To delete a word, write "deleted" on a new note.
Discover: writing is fast because nothing is sorted at write time.
Discover: reading gets slower as cards pile up, which is why merging matters.
2. Coin-flip counting (HyperLogLog). Each person flips a coin until it lands heads and writes down how many tails came first. Repeat about ten times. The group keeps only the longest run of tails anyone got. If the longest run is k tails, the number of flips was roughly 2 to the power k. Then split into two groups, estimate in each, and average the results to see the guess get steadier.
Discover: you can estimate a count without remembering a single item.
Discover: one lucky run can throw the guess off, so real systems average many small groups.
3. Spot the difference (rolling hash, chunking and Merkle tree). This game has two parts.
First, cut a sentence into chunks. Use A = 1, B = 2 and so on. Slide a window three letters wide along the sentence. Each time it moves, take away the letter that leaves and add the one that enters; this is a rolling hash. Wherever the window's total divides evenly by 7, cut the sentence there. Now add a word at the start and cut the new sentence the same way.
Discover: the cuts after the new word land in the same places, so only the first chunk changes. Cutting every fixed number of letters would have shifted every chunk.
Discover: the rolling update means each move costs one subtraction and one addition.
Then, find the changed chunk. Write each chunk's total on a card, giving two rows of cards, old and new. Add the cards in pairs, then add those totals in pairs, until each row has one total at the top. If the top totals differ, compare the two halves, and keep going down only the half that differs.
Discover: with eight chunks, the change is found in three steps instead of eight.
Discover: one top total tells you whether two huge copies match at all.
Discussion
LSM trees write first and tidy later. Where in your own life do you do the same, and what does it cost you?
HyperLogLog gives a good guess instead of an exact count. When is a guess good enough?
A Merkle tree trusts a single total at the top. What do we lose when we trust a summary instead of looking at everything?
How does each idea compare with Percy's warning about trusting the package instead of the thing itself?
Free-reading shelf
Designing Data-Intensive Applications, Martin Kleppmann
Database Internals, Alex Petrov
The Information, James Gleick
Code, Charles Petzold
The Message in the Bottle, Walker Percy
Up to 5 people. No technical background needed; the games explain everything. No screens: phones go in a basket at the door. Readings, pencils and tea are provided.
HSR Layout, Bengaluru. Exact address shared with approved guests.
Advertisement
Where is it happening?
Bengaluru, India
Event Location & Nearby Stays:
Know what’s Happening Next — before everyone else does.
Host or PublisherShorya Kumar











