MurmurHash

Setting compression.type is for applies compression at the batch level, not the individual message level.

Oh, because the LZ4 algorithm is looking at a massive block of similar JSON sensor data all at once.

Precisely.

It can find redundant patterns much more effectively than if it were inspecting tiny isolated messages.

This represents a collision between data modeling and infrastructure physics, commonly known as the hot key problem.

The hot key problem.

When a producer sends a message with a key like Custerid, the Kafka producer client applies an algorithm, specifically Murmur2, to hash that key into an integer.

It then performs a modulo operation against the total number of partitions.