TECHNICAL MEMORANDUMTM-26-010
SUMMARY. In 2012 we pitched a predictive-targeting provider on analysing its complete event stream live: 20,000 events per second on one machine. Making that claim work took more than a fast algorithm. I implemented the LMAX Disruptor design, worked close to the hardware, and tuned the network-card buffers under Linux. Mikio Braun's decaying counters maintained user-interest profiles over several time frames. We also had to persuade the customer's administrators to give us a bare-metal server in an otherwise virtualized environment. The result worked. It was an engineered technical demonstration, not yet a repeatable product.
1. BACKGROUND. Mikio Braun and I founded TWIMPACT out of TU Berlin in 2009; the product later became streamdrill. We built an in-memory analytics engine that updated trends, rankings, and profiles while events arrived. The prospective customer had a useful question: could it distinguish immediate purchase intent from longer-term interest for every user, across its entire stream?
For scale, Twitter reported 340 to more than 400 million tweets per day in 2012. Its 10% Gardenhose therefore averaged roughly 390 to 460 tweets per second. Analysing that feed already counted as big-data work. Our customer supplied 20,000 events per second and a better use for the answer. Its raw event rate was much higher; the tweets required more parsing. This is period context, not a comparison of processing cost.
2. FINDINGS.
2.1 The complete customer stream ran at 20,000 events per second. We processed it live on one bare-metal server.
2.2 The customer's administrators normally operated virtualized infrastructure. Giving our demonstration a physical machine was an exception and a distraction for them. We had to argue for it.
2.3 I implemented the LMAX Disruptor design and tuned buffer settings on the Linux network card. The hot path was designed around network, cache, and CPU behaviour rather than treated as ordinary application code.
2.4 Mikio's decaying-counter implementation maintained profiles over several time frames. A washing machine may signal intent today. Once bought, it may be irrelevant for another ten years.
2.5 The exponential-decay core was released publicly in 2015 as BSD-licensed Scala. It tracks millions of objects over configurable half-times and supports secondary indices. The code credits Mikio.
https://github.com/streamdrill/streamdrill-core
2.6 We demonstrated the applications at the customer's conference. The customer was then acquired, and the prospective deal ended.
3. CONCLUSIONS. We proved that the full stream could be analysed live on one machine. We did not prove that we had a product. The result belonged to the whole arrangement: an algorithm designed for the question, a data path designed around the machine, and bare-metal hardware we had fought to obtain.
That is a valid technical result, but not yet a repeatable deployment. The next job would have been to preserve its performance inside an environment the customer's administrators could operate as standard. The acquisition ended the opportunity before we crossed that gap.
4. ACTION.
4.1 State exactly what a demonstration proves.
4.2 Record every condition that made the result possible.
4.3 Treat exceptions to the customer's operating model as product work still to be done.
4.4 Design the algorithm, data path, and hardware as one system.
4.5 Plan the route from a successful demonstration to a deployment someone else can operate.