Berlin Buzzwords: Measuring 2.0 in production

At the Berlin Buzzwords 2017 conference I had the chance to talk about our new measuring infrastructure at bol.com. The talk continued the story of the Measurement 2.0 project which was introduced by Niels Basjes at Berlin Buzzwords 2016. In the original talk the infrastructure was conceptually introduced and this talk elaborated further by showing the experiences we had with running it in production on real user behavior.Bellow we provide the talks along with their abstracts.

Measuring 2.0 – Building the next generation webanalytics solution using Apache Flink.

At bol.com (the biggest online retailer in the Benelux) we want to help the customer find what they wanted. Automating this process requires us to understand what products/promotions we showed them and which of those made them happy. With the fine grained personalization that has been introduced over the last few years we see that just measuring ‘what page’ we showed (like all the standard web analytics systems do) is no longer enough. So we need something different. In order to get a solution that will support our business for the coming years we raised the bar to the top: Measure everything and analyze in near-realtime.

In this talk Niels Basjes explains the project “Measuring 2.0”, our next generation web analytics measuring and processing stack, which went live in January 2017. Niels goes into

• the custom built measuring system that will produce over 50000 measurements per second
• the processing system and the algorithms implemented with Apache Flink
• why we did not choose Storm or Spark for this task
• the development and operational hurdles needed to make this type of solution run in production
• the architectural concepts to make this data available in the personalization services we have

https://youtu.be/qwbGQDpxmVg

Half a year of Measuring 2.0 in production at bol.com

How do you measure the behavior of users firing 3,000 requests per second?

At bol.com, there are thousands of users visiting the site daily, and my team provides the measurement infrastructure providing near real time metrics in the interactions of these customers with the platform.

Our new measuring system, Measuring 2.0 is built using open-source big data technologies like Apache Flink, Kafka, Avro, and Parquet, supported with in-house technology.

In this talk, which was presented at Berlin Buzzwords 2017, we dive into the technical implementation of Measuring 2.0 and share our experiences of running it in production. During the talk I go through how we kept our Kafka cluster resilient enough to handle our peak loads, how we used Apache Flink to show trending products on our website in real-time, how we handle continuous changes in our measurement data structure using Avro, how we use Parquet to deal with the great quantities of data while also keeping them easily accessible, and share the lessons learned along the way.

https://www.youtube.com/watch?v=3OiWGTlQBYQ