Yandex Open-Sources System That Processes Clicks and Likes Without Waiting in a Queue
Yandex has open-sourced YTsaurusFlow, a technology that processes data streams immediately instead of waiting several hours.
Typically, data on clicks, likes and other user actions is accumulated first and processed later, which can take several hours. The new technology works differently.
The system is designed for high workloads: it processes each event exactly once and continues operating in the event of failures. Flow runs as part of YTsaurus, an open-source platform for storing and processing large volumes of data.
Yandex already uses the technology internally. At Yandex Market, it speeds up statistics collection; in antifraud systems, it helps detect malicious actors; and at Yandex Advertising, it helps select relevant offers. In the advertising service, the technology processes more than 100 GB of data and millions of events per second.
