Today we have the pleasure to share something truly special. In the video below is our first in-depth presentation of Perper (https://github.com/obecto/perper) – Obecto’s next-gen stream processing framework used as a backbone for the Apocryph.Network blockchain.
The presentation was given at a webinar organized by GridGain Systems, the main contributors to the Apache Ignite project. Presenting is one of our most senior team members and one of the architects behind Perper – Branimir Angelov.
In Obecto we have been working on an algo trading product, developing a full-stack platform for AI trading agents. Such a platform has complex data requirements ranging from data crawling, processing, training machine learning models, and executing real-time trading strategies.
On a technical level the biggest challenge in such a platform is to combine both online and offline data processing on a large scale by using high-level programming models like reactive streams. To design such architecture you usually have to operate a number of big data technologies (e.g. Apache Spark, Kafka and Akka) and impose various programming constraints to achieve high end-to-end performance.
On the other hand, we wanted to have a more minimalistic architecture approach focused on developer productivity with minimal operational overhead. With these goals in mind, we created Perper – a stream-based, horizontally scalable framework for asynchronous data processing, built on top of Apache Ignite. The programming model is based on the idiomatic language primitives for reactive streams (e.g. async enumerables / generators in C# and Python) and the operation / scaling aspect is based on Kubernetes and KEDA.
The project is in active development and you can check out our progress here: https://github.com/obecto/perper