At Privy, many of our services are fundamentally event-driven. Indeed our core product value lies in helping merchants capture arbitrary user interaction and reacting to opportunities as they arise in a tangible and timely manner.
A key criterion for new components and systems at Privy is that they must be elastic. We must be able to out-scale our fastest growing merchants, if we are to continue to provide an acceptable level of service.
In addition to scalability, our systems must be fault tolerant or resilient in that failure of one component should not affect the overall integrity of our system.
Systems that are responsive, resilient, elastic and message-driven are also known as Reactive Systems. The Reactive Manifesto, a community distillation of best-practices, provides a concise vocabulary for discussing reactive systems.
Systems built as Reactive Systems are more flexible, loosely-coupled and scalable. This makes them easier to develop and amenable to change. They are significantly more tolerant of failure and when failure does occur they meet it with elegance rather than disaster. Reactive Systems are highly responsive, giving users effective interactive feedback.
Being Reactive in Practice
It is important to note that the Reactive philosophy is independent of any specific application layer; these general requirements can be realized throughout the stack.
Indeed there are many open source frameworks that can used to build Reactive systems. Front-end examples include Backbone.js, Facebook's React, and Elm. These specific examples essentially handle input events and their effects as process networks.
A similar but distinct concept is the Actor model, which often arises in the context of highly concurrent and distributed background operations. The Actor model is a computational model that treats computational entities as primitives called actors. Actor behavior is defined by the ability to respond to messages received from other actors, the ability to send messages to other actors, and the ability to spawn new actors.
Actors saw much success in Erlang, a language originally designed for building telecommunication systems. For use-cases more specifically related to web applications, two popular Actor-based frameworks are Akka and Celluloid.
Celluloid is the underlying actor system used in Sidekiq, a background task framework for Ruby. Sidekiq is an integral component of the Privy backend - most of our asynchronous Ruby behavior occurs within Sidekiq Workers.
We're also within the early stages of deploying an Akka app for Business Intelligence, which happens to be the primary motivator for this series of blog posts.
Handling Streams of Data: Part 2
In the next post in this series, we'll examine methods for dealing with streams of data in an asynchronous and reactive, specifically responsive and elastic, manner.