blog image

Stateless and Stateful Event Processing

In previous blog articles, we discussed the potential of event stream processing as a means of improving customer experience and business efficiency. 

In this article, we wanted to look deeper at the type of analysis we might want to do on these event streams to bring the opportunity to life. 

Imagine a customer lifecycle which is comprised of a series of events such as the following:   

There are various things we might wish to do here to the event stream to filter, modify, analyse and respond to it.  For instance, maybe there is the business requirement to manually review all orders over a certain value being dispatched outside of the UK. 

The first class of transformations we would like to do are referred to as stateless, because they don’t require any history or memory in order to action.  For instance, filtering if the order value is greater than a certain number, or reformatting an Order ID are stateless operations.  These can happen quickly and can be scaled across many servers for inherent parallelism.  To a large degree, this is a solved problem with various business rules and event processing solutions able to carry out these stateless operations. 

The second class of changes are stateful.  An example might be the requirement to see if the same customer has placed high value 3 orders in the last 24 hours, or to aggregate the total for all of the orders dispatched today.  These are much harder to implement, because they require memory of the event stream, require access across different streams, and we need to ensure that stream processors have access to the right data at the right time.  In the above example, the stream processor needs to have access to at least 24 hours of order and customer data in order to keep up the running total. 

Stateful computations are powerful, and are where the untapped opportunities lie for companies to differentiate their businesses. 

Here are some further examples to bring this to life:

Customer Order Service Level Agreements – Identifying high value customer orders which have breached a service level agreement for being dispatched in time, increasing the risk of an unhappy customer;

Financial Transaction Monitoring – Identifying credit card transactions taking place in a foreign country which fail validation three times in an hour, indicating potential fraud;

Manufacturing And Internet Of Things – Identifying an industrial machine which is showing an increase in heat output over the last 30 minutes through its IOT sensors, indicating a potential fault;

Customer Satisfaction KPI Monitoring – Identifying specific call center agents who are taking longer than average to complete calls, and are receiving less than average customer satisfaction scores;

Travel Revenue Optimisation – Identifying opportunities for up-sell and cross sell of trip add-ons based on previous purchase data;

Sadly, stateful stream processing is also where stream processing becomes complicated.  Not only do you have the challenge of maintaining the right events in memory, you also have additional challenges such as the need for exactly once processing, the need to handle parallelism correctly, and a number of challenges around resilience and recovery from errors.  These will be discussed in more detail in a future article.