Why Complex Event Processing (CEP) is not enough for real-time decision making

ActiveViam |
October 15, 2013

Real-time decision making is commonly linked with Complex Event Processing (CEP). Indeed, CEP systems can extract and alert about meaningful events from streams of data. However, for decision makers to have context and turn a notification into a meaningful, actionable event, CEP must be supplemented with mixed workload and multidimensional capabilities.

Let’s take a look at what it means.


With business processes growing in complexity, managers are faced with an increasing amount of metrics and KPIs that they must follow to remain in control at all times. Managers cannot manually monitor hundreds of KPIs over time. It is obviously unrealistic for managers to sit behind their screen and wait for numbers to blink, trying to make sense of which blinking numbers are worth taking a closer look at. Put differently, managers don’t want to “pull” events, but rather, that meaningful events rsuch as an unusual pattern or an error, be pushed at them.

For example, consider an online retailer whose strategy is to offer the most attractive prices in order to maximize revenues. Because of the sheer volume of prices that must be monitored on hundreds of thousands of products, pricing specialists cannot check whether all products are cheaper than on competitor sites.  Instead they want to be alerted when meaningful outliers appear. For example, when the price performance of a “star” product with the highest conversion rate in the last 7 days changes by 5% in comparison to the next best competitor.

Unfortunately, hiring hundreds of analysts to monitor hundreds of KPIs would be difficult to justify. Managers and operational users want to be proactively alerted whenever their KPIs breach a limit, meet a combination of specific conditions, or evolve in an unusual way. In short, they want to be able manage situations by exception.


Complex Event Processing to the Rescue

What does event-driven KPI monitoring involve from a technology standpoint?

Complex Event Processing (CEP) seems to provide the best technological solution for management by exception, as it addresses real-time event processing. CEP systems track streams of data from multiple sources about “things that happen” in order to identify meaningful events, alert users, and respond to them as quickly as possible. Such systems are extremely useful to companies that base their competitive advantage on adapting to specific events as they occur. Information timeliness is what matters the most to these organizations, since they cannot rely on end-of-day data, let alone on data from yesterday. Real-time notification is a must-have, not a luxury.

Take the example of a digital advertising exchange that buys advertising space for its clients and chooses the best display strategy. To optimize display with conversion rates at the highest, the organization actually needs to mix and match several axis such as geography, browsing history, previously displayed pages, dates, and so on – a process that must be carried out in less than a millisecond. Real-time and event processing in this case are key enablers for maintaining the competitive advantage of the business.


Why Complex Event Processing is not enough

But is CEP enough? Can a CEP system make do without mixed workload and multidimensional analysis? The answer is no. You still need to run multidimensional analytics on transactional data. Here’s why.

What is Complex Event Processing? – Extract of the ActivePivot User Group 2012

While CEP systems definitely address event processing, they lack the decision context required to turn a simple alert into something meaningful. The main reason is that data “disappears” once events are processed. For instance, in the case of a central counterparty that clears billions of trades on behalf of their clearing members, there is little value in generating alerts on upcoming margin calls if the alert is not documented with additional contextual information, such as the history of past transactions and additional measures such as the Value at Risk.

In the case of ecommerce, dynamic pricing takes more than generating an alert as soon as the average prices of a competitor changes. To be able to decide whether or not to align to Amazon’s prices when an alert is received, pricing specialists also need contextual information about inventory levels, web traffic information, conversion rates etc.

Whilst the notification of an event is an invitation to further investigate, it cannot be considered as the analysis itself. It is only a means to an end. Business users still need to delve into the context from the moment they receive an alert about a limit breach on their KPIs. To that end, they need to be able to analyze the limit breach across as many dimensions as they want: by product, by country, by legal entity etc.

Decision makers also require the flexibility to create new limits on the fly as a response to a specific change in context. For example, if a country is suddenly faced with bankruptcy as was the case in Greece back in 2012, banks probably want to be able to create new limits on the fly on that specific country. Furthermore, users want these limits to be taken into account now and not as a result of a long cycle involving weeks of development. Time to market is the obstacle that traditional SQL-based systems typically stumble across: If you’re using a RDBMS, you will need to redevelop a stored procedure and update all subsequent reports each time a new limit needs to created. True multidimensional analytics systems, on the other hand, can allow users to create complex limits on the fly, by ‘magically’ applying a calculated measure to all dimensions.

To summarize, CEP technologies need to be supplemented by multi-dimensional analytics. Multidimensional analytics delivers the underlying context that eventually turns a notification into a meaningful alert, which business users can act upon.

Eventually next-generation analytics encompasses all the three enablers covered in our last three blog posts: Multidimensional analytics, mixed workload and CEP. Our vision for ActivePivot has evolved over the years with these fundamental principles in mind. At the end of the day, regardless of technological considerations, analytics should not be accessible only to a small number of subject-matter data experts. Analytics should be in the hands of people who run the business – whether they are traders, pricing specialists, supply chain managers, risk managers, product controllers, or marketers. Analytics must be aligned with the way decision makers work and make decisions.

Like this post? Please share it on your socials

Schedule a demo


Sorry! We were unable to process your
request. Please try again!


Your request is submitted successfully!
We will keep you up-to-date.