Streaming Analytics – Handling Continuous Data Flow

Blog Data Science

Many industries use different analytics solutions, but as the ongoing flow of relevant data is becoming more frequent we see that pure analytics does not really address the need to continuously process this data. Streaming analytics expands system’s processing capabilities to continuously aggregate and analyze relevant data.

Any analytical system can benefit from the application of streaming analytics. For instance, trading firms can perform iterative complex risk analyses, many times throughout the day rather than overnight with pure analytics systems. Communications companies analyze call data records in real time to offer better service and rates. Online retailers can optimize pricing by analyzing dynamic data.

However, as you can see, streaming analytics is more than just continuous processing and analysis of data; it also provides the ability monitoring any changes in data in real time. This type of analytics expands the application of traditional analytics systems in the business world, but there are other possible applications of continuous analytics.

Julian Hyde, founder of the Mondrian open-source OLAP engine, wrote about an interesting application of streaming analytics:

Social networks and web content feeds such as RSS have, in a few short years, added a dynamic component to the vast static content on the web. As less-sophisticated users have become more accustomed to consuming them, these feeds have become a ubiquitous part of the web experience.

Web feeds have an information content that is at present untapped. In the same way that a radical new approach — the search engine — was needed to harness the static information content of the web, a streaming analytics solution in this area becomes important sooner rather than later.

I believe that real-time web content feeds are a game changer. I call it the Streaming Web — a web where every piece of content is accessible via a URL and you can subscribe to be alerted immediately if a piece of content changes. Every page would become a potential feed, and there would be agents that allow us to collect and filter content we are interested in: be it a friend’s photo album or the price of a plane ticket.

Returning to the business world, take traditional OLAP systems for examples, these systems provide pure analytics capabilities, that is, aggregation and analysis of massive amounts of data during a set time when transactional flow is low. As we noted, there is greater need for analytics that monitor, aggregate, and analyze continuously flowing and frequently changing data.

In ActivePivot, we have developed a real time OLAP solution with streaming analytics abilities. Its in-memory engine and multithreading capabilities allow users to run continuous complex queries on massive amounts of existing data as well as new inputs and data changes, and receive results within seconds.

ActivePivot detects any new or modified data in the source system, and updates its OLAP cube in real time while continuously compressing data, saving memory to enable fast response to queries. The system’s real-time OLAP engine only aggregates new data impacts, without having to rebuild the OLAP cube each time, thus minimizing latency. It then pushes the changes to the front-end tool, which allows continuous real time data analytics.

To try out ActivePivot’s Streaming Analytics solution, see our live demo.