Introduction

ActiveViam has shown that its proprietary software Atoti is the leading technology to interactively analyse datasets from Gigabytes to Terabytes. Atoti+ can harness the largest machines of Cloud platforms and on-premise DataCenters to make all the CPUs work at a blazing speed on their Terabyte RAMs filled with client data. But this falls short in comparison to solutions like Snowflake capable of handling Petabytes of data.

In this context, ActiveViam started working on an integration with Snowflake and BigQuery, among others, offering users a direct access to their data via SQL in those solutions while adding the powerful computations and analysis tools of Atoti. 

Atoti possesses its own aggregation engine, capable of evaluating simple formulae such as SUM(t.a + t.b) – in a pseudo SQL – as well as complex formulae like a sliding summation SUM(t.population) OVER (partition by t.city order by t.date).
To compute such formulae, Atoti retrieves the underlying data from the cloud database before evaluating the formula. For example, it would ask for t.a and t.b and locally sum the two values.
A huge optimization would be to move this sum directly into the cloud database to benefit from its computation power, in addition to reducing the size of the result and the amount of transferred data.
This technique is commonly named “SQL Pushdown” and is present in many other products.

Expected work

The goal of this internship is to prototype a Query Engine capable of selecting branches of calculations that can be pushed down to a SQL Engine.

The initial phase will list the opportunities for SQL Pushdown among the calculations offered by the Atoti DSL.

In a second phase, from basic Abstract Syntax Trees representing those calculations, the prototype will suggest a SQL rewrite of those formulae. After that, we will work on detecting those patterns in larger AST where all computations cannot be pushed.
The challenge also includes dealing with result partitioning as well as intermediate stages sharing, to avoid duplicated calculations if possible.

This prototype will be coded in Java, the core language of Atoti.

To turn the prototype into a production grade component and make it part of the Atoti+ solution, this internship may open up the opportunity to be continued as a standard position in the R&D team.

About ActiveViam

Founded by a group of industry experts, ActiveViam understands the data analytics challenges faced by financial institutions across trading desks, risk, and compliance. That is why we pioneered the use of high-performance analytics in finance, helping the largest investment banks, asset managers and hedge funds make better decisions, explain results with confidence, and simulate the impact of their decisions.

Our mission is to deliver train-of-thought analysis on terabytes of data in the most cost-effective way so our customers can explain their results with confidence and model the scenarios that will optimize their business. We are a pure player specializing in risk data analytics for one of the fastest moving and most regulated industries with a presence in the world’s leading financial marketplaces: London, New York, Singapore, Sydney, Hong Kong, Paris and Frankfurt.

For more information please visit: www.activeviam.com