Introduction

ActiveViam has shown that its proprietary software Atoti is the leading technology to interactively analyse datasets from Gigabytes to Terabytes. Atoti can harness the largest machines of Cloud platforms and on-premise DataCenters to make all the CPUs work at a blazing speed on their Terabyte RAMs filled with client data. But this falls short in comparison to solutions like Snowflake capable of handling Petabytes of data.

In this context, ActiveViam started working on an integration with Snowflake and BigQuery, among others, offering users a direct access to their data via SQL in those solutions while adding the powerful computations and analysis tools of Atoti. 

Atoti customers often analyse fast-moving data, especially in verticals like investment banking and e-commerce. To complete the integration with cloud database systems, Atoti therefore needs to be aware of the changes in the underlying database to efficiently update its internal data structures and offer up-to-date information to its users.

Expected work

The goal of this internship is to prototype a system for streaming changes from BigQuery and Snowflake to Atoti in real-time.

The initial phase will study in depth the various mechanisms offered by BigQuery and Snowflake, such as Snowpipes, table metadata, as-of-date queries.

In a second phase, given a list of perimeters (subset of the data) used by the application, we will build a prototype capable of notifying which perimeter has been updated and ideally report the extent of the update – added rows, updated or deleted values.

Based on that information, we will design and implement a way to update internal components of Atoti stack.

One of the biggest challenges in this internship is the definition of real-time. Producing the notification within the 5 minutes that follow the update is a good performance. Offering this same notification under a minute is an impressive achievement. And a 1s delay would be an astonishing realisation.

This prototype will be coded in Java, the core language of ActiveViam software.

To turn the prototype into a production grade component and make it part of the ActivePivot solution, this internship may open up the opportunity to be continued as a standard position in the R&D team.

About ActiveViam

Founded by a group of industry experts, ActiveViam understands the data analytics challenges faced by financial institutions across trading desks, risk, and compliance. That is why we pioneered the use of high-performance analytics in finance, helping the largest investment banks, asset managers and hedge funds make better decisions, explain results with confidence, and simulate the impact of their decisions.

Our mission is to deliver train-of-thought analysis on terabytes of data in the most cost-effective way so our customers can explain their results with confidence and model the scenarios that will optimize their business. We are a pure player specializing in risk data analytics for one of the fastest moving and most regulated industries with a presence in the world’s leading financial marketplaces: London, New York, Singapore, Sydney, Hong Kong, Paris and Frankfurt.

For more information please visit: www.activeviam.com