Context
Turning hundreds of millions of financial data points into practical insight required more than a dashboard. The goal was to create an analytics foundation that could support internal teams and multiple partner organizations without turning the core platform into a heavily customized system.
This required a careful balance between analytics capability, platform performance, partner flexibility, and data privacy.
The Root
- Distributed Data Processing
- Precomputed Analytics
- Partner-specific Read Models
- Near Real-time Kafka Pipelines
- Governed Partner Bi Access
The Challenge
The data was sitting in a relational database that was optimized for, and belonged to, another operational application. That database was designed for primary financial workflows, not heavy analytical workloads. At the same time, the analytics platform needed to support around 200 domain-specific metrics, many of which were computationally expensive. Computing these metrics on demand was not practical, especially when the dashboard needed to load in seconds. Several constraints had to be addressed together:
- Hundreds of millions of financial data points
- Around 200 complex metrics
- Dashboard performance measured in seconds
- Protection of the operational system from heavy analytical queries
- Partner demand for custom metrics and advanced analytics
- Strict control over PII, data access, and partner-level boundaries
The larger architectural risk was clear: if every partner request was added directly into the dashboard, the product would become harder to maintain, harder to scale, and increasingly fragmented over time.
What Was Done
Separated Operational and Analytical Workloads
A dedicated analytics architecture was introduced so reporting and metric computation would not depend directly on the transactional system. Apache Spark was used to process large data volumes and compute complex metrics across multiple nodes in a horizontally scalable way.
This allowed analytical workloads to scale without slowing down the core financial application.
Built a Purpose-Optimized Serving Layer
Instead of calculating metrics when users opened the dashboard, the metrics were precomputed and written into a dedicated document-oriented database optimized for dashboard access patterns. This reduced query complexity, avoided expensive runtime computation, and allowed the dashboard to serve complex metrics in seconds despite the underlying data volume.
Created a Partner Analytics Path Without Overloading the Core Product
Advanced partners needed more flexibility than the standard dashboard could provide. Rather than continuously adding partner-specific features into the core analytics platform, a separate data access model was introduced. Using Apache Kafka-based streaming pipelines, partner-specific data was replicated into dedicated read models in near real time. These read models became trusted data sources for partner BI environments, allowing advanced users to connect tools such as Power BI and build their own reports, metrics, and analytical views.
This gave partners flexibility without weakening the core platform. It also preserved clear data access boundaries, partner-level isolation, and control over sensitive financial and personally identifiable information.
Outcome
The platform created a scalable analytics foundation for both internal users and partner organizations. It enabled:
- Analytics across hundreds of millions of financial data points
- Around 200 complex financial and operational metrics
- Reduced risk to the core transactional system
- Clear separation between platform-level analytics and partner-specific BI needs
- Advanced partner analytics without continuously customizing the core dashboard
Most importantly, the platform avoided a common failure pattern: turning a shared analytics product into a collection of partner-specific exceptions. Instead, it created clear architectural boundaries — one path for standard platform analytics, and another for advanced partner-driven analysis.
What This Represents
As platforms grow, data needs become more complex. The challenge is not only how to process more data, but how to design the right boundaries so the platform can keep growing without becoming difficult to maintain. Complex systems need clear separation between what belongs in the platform, what belongs in extensions, and what must remain protected at the data governance level. Without those boundaries, analytics platforms become slow, fragile, and overloaded with exceptions.
With the right architecture, data can become useful without compromising performance, privacy, or long-term system sustainability.
Note: This example reflects founder experience gained through prior engineering and technology leadership roles before Rootstone. It is not presented as work delivered by Rootstone as a company. Client names and identifying details have been omitted to preserve confidentiality, while focusing on the type of systems, constraints, and decisions that shape Rootstone’s approach today.
