insighting.io · SAP-to-Dashboard Data Pipeline
A complete end-to-end data pipeline built around a lightweight on-site sensor installed in each customer environment, enabling Insighting to securely extract operational and sales data from private SAP Business One systems and deliver it automatically to the Insighting cloud platform daily — with zero manual customer involvement.
SAP data is the richest —
and the most locked down.
insighting.io is a Sales Intelligence platform built for wholesalers and distributors, helping B2B sales teams uncover revenue opportunities in real time through clear, actionable insights — without the need for manual analysis.
The problem: every customer's data lives locked inside their own on-premises SAP server, completely inaccessible from the internet. Getting that data reliably, securely, and at scale into a cloud platform meant solving several hard problems simultaneously.
SAP's API uses cookie-based session management, requires OData pagination across thousands of records, and demands intricate cross-entity join queries for related data. Every customer runs a different SAP database, server configuration, and set of required entities. New customers need months or years of historical data backfilled before they see any value. And with several businesses continuously uploading data, there can be zero risk of cross-tenant data access or loss.
Our SAP system is on-premises and always has been — we assumed getting our data into a cloud dashboard would be a major IT project.
Three services. One
zero-touch pipeline.
We designed and built three purpose-built services that work together as a single cohesive pipeline. Data flows from a customer's private SAP server to their cloud dashboard every day — automatically, securely, and without any customer action required.
SapSensor is an on-premises extraction agent deployed directly alongside the customer's SAP B1 installation. It runs as a Windows Service or Linux daemon — a single self-contained binary with no dependencies. Each night it authenticates to SAP's Service Layer using cookie-based sessions and streams transactional data page by page to disk, never accumulating large datasets in memory. Every output file is AES-256 encrypted before transmission. Transient failures — network blips, SAP session drops — are handled automatically with exponential backoff retry logic.
Insighting Web Service is the secure cloud receiver. It handles all customer tenants simultaneously, each authenticating with their own credentials, with strict per-tenant isolation at both the auth and filesystem level. Every uploaded file is written simultaneously to a primary and backup directory — dual-write redundancy ensuring no data is ever lost even if a downstream step fails.
Distribution Service handles the last mile: delivering processed reports back to customers securely. Ops teams publish updated files with a single authenticated HTTP call; customers pull the latest report on demand. Path-traversal protection, IP logging, and strict per-tenant auth ensure the system is provably isolated across all tenants.
- SapSensor — self-contained on-premises extraction agent (Windows Service / Linux daemon) with no dependencies
- OData pagination and dynamic
$crossjoin/$expandquery engine for complex SAP data relationships - AES-256 encrypted file transmission with exponential backoff retry logic for transient failures
- Automatic historical backfill mode — new tenants go from zero to a fully populated dashboard in a single run
- Insighting Web Service — multi-tenant cloud gateway with per-tenant auth and filesystem isolation
- Dual-write redundancy and consume-and-delete atomicity preventing data loss and double processing
- Distribution Service — secure report distribution gateway with two-tier auth (admin + per-customer)
- Real-time pipeline health dashboard via forwarded structured error logs from all tenants
Built on .NET end-to-end.
Data arrives every morning.
Automatically.
- 5 customer tenants continuously stream data from private, on-premises SAP environments to the cloud — no VPN, no firewall changes, no middleware required
- Zero data loss since launch, thanks to dual-write redundancy and consume-and-delete atomicity
- SapSensor runs silently every day and self-recovers from any transient failure — no customer action ever required
- New customers go from zero to a fully populated dashboard within hours of agent installation — historical backfill happens in a single automated run
- The ops team has real-time visibility into the health of every customer's pipeline from a single dashboard via forwarded structured logs
Our SAP system is on-premises and always has been — we assumed getting our data into a cloud dashboard would be a major IT project. Instead, we installed an agent, pointed it at our SAP server, and by the next morning all our sales history was there. Since then it just runs on its own every day. We've never had to touch it.
yours?
Tell us what you're building. We'll tell you how fast we can get you there.