Long-Term Maintenance of an Open-Source Data Pipeline Solution

We help maintain OpenSnowcat, an open-source data engineering solution that lets businesses collect and enrich data for further processing or analysis. We are taking ownership of a technically demanding platform, with our efforts remaining effective over a long-term engagement. Our ongoing work strongly supports the platform’s security and stability for business users and developers. The maintenance engagement involves variable work volumes and unique technical challenges, requiring in-depth Scala expertise and great attention to detail.

Our client maintains OpenSnowcat – an open-source fork of Snowplow under the Apache 2.0 License. The solution is a behavioral data and event platform for large-scale enterprise data collection, enrichment, routing, and loading. While mature and in-demand, it requires ongoing maintenance and configuration, as the technology environment it’s typically used in changes over time.

During our partnership with the client, Amazon Web Services deprecated a library version crucial to OpenSnowcat’s interoperability with Kinesis. This required a transition to a newer version whose core principles differed significantly from the original, now-unsupported code. We adapted OpenSnowcat’s codebase to follow these changes, ensuring continued compatibility with AWS Kinesis, while keeping all user-facing processes and controls the same as before the update.

This is one of many such examples of our maintenance work on OpenSnowcat, where another service in the data engineering field changes its product, API, or library, forcing other tools to adjust as well. We address the issue promptly and directly in OpenSnowcat’s codebase, saving businesses the time and effort it would otherwise take to troubleshoot the dependency or issue themselves.