Micro Pipelines: Analyzing Big Data with Tiny Apps

One of our teams at Meltwater was recently faced with a problem that required relatively simple tasks applied to a large volume of data. To solve this we experimented with a pattern we call micro-pipelines, which are a sequence of microservices that work together to create efficient, fault tolerant systems.

This post provides an example of how we designed and built such a micro-pipeline.

Meltwater speaking at Cloud Native Rejekts Barcelona

Meltwater’s Foundation team has invested heavily in Kubernetes to provide this platform as a service to other development teams. Many of those teams have migrated to Kubernetes already.

Naturally our Foundation team learned a lot about what can, and will, go wrong when running a big enough Kubernetes cluster. Recently they attended Cloud Native Rejekts in Barcelona, where they shared some of these learnings.

Hosting the CloudNativeGbg meetup in Gothenburg

Meltwater has been using Kubernetes in production for over a year. The theme for this meetup was “difficulties and pitfalls” so we wanted to share our lessons learned from offering Kubernetes as an internal service to our engineering teams all around the globe.

It was the second edition of the CloudNativeGbg meetup and we hosted the meetup it in our Gothenburg office. A great attendance ensured very interesting discussions before, during and after the event. Thank you all for coming!