At the heart of Meltwater’s and Fairhair.ai’s information retrieval systems lies a collection of Elasticsearch clusters containing billions of social media posts and editorial articles.

The index shards in our clusters vary greatly in their access pattern, workload and size which presents some very interesting challenges.

This blog post describes how we use Linear Optimization modeling for distributing search and indexing workload as evenly as possible across all nodes in our clusters.

Quitsies is a distributed and disk persisted caching system that implements a subset of the Memcached text protocol. It was built as a minimal drop-in replacement for Memcached, and has been running in our production pipelines for over a year.

This post explains why we needed Quitsies, and how we went about building it. Quitsies is open source, so you can try it yourself.

Would you agree that your biases are affecting your recruitment process? We have been thinking about it and we were especially curious how we can improve our recruitment process by working with our biases and learning how to disarm those when hiring.

In this post we are sharing the tools and processes that we found useful. You can try them too!

Meltwater recently launched the Fairhair.ai data science platform. Part of this platform are several large Elasticsearch clusters, which serve insights over billions of social media posts and editorial articles. The nature of the searches that our customers need to run against this data quickly make the default load balancing behaviour of Elasticsearch insufficient.

In this post we explain how we built a custom search router using machine learning, that helps us to address the shortcomings of Elasticsearch’s default round-robin approach, and greatly improves search performance and fault tolerance.

Meltwater is excited to sponsor Brewing Agile in Gothenburg on October 12-13, 2018. This is the 4th year in a row that Meltwater is supporting Brewing Agile, so you can tell that we are honestly excited about this event.

This is the only conference about Agile in Gothenburg, and there are still tickets available, so don’t wait and sign up quickly.