In Meltwater’s Berlin office, we are enthusiastic users and advocates for Elixir and ruby. Hence we were excited to get the chance to host the Elixir Berlin meetup for the first time this November.

It was the #53’rd edition of the Elixir Berlin already, what a great streak!

Besides hosting the event and sponsoring food and drinks, we were happy to share some of our own Elixir learnings. For some of our presenters it was the first time on stage at a developer meetup, and we can only say that the Elixir Berlin crowd made it very easy for us, so thank you for that!

At the heart of Meltwater’s and Fairhair.ai’s information retrieval systems lies a collection of Elasticsearch clusters containing billions of social media posts and editorial articles.

The index shards in our clusters vary greatly in their access pattern, workload and size which presents some very interesting challenges.

This blog post describes how we use Linear Optimization modeling for distributing search and indexing workload as evenly as possible across all nodes in our clusters.

Quitsies is a distributed and disk persisted caching system that implements a subset of the Memcached text protocol. It was built as a minimal drop-in replacement for Memcached, and has been running in our production pipelines for over a year.

This post explains why we needed Quitsies, and how we went about building it. Quitsies is open source, so you can try it yourself.

Would you agree that your biases are affecting your recruitment process? We have been thinking about it and we were especially curious how we can improve our recruitment process by working with our biases and learning how to disarm those when hiring.

In this post we are sharing the tools and processes that we found useful. You can try them too!

Meltwater recently launched the Fairhair.ai data science platform. Part of this platform are several large Elasticsearch clusters, which serve insights over billions of social media posts and editorial articles. The nature of the searches that our customers need to run against this data quickly make the default load balancing behaviour of Elasticsearch insufficient.

In this post we explain how we built a custom search router using machine learning, that helps us to address the shortcomings of Elasticsearch’s default round-robin approach, and greatly improves search performance and fault tolerance.