Given the volume of many million posts per day that Meltwater has to process, we need a technology for search and storage that can handle this kind of volume.

We have been a pretty happy users of Elasticsearch since the 0.11.X days. While we have been through some up and downs, in the end we think our choice of technology was the right one.

Elasticsearch is used to back our main media-monitoring application, where customers are able to search and analyze media data, such as News articles, (public) Facebook posts, Instagram posts, blogs and Tweets. We gather this content using a mix of APIs and crawling, enrich them and make them searchable using Elasticsearch.

In this post, we share what we’ve learned, how you can tweak Elasticsearch to improve its performance, and which pitfalls to circumvent.

Meltwater just released awsudo, an npm package designed to make assuming AWS IAM roles on the command line trivial.

Assuming an AWS IAM role on the command line is not a simple process, and we wanted it to be a human-readable one-liner. Meltwater’s awsudo achieves this with a straightforward command that you can put in front of any awscli operation.

There are other packages out there that accomplish similar goals, but in this post we will discuss why we chose to roll our own.

We just finished our quarterly Meltwater-internal conference. As this is the 6th time that we have run the event, we have gathered some experience by now and I thought it was a good time to share what we have learned about planning and executing such an event.

I wanted to share the approach we take to hosting this because I think a lot of people struggle with how to host these things in a beneficial way. When your company is making the large investment an internal conference requires, you want to make sure it runs smoothly and you get the most out of it. For that preparation is key!

While building Meltwater’s customer facing API, we have been working on a better versioning strategy. Our goal is to build a stable API for clients, but still continue to push out new features and enhancements in as short iterations as possible.

Inspired by Stripe’s API, we have been working on implementing rolling, point-in-time versioning for our API.

We have put together a talk to share our learnings at meetups/conference, but will also write this up in a blog post in the near future.

An API Gateway is a design pattern often used in microservice architecture to provide a single access point to the underlying system. While building the Meltwater API, we have frequently used this design pattern.

In this article we explain why we have found it crucial to have meaningful tests for our API gateway. We also show you our test setup, and how you can use this approach yourself. You should continue reading if you need a simple yet effective way to verify the basic correctness of your API gateway.