Knowing when to stop retrying

Last year we suffered what is a well-known incident to many software developers working on distributed systems: a retry storm, an antipattern where issues get worsened by retry logic that increases load when a service is already struggling.

Our software stack had grown in complexity. We had added new service layers and with them more retry logic, each multiplying the number of retries we do. So when our underlying Elasticsearch cluster experienced an increase in errors, all layers started doing their retries and we ended up DDoSing ourselves. In the worst case scenario, a single request would have been retried over 100 times during the incident.

This post is not about the main fix to this problem. The TL;DR on that was to align between services on when retries should be made (generally we want to trust that the service returning the error has already retried) and tighten up what status codes get retried. This post is about a related improvement that came out of this work on how we handle errors and retries. It’s about how we stopped retrying during incidents.

Our API’s place in the Meltwater software stack

My team owns a Middleware API that sits between the search cluster and the frontend teams’ backends for frontends (BFFs). We make sure content returned is allowed to be viewed by the customer and add additional query time enhancements to the results.

That search cluster is a big Elasticsearch cluster that all our customers query (there are multiple previous posts on our cluster such as this one on how we optimize shard allocation). The shared nature of this means that a single customer can in the worst case affect the performance of all other customers. While we do have an ever-evolving set of guardrails to protect against this, we have seen occasional issues with a high load on the cluster.

As all requests through our API will eventually hit the search cluster, its performance and health matter to us. If the cluster is slow, our API is also slow. Still, if we can retry a failing request to get results, we want to do that. This is where the conflict arises as the retries might be making things worse.

This and the already ongoing work to improve after the retry storm made us think a bit further about how we want retries to work inside our API. Endlessly hammering the cluster with retries during an incident is not how we wanted things to work. It makes any load problems worse, and consumers are likely not getting a successful response out of it anyway. They just wait longer for the eventual error message. What we want to find is a balance, where requests are retried when needed during normal operations, but when we start experiencing issues, we want to stop retries. We want to let the system recover and not do the requests that are most likely not going to succeed anyway. Once it recovers, we start doing retries.

Adaptive retries

With that in mind we decided to adopt our own version of adaptive retries. The same idea shows up as a “retry budget” in Google’s SRE book and as the default retry behaviour in the AWS SDKs. Adaptive retries are a way to adjust the number of retries based on the errors coming from the system being called. Our API keeps a token pool where each token represents a retry attempt. When a request is retried, we take a token from the pool and use it to retry the request. On each successful request we calculate if our refill rate means new tokens should be added to the pool. If a request asks for a token when the pool is empty, we do not retry; instead, the last error is returned. The result from this is that when the underlying system can’t successfully handle requests, we stop retrying until the system recovers.

Since we already use Failsafe, a Java library for retries, circuit breakers and other fault tolerance patterns, for our retries, we decided to build this on top of Failsafe’s RetryPolicy. A circuit breaker would also stop the load, but it blocks every request, including first attempts that might succeed. The token pool only cuts the extra load from retries. We simply wrap our RetryPolicy with our AdaptiveRetryPolicy and implement our own getOrRetry method. When the pool is exhausted, we throw a TokensExhaustedException to prevent further retries and instead rethrow the last error. To achieve this, we configured our RetryPolicy to not retry on TokensExhaustedException.

fun <T> getOrRetry(block: () -> T): T {
    return try {
        retryPolicy.getOrRetry {
            // Retried requests require a token. If the pool is empty,
            // stop retrying and return the original failure.
            if (isRetry() && !tokenPool.consumeToken()) {
                throw TokensExhaustedException(lastError!!)
            }

            try {
                val result = block()

                // Tokens are refilled only after a successful request.
                // During an incident, failed requests therefore cannot
                // replenish the retry capacity.
                tokenPool.maybeRefill()

                result
            } catch (exception: Exception) {
                lastError = exception
                throw exception
            }
        }
    } catch (exception: TokensExhaustedException) {
        // The retry budget is exhausted, so expose the original error.
        throw exception.cause!!
    }
}

The result

We rolled this out with an initial pool size of 25 tokens per API instance and a refill rate of 1 token per second. Given our standard request rate of around 40–100 requests per second spread out over 4 instances, we can empty our pool within a couple of seconds if all requests start failing, stopping ourselves from doing any more retries until the service recovers. This also means that once the pool is empty, the occasional successful request still refills one token per second, so each instance can still retry roughly one request per second during an incident.

When our Elasticsearch cluster was more recently overloaded, our logs and metrics show that we quickly stopped retrying, and that we avoided more than 23,000 retries in the 15 minutes it lasted. This corresponds to nearly 60% of the request rate we had when the incident started and would be a significant increase in load.

During normal operations we do see the occasional retry but not enough to exhaust our retry pool, so retries are not blocked.

Potential improvements

• The main one is to make the pool shared between running instances of the API. Today each instance is running in a Kubernetes pod, each with its own pool. This means that if we scale our service horizontally, we also increase the number of retries we can do.

• We could tweak our numbers. While the current setup seems like a good start, we have so far not done any deeper analysis to tune the pool size and refill rate. As both our current retry pool size and refill rate are based on quick assumptions we made during development.

Summary

Potential improvements aside, we are really happy with the outcome of this work. We can clearly see how retries are skipped when system load gets problematic, while normal operations work just as before. The exact thing we wanted to achieve when we started looking at this.

So if you take one thing from this post: retries help when failures are infrequent but can hurt when the system behind them is overloaded. Your retry logic needs a way to tell the difference. A token pool is a simple way to do that.