One of the most common questions our media relations customers ask is “which journalists cover this subject?” Until recently, we answered it with a keyword match against the articles each journalist has published. This approach has two limitations:
- It matches exact words only. A search for
"electric vehicles"misses a journalist who writes about"EVs". - It cannot tell a specialist from someone who mentioned the term once. A search for
"AI"returns every journalist who mentioned AI in passing alongside the reporters who cover it full time, so common terms return thousands of loosely related results.
To address this, we introduced the concept of fluid topics: short, free-text labels such as "Carbon Markets" or "Streaming Rights", generated from recent works of journalists and publicly available information about them. We call them fluid because they are free text instead of entries from a fixed list, and they change as a journalist’s coverage changes. Each topic is embedded as a vector, and a new topic search matches queries against those vectors by semantic similarity.
This post walks through the pipeline end to end. It covers how topics are curated, embedded, indexed and searched. It also covers the changes we made along the way, including switching embedding models, cleaning up orphaned topics, and handling queries that embeddings alone cannot resolve.
Architecture
Journalists are stored as contacts in our database. The stages below and the rest of this post use both terms. The feature spans four stages, connected by Kafka:
- An LLM enrichment agent researches a journalist and predicts a short list of fluid topics, each labelled
primaryorsecondary. - The contact store owns the journalist record. It deduplicates and normalises the topics, keeps the most relevant ones, computes an embedding for each new topic, and publishes the updated contact.
- The search indexer writes the contact into an OpenSearch database. It also maintains a separate topic index. A topic such as
"Climate Policy"appears on thousands of contacts, but the topics index stores it only once, with its vector. - The contact search API exposes a
topicQueryparameter. It embeds the query, finds the nearest topics, and ranks contacts by how well their topics match.
Fluid Topic Curation
A fluid topic is only useful if someone typing that phrase into a search box would expect to find specialists in it. This drives all of the curation rules.
What a good topic looks like
The enrichment prompt asks for short noun phrases of two or three words, at specialist granularity. For example, we prefer "Breast Cancer" over "Healthcare" and "Streaming Rights" over "Media". We convert named entities into reusable topics, so "Chelsea Flower Show" becomes "Horticulture Events". Time-bound topics like "Q4 Earnings" are rejected. Granularity matters more than wording here. Because matching is semantic, near-synonyms are found anyway. An over-broad topic like "Technology" sits close to almost everything in embedding space, which makes a journalist a false match for many searches.
Where topics come from
The agent combines two independent signals:
- Web research: bios, social profiles and outlet pages. This captures how a journalist describes themselves, but self-descriptions go stale.
- Article history: top articles from the last 3 months, sampled with a recency bias.
When the two overlap, web research decides the wording, because it uses phrasing a human recognises, and article history decides the prominence, because it shows what the journalist writes about most. Syndicated copies of the same wire story are collapsed, so a story republished 40 times does not look like 40 topics. The random sample is seeded on the contact ID, so re-runs and evaluations are reproducible.
Normalization
Topics are stored once and shared across contacts, so two strings that look the same must resolve to the same row. They are often different strings underneath. An é can be stored as one character, or as an e followed by a combining accent, so "Café Culture" can arrive in two forms that look identical on screen. A double space or a trailing space has the same effect. We apply Unicode NFC normalisation, which stores accented characters in one form, and collapse whitespace. Our first version also lowercased everything, which looked like an obvious win for deduplication. It was not. Casing carries meaning that the embedding model uses. Lowercased, "US Politics" becomes "us politics" and "IT Security" becomes "it security", where "us" and "it" read as ordinary words instead of acronyms. We now preserve casing on both the stored topics and the query.
Topic Embedding
Once curated, every new topic text is embedded and stored alongside it. Because thousands of journalists share topics like "Climate Policy", the topic table is keyed by the normalised text, and a vector is computed only the first time a topic is seen.
We started with the internal embedding service built on Snowflake’s snowflake-arctic-embed-l-v2.0 model, which was already used by other teams in the company. It was already in production, multilingual and cheap to call, so it was the natural starting point. We stored 256-dimensional vectors and moved on to search.
Indexation & kNN Search
Attempt one: kNN over nested contact topics
The first version indexed each contact’s topics as nested documents, each with its own knn_vector field, and ran a kNN query straight against the contact index. Each contact was scored on its best topic, multiplied by a prominence weight, so a contact covering one topic deeply outranked a contact covering many loosely related ones.
This worked in tests and failed on real data. Popular topics are stored once per contact. A query close to "Climate Change" finds that vector thousands of times over, so the top k nearest vectors are dominated by copies of one or two topics, and k (set to 500 at the time) no longer meant anything a caller could reason about. Also, by default OpenSearch only returns the nearest nested vector per contact. We needed expand_nested_docs so a primary topic that was not the nearest one could still earn its weight. Also, kNN filters are evaluated against the parent document, so they cannot express conditions on nested fields.
Attempt two: two-stage search over a dedicated topics index
The fix was to separate which topics are relevant from which contacts cover them:
- Stage one: run kNN against a dedicated topics index containing one document per distinct topic.
kis now the number of nearest distinct topics (default 10). - Stage two: run an ordinary query against the contact index, matching contacts whose nested topic is one of those
k. Each (topic, prominence) pair becomes aconstant_scoreclause whose boost is the cosine similarity times the prominence weight. A contact that carries several of the matched topics is scored on its best one, so a journalist with both"Solar Power"and"Wind Energy"scores as high as their closest topic.

The example below uses the query "renewable energy" with k = 3. The prominence weights (1.0 for primary, 0.5 for secondary) are for illustration.
"renewable energy" with k = 3
We use constant_score because a bare term query scores by term statistics, which lets a rare topic string outrank a relevant one. With constant_score, the term match decides whether a contact matches, and the boost decides its rank. All the other parts of the request (country, outlet type and other filters) go into filter context, so the final _score is pure topical fit and can be explained to the user. Callers can request topicRelevanceScore and per-topic similarity in the response, and can override the prominence weights and similarity floor per request, which makes tuning against live traffic easier than redeploying config.
The topics index is populated as a side effect of contact indexing, and a topic like "Climate Policy" arrives on hundreds of contact updates a day. To avoid rewriting the same document each time, we use the message timestamp truncated to the day as the document version, with OpenSearch’s EXTERNAL versioning. Every repeat write of a topic within the same day becomes a no-op on the server, across all indexer pods, without any coordination between them.
Changing the Embedding Model
With the search mechanics in place, we needed to know whether the results were good. We built a comparison harness that ran 17 real customer queries ("education", "data centers", "wildfires", "Korean skincare", "cyclospora", "Proposition 12" and others) through topic search at k = 3, 5, 10, and side by side through the legacy keyword search over 14, 30 and 90-day article windows. For every contact surfaced, the harness pulled their recent articles and wrote everything to a spreadsheet, one tab per query, so a human could judge whether each journalist covered the topic.
The internal service was built for article text
Our internal embedding service is set up for article text. It splits each article into chunks of about 100 words before embedding them, so the model usually sees a paragraph with plenty of context. In our case, we are comparing a two or three-word search phrase against two or three-word topic labels, with no surrounding context to lean on. On inputs that short, the model matched on spelling more than meaning. Some examples from the comparison:
| Query | Internal service | Short-text model |
|---|---|---|
"cyclospora" (a food-borne parasite) | 21 contacts; top 10 included "cyclocross", a cycling discipline | 11 contacts, all covering cyclospora outbreaks |
"Korean skincare" | 26 contacts, only exact K-beauty labels | 379 contacts, reaching skincare specialists |
"Proposition 12" | 0 contacts | 24 contacts, covering California ballot measures |
"cyclospora" and "cyclocross" only share a prefix, but the internal service placed them close enough to rank a cycling reporter for a food-safety query. It also missed related topics. It did not connect "Korean skincare" with "skincare", or a ballot measure with ballot measures coverage.
We then tested models built for short text, with the search mechanics unchanged:
- SBERT (
all-mpnet-base-v2), a sentence embedding model run in-process through ONNX Runtime. It needs no network hop, but it requires a glibc-based container image and model files baked into the image. - Azure OpenAI
text-embedding-3-smallrequested at 256 dimensions so it fits the same index layout as before.
During the evaluation, each model had its own topics index and its own cache namespace, selected per request. Vectors from different models are not comparable, and a cached vector from one model used against another model’s index would return wrong results. text-embedding-3-small handled short labels best, so we switched to it. Its results are in the right-hand column above.
Orphan Topic Cleanup
Fluid topics change over time. Every time the agent re-enriches a journalist, some topics are replaced, and a topic that no remaining contact carries becomes an orphan. Orphans cause problems in search. Stage one can return an orphan as one of the top k topics, and stage two then finds zero contacts for that slot. With k = 10, a few orphans near a popular query visibly shrink the result list.
We clean up orphans in two places:
- Database: a scheduled job in the contact store deletes topic rows with no remaining contact mapping.
- Search index: a scheduled job checks each topic in the topics index and deletes the ones no contact carries.
Descriptor Fallback
Even with a better model, queries for niche terms still returned weak results. The table above shows the short-text model finding 24 contacts for "Proposition 12", but they were people covering California ballot measures in general. Embeddings of a bare name carry no knowledge of what it refers to. In our measurements, "Proposition 12" (a California ballot measure on farm animal welfare) sat at a cosine of about 0.10 from "Animal Welfare", further apart than two unrelated words, because nothing in the string is about animals. "Catch-22" landed in a cluster of fishing topics because “catch” was read literally. Legislation, court cases, book titles, codenames and scandals all behave this way, and PR teams search for them often.
We use an LLM to turn the name into a short descriptor of its subject. A descriptor such as "farm animal welfare law" reaches "Animal Welfare" at about 0.73. In the comparison run, we searched on the descriptor directly. The LLM described "Proposition 12" as "animal welfare", and the results went from 24 loosely related ballot-measure reporters to 567 animal welfare specialists. The prompt is short and uses examples:
Catch-22 -> no-win bureaucratic paradox
Roe v. Wade -> abortion rights ruling
SB 1047 -> AI safety regulation
Dieselgate -> vehicle emissions scandal
renewable energy -> NONE
In production, the descriptor is only a fallback. Generating one took 3 to 9 seconds per query in our tests, far longer than the search itself, so we only call the LLM when the original query returns no contacts. This means "Proposition 12" does not trigger it today. It returns 24 contacts, and the user gets those quickly instead of waiting for a better list. We accepted that trade-off because most queries return reasonable results on their own. Descriptors are cached in Redis, so a repeated query never calls the model again.
Conclusion
Fluid topics describe journalists in their own terms and let customers search in theirs. It took several iterations to get here. Most of the lessons apply beyond our use case:
- Search distinct concepts, then the entities that carry them. Running kNN directly over vectors duplicated across thousands of documents makes
kmeaningless, because the top results fill up with copies of the same topic. A small dedicated index of distinct topics fixes this. - Evaluate with humans in the loop. The side-by-side comparison against our legacy search, with articles attached, showed us the shortcomings of each approach and also confirmed the fix.
- Embeddings do not know what niche terms refer to. A small LLM call that expands these words into descriptions fixes queries that no embedding model can.
Fluid topics are now live in contact search. We built topic matching first because our data showed that most user queries are about topics. Topic matching uses kNN, which returns whatever sits closest in embedding space. For concepts, close is good enough. For names, it is not. A query for Tesla lands near electric vehicles in general, so a journalist who has written about electric vehicles but never about Tesla can still rank highly. The next phase, intent mining on the query, solves this. It identifies the people, products, organisations and locations in what the user types and matches them against the entities in journalists’ articles. A search for Tesla then returns journalists who wrote about Tesla.
Topics tell us what a journalist covers. Entities tell us exactly who and what they write about. Together they bring contact search closer to how our customers think about the journalists they want to reach.
