AI Hardware & Infrastructure · 19 Nov 2026 · 00:58 CET
Exa boosts embedding throughput by 50% to index billions of documents on AWS AI chips

Publisher preview · OZZZER analysis pending editorial review.
PUBLISHER ARTICLE PREVIEW
From the original article
At Exa, we are building one of the few true web indexes by aggregating and serving knowledge from across the web, along with private and synthetic sources. We optimize the index for semantic search, grounding agents and humans in true, high-quality data. We use multi-tower semantic machine learning (ML) models to embed web data and query data into the same vector space, so users can express complex queries in natural language alongside traditional keyword search.
The keystone technology behind this is an in-house vector database. It must ingest as much high-quality web data as possible and embed all of it. The unit economics of embedding therefore underlie everything we do. At the scale of tens or hundreds of billions of documents, finding the hardware that delivers the best performance per dollar isn’t a tuning detail.
Any per-document cost improvement compounds into a significant number.
This post details how we optimized our embedding infrastructure for cheaper, more flexible inference workloads on AWS Neuron hardware (AWS Inferentia2 and AWS Trainium2), and how an agentic development workflow drove the optimization.
Inference runs on an asynchronous pipeline, but it must be high-volume and fast to serve recent data quickly and shrink the time between discovery and serving.
Every structured web document we rank as high enough quality must be embedded before it can be served. This figure shows how raw web data moves from Crawlers into an asynchronous pipeline of Parse, Moderate and filter, then Embed, with the result served from a vector database.
Our experience with the Neuron hardware family began with a backfill. Moving to a superior document model meant re-embedding our existing corpus. We selected 30 billion high-priority documents for re-embedding. A migration like this temporarily doubles the embedding load, which made it exactly the right moment to re-evaluate our hardware choices.
The preceding figure shows two parallel pipelines sharing the crawl and parse stages, one embedding with the live model into the live index and one re-embedding priority documents with the new model into the new index, which is what doubles the load. This doubling was a clear trigger to reconsider our hardware and optimization choices.
AWS Inferentia2 figures refer to performance on the smallest instance type, an Inferentia2 xlarge instance (inf2.xlarge), with one chip and two NeuronCores.
Source
AWS AI · 19 Nov 2026 · 00:58 CET
Open the original at AWS AI ↗