Production NLP systems use embeddings differently depending on the task. Semantic search, document classification, topic clustering, and duplicate detection each require distinct embedding strategies, model choices, and similarity metrics. Getting these details right is what separates demos from deployed systems.

Embeddings in NLP are not one-size-fits-all. Each task demands a specific configuration. For semantic search and RAG: use asymmetric embedding models (e.g., msmarco-distilbert) where query and document representations are optimized differently, since short queries should match long documents. Use cosine similarity as the distance metric and HNSW indexing for fast retrieval. For document classification: fine-tune a sentence transformer on labeled examples rather than relying on zero-shot embeddings. Even 100–500 labeled examples can yield substantial accuracy gains via SetFit, a few-shot fine-tuning technique for sentence transformers. For topic clustering: embed documents and apply dimensionality reduction (UMAP to 2–5 dimensions) before clustering with HDBSCAN. BERTopic automates this pipeline and produces human-interpretable topic labels. For duplicate and near-duplicate detection: use locality-sensitive hashing (LSH) on embedding vectors to efficiently identify candidates before exact cosine comparison — essential for deduplicating datasets at scale. For multilingual NLP: models like LaBSE and paraphrase-multilingual-mpnet-base-v2 produce language-agnostic embeddings where semantic equivalents in different languages cluster together. Choosing the right embedding strategy per task is the practitioner skill that most impacts real-world NLP system quality.

Fine-Grained NLP with Embeddings: Retrieval, Classification, and Clustering