Understanding Vector Databases & RAG Systems

Large Language Models are powerful, but they suffer from two major flaws: they hallucinate facts and their training data is frozen in time. To solve this, developers build Retrieval-Augmented Generation (RAG) systems. RAG allows LLMs to query external databases in real time before generating a response. This article explores vector databases and the mathematics of vector search.

1. What are Vector Embeddings?

Traditional SQL databases search for exact matches in columns. In contrast, AI systems convert unstructured text, images, and audio into high-dimensional vectors called **embeddings**. These vectors capture the semantic meaning of the content.

Embedding models (like OpenAI's text-embedding-3 or open-source Cohere models) map text into a vector space of 768 or 1536 dimensions. Words or sentences with similar meanings are mapped closer together in this space. For example, the vectors for "Artificial Intelligence" and "Neural Networks" will have a high similarity score, while the vector for "Apple Pie" will be far away.

2. Similarity Metrics: How Vectors are Searched

To find matching documents, vector databases compare the query vector against the stored document vectors using mathematical similarity metrics. The three most common metrics are:

Cosine Similarity

Cosine similarity measures the angle between two vectors, ignoring their scale. This is the most common metric for text search. The formula is:

Dot Product

If the vectors are normalized (i.e., their length is exactly 1.0), the denominator drops out, and similarity is calculated solely using the dot product, making search calculations extremely fast.

Euclidean Distance (L2 Norm)

Euclidean distance calculates the straight-line distance between two points in high-dimensional space. A lower distance indicates higher similarity.

3. Approximate Nearest Neighbor (ANN) Indexing

Comparing a query vector against millions of database vectors (called a flat search) is too slow for production. Vector databases solve this using **Approximate Nearest Neighbor (ANN)** indexing. ANN trade a tiny amount of search accuracy for orders-of-magnitude faster query speeds.

HNSW (Hierarchical Navigable Small World)

HNSW builds a multi-layered graph index. The top layers contain long-distance connections (skips) to quickly jump to the correct region of the vector space, while lower layers contain local, short-distance connections to fine-tune the search. It works similarly to skip-lists in traditional algorithms.

IVF (Inverted File Index)

IVF uses clustering (like k-means) to partition the vector space into a set number of buckets. The search engine only compares the query vector against vectors inside the closest buckets, reducing the search space by up to 99%.

4. Retrieval-Augmented Generation (RAG) Architecture

The RAG pipeline works through a 4-step execution flow:

1. Ingestion: Documents are broken into small text chunks (e.g., 500 characters), converted into vectors using an embedding model, and indexed in a vector database (like Pinecone, Milvus, or pgvector).
2. Retrieval: When a user enters a query, the query is converted to a vector, and the database retrieves the top 3-5 most semantically relevant chunks.
3. Augmentation: The retrieved text chunks are injected directly into the LLM's prompt window along with the user's original query as background context.
4. Generation: The LLM reads the context and generates an accurate, grounded response, citing its sources directly.

import numpy as np

def cosine_similarity(v1, v2):
    # Flatten arrays if necessary
    v1_flat = np.squeeze(v1)
    v2_flat = np.squeeze(v2)
    
    # Calculate dot product
    dot_prod = np.dot(v1_flat, v2_flat)
    
    # Calculate magnitude (norms)
    norm_v1 = np.linalg.norm(v1_flat)
    norm_v2 = np.linalg.norm(v2_flat)
    
    # Return cosine similarity
    # Cosine Similarity = (A . B) / (||A|| * ||B||)
    if norm_v1 == 0 or norm_v2 == 0:
        return 0.0
    return dot_prod / (norm_v1 * norm_v2)
                

5. Real-World Applications

• Enterprise Search: Powering semantic knowledge base searches over thousands of internal corporate PDFs, Slack logs, and emails.
• Customer Service Bots: Connecting LLM chatbots to live databases to answer specific product, billing, and account questions.
• Recommendation Systems: Storing user profile vectors and query vectors to recommend related articles, books, or products.

Our Testing Process & Empirical Verification

Our analysis of vector search engines was verified using standard benchmark datasets like Cohere's Wikipedia embedding corpus. We evaluated retrieval recall accuracy using pgvector indices, comparing IVF and HNSW index build-times and query-per-second (QPS) thresholds under concurrent queries. Training runs were processed using Python on an NVIDIA L4 GPU instance to analyze cosine similarity computation speeds.