Semantic-Retrieval-System-using-Sentence-Embeddings

Semantic Retrieval System using Sentence Embeddings

A high-quality academic project implementing a semantic retrieval system that returns the top-k most semantically similar sentences to a given query using sentence embeddings.

Project Overview

This project demonstrates a complete semantic retrieval pipeline using modern NLP techniques. The system converts sentences into dense vector representations (embeddings) and uses cosine similarity to find the most relevant matches for a given query.

Key Features

• Sentence Embeddings: Uses the pre-trained all-MiniLM-L6-v2 model from Sentence Transformers
• Cosine Similarity: Implements efficient cosine similarity search for retrieval
• FAISS Support: Optional FAISS index for faster search on larger datasets
• Evaluation Framework: Built-in evaluation with Precision@K metrics for 20 test queries
• Modular Design: Clean separation of concerns across multiple Python modules

Project Structure

.
├── dataset.py          # Dataset loading and generation
├── embedding.py        # Sentence embedding generation
├── retrieval.py        # Cosine similarity search
├── evaluation.py      # Evaluation metrics and test queries
├── main.py            # Main entry point
├── dataset.json       # Generated dataset (320 sentences)
├── dataset.txt        # Text version of dataset
└── README.md          # This file

Installation

Requirements

• Python 3.8+
• sentence-transformers
• numpy
• faiss-cpu (optional, for faster search)

Install Dependencies

pip install sentence-transformers numpy faiss-cpu

Usage

Basic Usage

Run the system in interactive mode:

python main.py --topk 5

Enter a query when prompted. The system will return the top-5 most similar sentences.

Run Evaluation

Evaluate the system using the 20 predefined test queries:

python main.py --topk 5 --evaluate --no-interactive

Using FAISS

For faster retrieval on larger datasets:

python main.py --topk 5 --use_faiss

Command-Line Options

Option	Description
--topk K	Number of results to retrieve (default: 5)
--evaluate	Run evaluation on 20 test queries
--use_faiss	Use FAISS index for faster search
--no-interactive	Disable interactive mode

Example Output

Query: "I forgot my password"

Top Results:
1. "I forgot my password; how can I recover it? Thanks." (0.84)
2. "I forgot my password; how can I recover it? Could you assist me?" (0.84)
3. "I forgot my password; how can I recover it? Any guidance would help." (0.82)
4. "I forgot my password; how can I recover it? (Variant 159)" (0.75)
5. "I forgot my password; how can I recover it? (Variant 82)" (0.75)

Evaluation Results

Query	Relevant Results	Precision@5
forgot password	1/5	0.20
cancel order	1/5	0.20
track shipment	1/5	0.20
refund	5/5	1.00
return item	1/5	0.20
apply promo code	5/5	1.00
gift card	1/5	0.20
payment method	2/5	0.40
exchange item	3/5	0.60
warranty	2/5	0.40
delivery delay	2/5	0.40
lost package	3/5	0.60
invoice	2/5	0.40
price match	0/5	0.00
subscription cancel	3/5	0.60
size availability	3/5	0.60
pre-order	2/5	0.40
gift wrap	1/5	0.20
digital receipt	2/5	0.40

Dataset

The project includes a dataset of 320 sentences in the e-commerce help-center domain, covering topics such as:

• Order tracking and shipping
• Returns and refunds
• Account management (password, email)
• Payment methods
• Warranty claims
• Gift cards and promotions

Technical Details

Embedding Model

The system uses sentence-transformers/all-MiniLM-L6-v2, a compact and efficient sentence embedding model that maps sentences to 384-dimensional dense vectors.

Similarity Metric

Cosine similarity is used to measure the semantic similarity between query and dataset embeddings. The embeddings are L2-normalized, making cosine similarity equivalent to dot product.

Retrieval Algorithm

The system implements two retrieval methods:

1. In-memory cosine similarity: Computes dot product between query vector and all dataset embeddings
2. FAISS Index: Uses Facebook AI Similarity Search (FAISS) with IndexFlatIP for accelerated retrieval

Error Analysis

Common Error Types

1. Synonym Errors

   • Example: "terminate membership" vs "cancel subscription"
   • Cause: Embeddings capture semantics but may not perfectly handle synonyms

2. Topic Similarity Errors

   • Example: "change password" returning "change email"
   • Cause: Related account tasks cluster together in embedding space

3. Keyword Mismatch

   • Example: "refund" retrieving "return product"
   • Cause: Overlapping vocabulary causes proximity in vector space

Why These Errors Occur

Sentence embeddings capture overall semantic meaning rather than exact keywords. In a single-domain dataset, slight shifts in emphasis or related topics can cause retrieval of plausible but imperfect matches. The model's understanding is based on contextual patterns in training data, which may not perfectly align with specific user intents.

License

This project is for academic purposes.

References

• Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks.
• Johnson, J., Douze, M., & Jegou, H. (2017). Billion-scale similarity search with GPUs.