Semantic Search in Python with FAISS, ScaNN, ChromaDB, and Cosine Similarity

Keyword-based search works well when users know exactly what words exist in the data.

But real searches are rarely that clean.

A user might search for:

“people working on machine learning”

while the underlying data contains:

“AI Engineer specializing in PyTorch and neural networks”

Traditional keyword matching may struggle because the words are different. Semantic search approaches the problem differently: instead of comparing words directly, it represents text as vectors and searches for items that are close in meaning.

That idea is what led me to build Semantic Search Lib, an early-stage Python project for experimenting with semantic search across multiple vector-search backends.

The goal is straightforward:

Use one semantic-search pipeline while being able to switch between different search engines.

The project currently supports:

Cosine Similarity
FAISS
ScaNN
ChromaDB

It also exposes the functionality through Python, a CLI, and FastAPI.

Important: This is an early-stage experimental project built for learning and iteration. It has known limitations and should not be considered production-ready.

What Is Semantic Search?

Traditional search usually compares the words in a query with the words stored in documents.

Semantic search tries to compare their meaning instead.

The basic process looks like this:

Data
  ↓
Text Representation
  ↓
Embeddings
  ↓
Vector Search
  ↓
Top-K Results

The key component is the embedding.

An embedding model converts a piece of text into a numerical vector.

Conceptually:

"Python developer working with machine learning" ↓

[0.13, -0.42, 0.81, 0.17, ...]

Another sentence with a similar meaning should produce a vector located relatively close to it in the embedding space.

That means a search query can also be embedded:

"AI engineer using Python"
↓
[0.11, -0.39, 0.79, 0.21, ...]

A vector-search algorithm can then find the stored embeddings most similar to the query vector.

Instead of asking:

“Do these sentences contain the same words?”

we can ask:

“Are these sentences located near each other in semantic space?”

Why I Built Semantic Search Lib

There are already powerful tools available for vector search.

FAISS exists.

ScaNN exists.

ChromaDB exists.

Cosine similarity is easy to implement with libraries such as scikit-learn.

The part I wanted to explore was slightly different.

I wanted to keep the surrounding semantic-search pipeline mostly consistent while experimenting with different search engines.

Instead of rebuilding everything when switching backends, I wanted something closer to:

search = SemanticSearch(backend="faiss")

or:

search = SemanticSearch(backend="chromadb")

The embedding and search workflow remains conceptually the same.

The underlying engine changes.

That makes the project useful as an experimentation layer for understanding how different approaches behave.

The Semantic Search Pipeline

The library follows five main stages.

1. Load the Data

The first step is obtaining the information that should become searchable.

The project currently includes support for working with structured CSV data.

Imagine a dataset containing:

name,age,role,company
John Doe,30,Engineer,CosmicCat
Jane Smith,28,Data Scientist,CosmicCat

Raw structured data isn't always the best representation to send directly into a language embedding model.

So the next step is to transform it.

2. Turn Structured Data Into Meaningful Text

The project includes a DataProcessor that can format rows using templates.

For example:

name: John Doe
age: 30
role: Engineer
company: CosmicCat

could become:

John Doe is a 30-year-old Engineer working at CosmicCat.

That sentence carries the structure of the original data while presenting it in a format that a sentence embedding model can interpret naturally.

This stage is important because embeddings are only as useful as the information represented in the text being embedded.

The pipeline therefore becomes:

Structured Data
      ↓
Template
      ↓
Natural-Language Representation
      ↓
Embedding

3. Generate Embeddings with Sentence Transformers

Semantic Search Lib uses Sentence Transformers to turn text into vector representations.

The default embedding model in the project is:

all-MiniLM-L6-v2

The embedding layer can also be configured to use another supported model.

Conceptually:

texts = [
    "John is a machine learning engineer.",
    "Sarah develops backend applications.",
]
embeddings = embedder.encode(texts)

The output is no longer human-readable text.

It is a matrix of numerical vectors.

Document 1 → [0.12, -0.34, 0.56, ...]
Document 2 → [0.08,  0.11, 0.42, ...]

These vectors are what the search backend indexes.

One Interface, Multiple Vector Search Backends

This is the central idea behind the project.

Rather than coupling the entire application to one search implementation, SemanticSearch provides a shared entry point for multiple backends.

Currently there are four.

Cosine Similarity

Cosine similarity is the simplest backend in the project and provides a useful baseline.

It measures the angle between two vectors rather than directly comparing their magnitude.

The closer their directions are, the more similar the vectors are considered.

For smaller datasets and experiments, this can be perfectly adequate.

It is also useful when learning because there is relatively little infrastructure between the embedding and the similarity calculation.

Conceptually:

Query Vector
     ↓
Compare against every document vector
     ↓
Calculate similarity
     ↓
Sort
     ↓
Return Top-K

The downside becomes more apparent as the number of vectors grows.

FAISS

FAISS is a vector similarity-search library designed for efficient similarity search over dense vectors.

Instead of treating the vector index as a simple array that must always be processed manually, FAISS provides specialized indexing and search structures.

In Semantic Search Lib, FAISS can be selected as the backend while keeping the higher-level search workflow similar.

For example:

search = SemanticSearch(backend="faiss")

The rest of the application shouldn't need an entirely different semantic-search architecture just because the underlying engine changed.

ScaNN

The project also includes support for ScaNN, Google's Scalable Nearest Neighbors implementation.

ScaNN is designed around efficient nearest-neighbor search and is another useful backend to experiment with when learning about vector retrieval.

From the library's point of view, the important idea remains the same:

SemanticSearch
      ↓
Shared workflow
      ↓
ScaNN backend

This is exactly the abstraction I wanted to explore with this project.

ChromaDB

The fourth backend is ChromaDB.

ChromaDB differs slightly from simply running a mathematical similarity function because it is designed as a vector database.

That means it can work with embeddings while also storing documents and associated information.

Again, the goal isn't to pretend all four technologies behave identically internally.

They don't.

The goal is to provide a common layer where I can experiment with them without rebuilding the entire semantic-search pipeline each time.

Searching Through One High-Level API

At a high level, the desired developer experience looks something like:

from semantic_search import SemanticSearch
search = SemanticSearch(
backend="faiss"
)
search.index(data)
results = search.search(
"machine learning engineer",
top_k=5
)

If I want to experiment with another backend, the architectural idea is that I should be able to change the backend rather than redesign the application.

For example:

search = SemanticSearch(
    backend="chromadb"
)

That is the main theme of the project:

Switch engines, not the entire semantic-search pipeline.

Three Ways to Use the Project

I also wanted the library to be accessible through more than one interface.

At the moment, there are three approaches.

Python Library

The most direct option is importing the semantic-search components into another Python project.

This is useful when semantic search is only one part of a larger application.

Your Python Application
        ↓
Semantic Search Lib
        ↓
Embedding Model
        ↓
Selected Backend

Command-Line Interface

There is also a CLI for experimenting with indexing and searching directly from the terminal.

The idea is to make simple experiments possible without first creating another application around the library.

This area is still early and is one of the parts of the project that needs additional work.

FastAPI

The project also includes a small FastAPI layer.

The API exposes operations for indexing data and performing searches.

Conceptually:

Client
  ↓
HTTP Request
  ↓
FastAPI
  ↓
SemanticSearch
  ↓
Vector Backend

This makes it possible to place the semantic-search layer behind an HTTP interface and call it from another application.

For example, a Next.js frontend or another backend service could eventually communicate with the search service through an API instead of importing the Python package directly.

Why Backend Abstraction Is Interesting

The purpose of this project isn't to claim that every vector-search engine is interchangeable.

Each backend has different behavior, capabilities, performance characteristics, storage models, and tradeoffs.

That's exactly why experimenting with several of them is useful.

A shared abstraction makes it easier to ask questions like:

How does the simple cosine baseline behave? How does FAISS compare? What changes when using ScaNN?

What does using a vector database such as ChromaDB change?

without letting the surrounding application dominate the experiment.

The application pipeline stays recognizable:

Data
   ↓
Representation
   ↓
Embedding
   ↓
Backend
   ↓
Results

Only the backend changes.

Current Limitations

Semantic Search Lib is deliberately being shared as an early-stage project.

It is not production-ready, and there are several things I still want to improve.

For example, some current backend implementations assume the 384-dimensional embeddings produced by the default all-MiniLM-L6-v2 model. Supporting arbitrary embedding models properly requires making index dimensions dynamic.

The CLI also needs better persistence. An index created in one process should be persisted or reloadable rather than existing only in application memory.

There is additional work needed around:

Persistent indexes Backend consistency Error handling Metadata and filtering Testing Configuration Performance Different embedding dimensions

Larger datasets

These aren't details I want to hide behind a polished open-source announcement.

They're part of the reason I'm publishing the project.

It represents a point in the building process, not the finished destination.

What I Learned From Building It

One useful takeaway from this project is that semantic search isn't really one technology.

It is a pipeline of decisions.

You have to decide:

How should the original data be represented?

Then:

Which embedding model should convert that representation into vectors?

Then:

Where should those vectors live?

Then:

How should nearest neighbors be found?

And finally:

How should the retrieved results be interpreted and returned to the application?

The vector database or search engine is important, but it is only one part of the system.

Thinking about the entire pipeline has been more valuable to me than simply connecting an embedding model to one vector database and calling the experiment finished.

What's Next?

There are several directions I want to explore as the project evolves.

Persistent indexes are an obvious improvement.

I'd also like to improve support for different embedding models, make backend behavior more consistent, strengthen automated testing, and experiment further with metadata and filtering.

Another interesting direction is hybrid search, where semantic vector retrieval can be combined with traditional lexical approaches such as BM25.

There is also plenty of room for benchmarking.

A shared interface becomes much more useful when it can help systematically compare:

search quality latency indexing performance memory usage different embedding models

different vector backends

But those are future iterations.

For now, the project remains intentionally small and experimental.

Try Semantic Search Lib

Semantic Search Lib is open source and available on GitHub:

github.com/pouyasolltani81/semantic-search-lib

If you're learning about:

semantic search
vector search
embeddings
Sentence Transformers
FAISS
ScaNN
ChromaDB
Python AI development

you may find the source code useful as a small project to explore.

And if you find an issue, have an idea, or want to improve something, feedback and contributions are welcome.

This is CosmicCat AI Builds #005 — another experiment in building, learning, and documenting the process.