What are Vector Embeddings and Vector Databases?

Why Rememberizer is more than just a database or keyword search engine

Rememberizer uses vector embeddings in vector databases to enable searches for semantic similarity within user knowledge sources. This is a fundamentally more advanced and nuanced form of information retrieval than simply looking for keywords in content through a traditional search engine or database.

Visualization of a multidimensional vector space

How Rememberizer Uses Vector Embeddings

In their most advanced form (as used by Rememberizer), vector embeddings are created by language models with architectures similar to the AI LLMs (Large Language Models) that underpin OpenAI's GPT models and ChatGPT service, as well as models/services from Google (Gemini), Anthropic (Claude), Meta (LLaMA), and others.

This makes vector embeddings a natural choice for discovering relevant knowledge to include in the context of AI model prompts. The technologies are complementary and conceptually related. For this reason, most providers of LLMs as a service also produce vector embeddings as a service (for example: Together AI's embeddings endpoint or OpenAI's text and code embeddings).

Understanding Vector Embeddings

What does a vector embedding look like? Consider a coordinate (x,y) in two dimensions. If it represents a line from the origin to this point, we can think of it as a line with a direction—in other words, a vector in two dimensions.

In the context of Rememberizer, a vector embedding is typically a list of several hundred numbers (often 768, 1024, or 1536) representing a vector in a high-dimensional space. This list of numbers can represent weights in a Transformer model that define the meaning in a phrase such as "A bolt of lightning out of the blue." This is fundamentally the same underlying representation of meaning used in models like GPT-4. As a result, a good vector embedding enables the same sophisticated understanding that we see in modern AI language models.

Beyond Text: Multimodal Embeddings

Vector embeddings can represent more than just text—they can also encode other types of data such as images or sound. With properly trained models, you can compare across media types, allowing a vector embedding of text to be compared to an image, or vice versa.

Currently, Rememberizer enables searches within the text component of user documents and knowledge. Text-to-image and image-to-text search capabilities are on Rememberizer's roadmap for future development.

Real-World Applications

Major technology companies leverage vector embeddings in their products:

• Google uses vector embeddings to power both their text search (text-to-text) and image search (text-to-image) capabilities (reference)

• Meta (Facebook) has implemented embeddings for their social network search (reference)

• Snapchat utilizes vector embeddings to understand context and serve targeted advertising (reference)

How Rememberizer's Vector Search Differs from Keyword Search

Keyword search finds exact matches or predetermined synonyms. In contrast, Rememberizer's vector search finds content that's conceptually related, even when different terminology is used. For example:

• A keyword search for "dog care" might miss a relevant document about "canine health maintenance"

• Rememberizer's vector search would recognize these concepts as semantically similar and return both

This capability makes Rememberizer particularly powerful for retrieving relevant information from diverse knowledge sources.

Coming soon: Vector Search Process Visualization

This diagram will illustrate the complete semantic search workflow in Rememberizer:

• Document chunking and preprocessing

• Vector embedding generation process

• Storage in vector database

• Search query embedding

• Similarity matching calculation

• Side-by-side comparison with traditional keyword search

Technical Resources

To deeply understand how vector embeddings and vector databases work:

• Start with the overview from Hugging Face

• Pinecone (a vector database service) offers a good introduction to vector embeddings

• Meta's FAISS library: "FAISS: A Library for Efficient Similarity Search and Clustering of Dense Vectors" by Johnson, Douze, and Jégou (2017) provides comprehensive insights into efficient vector similarity search (GitHub repository)

The Foundation of Modern AI

The technologies behind vector embeddings have evolved significantly over time:

• The 2017 paper "Attention Is All You Need" (reference) introduced the Transformer architecture that powers modern LLMs and advanced embedding models

• "Approximate Nearest Neighbors: Towards Removing the Curse of Dimensionality" (1998, 2010) established the theory for efficient similarity search in high-dimensional spaces

• BERT (2018, reference) demonstrated the power of bidirectional training for language understanding tasks

• Earlier methods like GloVe (2014, reference) and Word2Vec (2013, reference) laid the groundwork for neural word embeddings

For technical implementation details and developer-oriented guidance on using vector stores with Rememberizer, see Vector Stores.

One remarkable aspect of Transformer-based models is their scaling properties—as they use more data and have more parameters, their understanding and capabilities improve dramatically. This scaling property was observed with models like GPT-2 and has driven the rapid advancement of AI capabilities.

Google researchers were behind the original Transformer architecture described in "Attention Is All You Need" (patent reference), though many organizations have since built upon and extended this foundational work.