Creating & Ingesting Your Own Embeddings in Weaviate | Vector Databases for Beginners | Part 7

Why It Matters

It demonstrates a low‑cost, end‑to‑end method for deploying custom embeddings in a production‑grade vector database, accelerating AI‑driven search and recommendation prototypes.

Key Takeaways

• Use SentenceTransformers to generate embeddings for text data.
• Sample 100 papers from arXiv dataset for demo purposes.
• Configure Weaviate collection with vectorizer set to none.
• Insert embeddings via insert_many into Weaviate sandbox cluster.
• Free Weaviate Cloud sandbox enables quick vector database testing.

Summary

The video walks viewers through building custom text embeddings with a SentenceTransformers model from HuggingFace and loading them into a Weaviate vector database. The presenter demonstrates the workflow in a Google Colab notebook, pulling a subset of 100 arXiv paper titles and abstracts, generating embeddings for each record, and attaching them to a pandas DataFrame.

Key steps include installing the transformers and sentence‑transformers libraries, selecting the “modern‑bert‑base” model, iterating over the sample to compute vectors, and preparing the data schema for Weaviate. A free Weaviate Cloud sandbox cluster is created in Europe, the client connection is configured with the endpoint URL and API key, and the collection is defined with a vectorizer set to none because the vectors are pre‑computed.

The tutorial then uses the client’s insert_many method to bulk‑load the title, abstract, combined text, and the generated embeddings into the newly created collection. The presenter verifies the upload by refreshing the Weaviate console explorer, where the vectors and associated metadata appear correctly.

By showing a complete end‑to‑end pipeline—from data sampling and embedding generation to cloud‑based vector storage—the video illustrates how developers can quickly prototype semantic search or recommendation systems without managing their own infrastructure, leveraging Weaviate’s free sandbox for experimentation.

Original Description

In part 7, we walk through a full hands-on workflow for generating embeddings externally and importing them into Weaviate using the Bring Your Own Vectors approach.

In this section, we're going to go over:

- Generating embeddings using a Hugging Face model (ModernBERT) in Google Colab

- Sampling and preparing a large dataset for embedding generation

- Converting text (titles + abstracts) into vector embeddings using Sentence Transformers

- Setting up a free Weaviate Cloud sandbox cluster

- Connecting Colab to Weaviate using API keys and cluster endpoints

- Creating a custom collection with vectorizer disabled for external embeddings

- Inserting embeddings, metadata, and text into Weaviate at scale

This workflow shows how easy it is to bring your own vectors into Weaviate and manage your embeddings end-to-end—giving you full control over vector generation, storage, and retrieval.

#EmbeddingGeneration #Weaviate #APIIntegration #SentenceTransformers #GoogleColab

.

.

.

Learn data science, AI, and machine learning through our hands-on training programs: https://www.youtube.com/@Datasciencedojo/courses

Check our community webinars in this playlist: https://www.youtube.com/playlist?list=PL8eNk_zTBST-EBv2LDSW9Wx_V4Gy5OPFT

Check our latest Future of Data and AI Conference: https://www.youtube.com/playlist?list=PL8eNk_zTBST9Wkc6-bczfbClBbSKnT2nI

Subscribe to our newsletter for data science content & infographics: https://datasciencedojo.com/newsletter/

Love podcasts? Check out our Future of Data and AI Podcast with industry-expert guests: https://www.youtube.com/playlist?list=PL8eNk_zTBST_jMlmiokwBVfS_BqbAt0z2