Traditional vocational training is evolving fast. Textbooks, rigid schedules, and costly equipment are giving way to dynamic, cloud-based platforms that bring hands-on learning to any device. From AI-driven welding simulators to cloud-powered automotive diagnostics, new tools are making skill development more accessible, cost-effective, and responsive to industry demands. These innovations aren’t just enhancing training – […]
Cloud computing still continues to be a dominant force in the technological world. Many rely on it for purposes such as data storage, networking, virtualization, and others. As such, there is an ongoing demand that should be fulfilled by those who are familiar with cloud computing technology. This guide will discuss the key skills that […]
Block storage is a fundamental technology in modern data storage solutions, enabling efficient and high-performance access to data for a wide range of applications, from databases to file systems. Designed to provide flexibility and scalability, block storage allows data to be stored in blocks that can be independently managed, which streamlines storage management across different […]
Introduction Object storage is a modern data storage architecture designed to handle vast amounts of unstructured data. Unlike traditional storage methods such as block or file storage, object storage excels in managing large-scale data like multimedia, backups, and archives. This tutorial introduces the core concepts of object storage, its benefits, and how it fits into […]
The PostgreSQL Global Development Group has unveiled PostgreSQL 17, the latest iteration of the world’s most advanced open-source database. Building on over 35 years of engineering excellence and a vibrant global community, PostgreSQL 17 introduces significant performance enhancements, scalability improvements, and new features tailored for both experienced developers and growing data management needs. Why PostgreSQL […]
Are you struggling to manage your data infrastructure on Kubernetes? Do you want a solution that is easy to use, powerful, and scalable? Then you might need KubeBlocks!
KubeBlocks is an open-source platform that makes it easy to deploy, manage, and scale databases, data warehouses, and other data-intensive applications on Kubernetes. It provides a rich set of features for monitoring, backup, and recovery, and it supports a wide variety of databases, including MySQL, PostgreSQL, MongoDB, Kafka, Redis, Vector databases, Pulsar, and more.
▬▬▬▬▬▬ Sponsorships ▬▬▬▬▬▬ If you are interested in sponsoring this channel, please use https://calendly.com/vfarcic/meet to book a timeslot that suits and we’ll go over the details. Or feel free to contact me over Twitter or LinkedIn (see below)
Today we delve into the world of Kubernetes resources, Custom Resources (CRs), Custom Resource Definitions (CRDs), and Operators. Learn how these work together.
▬▬▬▬▬▬ Sponsoships ▬▬▬▬▬▬ If you are interested in sponsoring this channel, please use https://calendly.com/vfarcic/meet to book a timeslot that suits and we’ll go over the details. Or feel free to contact me over Twitter or LinkedIn (see below)
Explore the power and convenience of Aiven, a single service that allows you to manage all your databases in the cloud. Discover how Aiven simplifies database management, provides easy scalability, and ensures robust security. Take your cloud infrastructure to the next level with Aiven’s comprehensive solution. Use https://console.aiven.io/signup?referral_code=rgmrwa4y5i7r9o9y1a1u for extra free credits
▬▬▬▬▬▬ Sponsoships ▬▬▬▬▬▬ If you are interested in sponsoring this channel, please use https://calendly.com/vfarcic/meet to book a timeslot that suits and we’ll go over the details. Or feel free to contact me over Twitter or LinkedIn (see below)
Vector databases have been getting a lot of attention since the developer community realized how they can enhance large language models (LLMs). Weaviate is an open source vector database that enables modern search capabilities, such as vector search, hybrid search, and generative search. With Weaviate, you can build advanced LLM applications, next-level search systems, recommendation systems, and more.
This article explains what vector databases are and highlights key features of the Weaviate vector database. Learn how to install Weaviate on Docker using Docker Compose so you can take advantage of semantic search within your Dockerized environment.
Introducing the Weaviate vector database
The core feature of vector databases is storing vector embeddings of data objects. This functionality is especially helpful with the growing amount of unstructured data (e.g., text or images), which is difficult to manage and process with traditional relational databases. The vector embeddings are a numerical representation of the data objects — usually generated by a machine learning (ML) model — and enable the search and retrieval of data based on semantic similarity (vector search).
Vector databases do much more than just store vector embeddings: As you can imagine, retrieving data based on similarity requires a lot of comparing between objects and thus can take a long time. In contrast to other types of databases that can store vector embeddings, a vector database can retrieve data fast. To enable low-latency search queries, vector databases use specific algorithms to index the data.
Additionally, some vector databases, like Weaviate, store the vector embeddings and the original data object, which lets you combine traditional search with modern vector search for more accurate search results.
With these functionalities, vector databases are usually used in search or similar tasks (e.g., recommender systems). With the recent advancements in the LLM space, however, vector databases have also proven effective at providing long-term memory and domain-specific context to conversational LLMs. This means that you can leverage LLM capabilities on your private data or your specific field of expertise.
Key highlights of the Weaviate vector database include:
Open source: Weaviate is open source and available for anybody to use wherever they want. It is also available as a managed service with SaaS and hybrid SaaS options.
Horizontal scalability: You can scale seamlessly into billions of data objects for your exact needs, such as maximum ingestion, largest possible dataset size, maximum queries per second, etc.
Lightning-fast vector search: You can perform lightning-fast pure vector similarity search over raw vectors or data objects, even with filters. Weaviate typically performs nearest-neighbor searches of millions of objects in considerably less than 100ms (see our benchmark).
Combined keyword and vector search (hybrid search): You can store both data objects and vector embeddings. This approach allows you to combine keyword-based and vector searches for state-of-the-art search results.
Optimized for cloud-native environments: Weaviate has the fault tolerance of a cloud-native database, and the core Docker image is comparably small at 18 MB.
Modular ecosystem for seamless integrations: You can use Weaviate standalone (aka “bring your own vectors”) or with various optional modules that integrate directly with OpenAI, Cohere, Hugging Face, etc., to enable easy use of state-of-the-art ML models. These modules can be used as vectorizers to automatically vectorize any media type (text, images, etc.) or as generative modules to extend Weaviate’s core capabilities (e.g., question answering, generative search, etc.).
Prerequisites
Ensure you have both the docker and the docker-compose CLI tools installed. For the following section, we assume you have Docker 17.09.0 or higher and Docker Compose V2 installed. If your system has Docker Compose V1 installed instead of V2, use docker-compose instead of docker compose. You can check your Docker Compose version with:
$ docker compose version
How to Configure the Docker Compose File for Weaviate
To start Weaviate with Docker Compose, you need a Docker Compose configuration file, typically called docker-compose.yml. Usually, there’s no need to obtain individual images, as we distribute entire Docker Compose files.
You can obtain a Docker Compose file for Weaviate in two different ways:
Docker Compose configurator on the Weaviate website (recommended): The configurator allows you to customize your docker-compose.yml file for your purposes (including all module containers) and directly download it.
Manually: Alternatively, if you don’t want to use the configurator, copy and paste one of the example files from the documentation and manually modify it.
Setting up a persistent volume to avoid data loss when you restart the container and improve reading and writing speeds is recommended.
You can set a persistent volume in two ways:
With a named volume: Docker will create a named volume weaviate_data and mount it to the PERSISTENCE_DATA_PATH inside the container after starting Weaviate with Docker Compose:
services:
weaviate:
volumes:
- weaviate_data:/var/lib/weaviate
# etc.
volumes:
weaviate_data:
With host binding: Docker will mount ./weaviate_data on the host to the PERSISTENCE_DATA_PATH inside the container after starting Weaviate with Docker Compose:
services:
weaviate:
volumes:
- ./weaviate_data:/var/lib/weaviate
# etc.
Step 3: Modules
Weaviate can be used with various modules, which integrate directly with inferencing services like OpenAI, Cohere, or Hugging Face. These modules can be used to vectorize any media type at import and search time automatically or to extend Weaviate’s core capabilities with generative modules.
You can also use Weaviate without any modules (standalone). In this case, no model inference is performed at import or search time, meaning you need to provide your own vectors in both scenarios. If you don’t need any modules, you can skip to Step 4: Runtime.
Configure modules for Docker Compose file (Figure 3):
Figure 3: The Weaviate Docker Compose configurator step to define if modules will be used, or if running standalone is desired.
Currently, Weaviate integrates three categories of modules:
Retriever and vectorizer modules automatically vectorize any media type (text, images, etc.) at import and search time. There are also re-ranker modules available for re-ranking search results.
Reader and generator modules can be used to extend Weaviate’s core capabilities after retrieving the data for generative search, question answering, named entity recognition (NER), and summarization.
Other modules are available for spell checking or for enabling using your custom modules.
Note that many modules (e.g., transformer models) are neural networks built to run on GPUs. Although you can run them on CPU, enabling GPU `ENABLE_CUDA=1`, if available, will result in faster inference.
In the final step of the configurator, select Docker Compose for your runtime (Figure 4):
Figure 4: The final step of the Weaviate Docker Compose configurator where “Docker Compose” can be selected as the runtime.
Step 5: Download and further customization
Once your configuration is complete, you will see a snippet similar to the following to download the docker-compose.yml file, which has been adjusted to your selected configuration.
After downloading the Docker Compose file from the configurator, you can directly start Weaviate on Docker or customize it further.
You can set additional environment variables to further customize your Weaviate setup (e.g., by defining authentication and authorization). Additionally, you can create a multi-node setup with Weaviate by defining a founding member and other members in the cluster.
Founding member: Set up one node as a “founding” member by configuring CLUSTER_GOSSIP_BIND_PORT and CLUSTER_DATA_BIND_PORT:
weaviate-node-1: # Founding member service name
... # truncated for brevity
environment:
CLUSTER_HOSTNAME: 'node1'
CLUSTER_GOSSIP_BIND_PORT: '7100'
CLUSTER_DATA_BIND_PORT: '7101'
Other members in the cluster: For each further node, configure CLUSTER_GOSSIP_BIND_PORT and CLUSTER_DATA_BIND_PORT and configure to join the founding member’s cluster using the CLUSTER_JOIN variable:
weaviate-node-2:
... # truncated for brevity
environment:
CLUSTER_HOSTNAME: 'node2'
CLUSTER_GOSSIP_BIND_PORT: '7102'
CLUSTER_DATA_BIND_PORT: '7103'
CLUSTER_JOIN: 'weaviate-node-1:7100' # This must be the service name of the "founding" member node.
Optionally, you can set a hostname for each node using CLUSTER_HOSTNAME.
Note that it’s a Weaviate convention to set the CLUSTER_DATA_BIND_PORT to 1 higher than CLUSTER_GOSSIP_BIND_PORT.
How to run Weaviate on Docker
Once you have your Docker Compose file configured to your needs, you can run Weaviate in your Docker environment.
Start Weaviate
Before starting Weaviate on Docker, ensure that the Docker Compose file is named exactly docker-compose.yml and that you are in the same folder as the Docker Compose file.
Then, you can start up with the whole setup by running:
$ docker compose up -d
The -d option runs containers in detached mode. This means that your terminal will not attach to the log outputs of all the containers.
If you want to attach to the logs of specific containers (e.g., Weaviate), you can run the following command:
Congratulations! Weaviate is now running and is ready to be used.
Stop Weaviate
To avoid losing your data, shut down Weaviate with the following command:
$ docker compose down
This will write all the files from memory to disk.
Conclusion
This article introduced vector databases and how they can enhance LLM applications. Specifically, we highlighted the open source vector database Weaviate, whose advantages include fast vector search at scale, hybrid search, and integration modules to state-of-the-art ML models from OpenAI, Cohere, Hugging Face, etc.
We also provided a step-by-step guide on how to install Weaviate on Docker using Docker Compose, noting that you can obtain a docker-compose.yml file from the Weaviate Docker Compose configurator, which helps you to customize your Docker Compose file for your specific needs.
Visit our AI/ML page and read the article collection to learn more about how developers are using Docker to accelerate the development of their AI/ML applications.
This video explores the fundamentals to master database schema management using Atlas Kubernetes Operator. Learn how to efficiently manage your database schemas in a Kubernetes environment, improving scalability and reducing maintenance efforts.
▬▬▬▬▬▬ Sponsoships ▬▬▬▬▬▬ If you are interested in sponsoring this channel, please use https://calendly.com/vfarcic/meet to book a timeslot that suits and we’ll go over the details. Or feel free to contact me over Twitter or LinkedIn (see below)
Rate limiting is a crucial mechanism used to control the flow of incoming requests and protect web applications and APIs from abuse and overload. It ensures that clients do not make an excessive number of requests within a specified time window. Redis, as a high-performance in-memory data store, provides an excellent solution for implementing rate […]
Should we run Databases in Kubernetes? If we should, what’s the best way to do it? In this video, we’ll explore the options and see how CloudNativePG can help us.
▬▬▬▬▬▬ Sponsoships ▬▬▬▬▬▬ If you are interested in sponsoring this channel, please use https://calendly.com/vfarcic/meet to book a timeslot that suits and we’ll go over the details. Or feel free to contact me over Twitter or LinkedIn (see below)
It’s time to build an internal developer platform (IDO) with Crossplane, Argo CD, SchemaHero, External Secrets Operator (ESO), GitHub Actions, Port, and a few others.
▬▬▬▬▬▬ Sponsoships ▬▬▬▬▬▬ If you are interested in sponsoring this channel, please use https://calendly.com/vfarcic/meet to book a timeslot that suits and we’ll go over the details. Or feel free to contact me over Twitter or LinkedIn (see below)
Are you interested in learning more about Graph Databases and Docker? Do you want to connect with like-minded professionals and expand your network in these areas? Then mark your calendars for March 11th, 2023, because the Docker Bangalore and Graph Database Bangalore Meetup groups are coming together to host a half-day event that you won’t […]
Connexion
Traditional vocational training is evolving fast. Textbooks, rigid schedules, and costly equipment are giving way to dynamic, cloud-based platforms that bring hands-on learning to any device. From AI-driven welding simulators to cloud-powered automotive diagnostics, new tools are making skill development more accessible, cost-effective, and responsive to industry demands. These innovations aren’t just enhancing training – […]