MongoDB is an open-source database program. Classified as a NoSQL database program, the data stored in MongoDB is in JSON-like format. Instead of tables and rows, it stores data in collections of documents. Each collection contains related documents, and each document is a JSON-like object. The data model of MongoDB provides the flexibility in storing data with an optional schema.
The MongoDB Kubernetes Operator is an extension for Kubernetes. It allows deployment and management of MongoDB server on a Kubernetes cluster. It supports the deployment of replica sets in which a group of MongoDB servers stores the same data providing redundancy and high availability for production usage.
This guide walks you through the deployment of a MongoDB replica set on Vultr Kubernetes Engine (VKE) using the MongoDB Kubernetes Operator, securing the deployment with a TLS certificate generated using cert-manager, and deploying Vultr Load Balancers for external access.
Deploy a Vultr Ubuntu 20.04 instance to use as a management workstation.
Deploy a Kubernetes cluster at Vultr with at least three or more nodes with 2GB RAM and 2 CPUs.
On the management workstation:
Install kubectl.
Download your VKE configuration and configure kubectl.
Install Vultr Container Storage Interface.
Install Vultr Cloud Controller Manager.
You'll perform the rest of the steps in this guide from your management workstation.
The MongoDB Kubernetes Operator is a custom resource definition and a controller that simplifies the MongoDB server's deployment and management. This section explains the steps to install the custom resource definition, the role bindings, and the operator.
Clone the MongoDB Kubernetes Operator repository.
# git clone https://github.com/mongodb/mongodb-kubernetes-operator.git
Switch to the repository directory.
# cd mongodb-kubernetes-operator
Install the custom resource definition.
# kubectl apply -f config/crd/bases/mongodbcommunity.mongodb.com_mongodbcommunity.yaml
The above command installs the MongoDBCommunity resource type in the cluster. This resource allows the deployment and scaling of the MongoDB replica set. You use this resource for defining the configuration of the MongoDB replica set.
Create a new namespace.
# kubectl create namespace mongodb
A namespace isolates the resources related to a single application in the cluster. The above command creates a new namespace named mongodb that you use in the next steps to contain all the resources related to the MongoDB replica set.
Install the necessary roles and role bindings.
# kubectl apply -k config/rbac -n mongodb
The above command installs a specific set of permissions required by the MongoDB Kubernetes Operator to interact with the resources in the cluster. The -n flag specifies the namespace to apply the changes.
Install the MongoDB Kubernetes Operator.
# kubectl apply -f config/manager/manager.yaml -n mongodb
The above command installs the MongoDB Kubernetes Operator in the cluster. It creates a new pod in the mongodb namespace and watches for state changes related to the MongoDBCommunity resource type.
Verify the installation.
# kubectl get pods -n mongodb
Output.
NAME READY STATUS RESTARTS AGE
mongodb-kubernetes-operator-648bf8cc59-mc5sm 1/1 Running 0 10s
The cert-manager add-on for Kubernetes provides custom resource definitions that allow issuing TLS certificates. It installs three custom resources in the cluster: Issuer, ClusterIssuer, and Certificate. The Certificate resource requires an issuer parent resource, and it generates a Kubernetes secret which contains a pair of the TLS certificate and the private key.
Install the cert-manager add-on.
# kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml
The above command installs all the Kubernetes resources provided by the cert-manager add-on in the cluster.
Create a new configuration directory.
# mkdir ~/mongodb-replica-set
Switch to the configuration directory.
# cd ~/mongodb-replica-set
This guide uses a self-signed TLS certificate to secure the MongoDB replica set. A valid TLS certificate requires a certificate authority signature. To sign the TLS certificate for the MongoDB replica set, you create an issuer that generates a self-signed certificate authority.
Using a text editor, create a new file named certificate.yaml.
# nano certificate.yaml
Add the following contents to the file.
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: mongodb-ca-issuer
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: mongodb-ca-cert
spec:
isCA: true
commonName: mongodb-ca-cert
secretName: mongodb-ca-cert-key-pair
privateKey:
algorithm: ECDSA
size: 256
issuerRef:
name: mongodb-ca-issuer
kind: Issuer
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: mongodb-issuer
spec:
ca:
secretName: mongodb-ca-cert-key-pair
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: mongodb-cert
spec:
commonName: '*.mongodb-svc.mongodb.svc.cluster.local'
secretName: mongodb-cert-key-pair
dnsNames:
- '*.mongodb-svc.mongodb.svc.cluster.local'
- 'mongodb-0.example.com'
- 'mongodb-1.example.com'
- 'mongodb-2.example.com'
issuerRef:
name: mongodb-issuer
kind: Issuer
The above configuration creates four resources: the mongodb-ca-issuer Issuer resource allows the creation of the mongodb-ca-cert Certificate resource, which is a self-signed certificate authority. It uses the self-signed certificate authority to create the mongodb-issuer Issuer resource, which allows creating the mongodb-cert Certificate resource, which you use to secure the MongoDB replica set.
The commonName and the dnsName attributes in the mongodb-cert Certificate resource contain the certificate's hostnames. The commonName attribute is a wildcard followed by the syntax of the hostname used by the MongoDB Kubernetes Operator to provision the servers locally. The syntax of the hostname is as follows:
<metadata.name>-<member-number>.<metadata.name>-svc.<namespace>.svc.cluster.local
The <metadata.name> is the name of the MongoDBCommunity resource. In the next steps, you build the configuration for the MongoDB replica set using mongodb as the metadata name and the number of MongoDB replica set members you want to spawn in the cluster.
Replace and add a list of additional hostnames for external access in the dnsName attribute according to the number of members you want to spawn. You use these additional hostnames as replica set horizons in the next steps.
Apply the configuration.
# kubectl apply -f certificate.yaml -n mongodb
Verify the certificate state.
# kubectl get certificate -n mongodb
Output.
NAME READY SECRET AGE
mongodb-ca-cert True mongodb-ca-cert-key-pair 11s
mongodb-cert True mongodb-cert-key-pair 10s
A MongoDB replica set is a group of two or more instances of the MongoDB server that store the same data to enable redundancy and high availability for production usage. The MongoDB Kubernetes Operator enables you to deploy a MongoDB replica set on your VKE cluster using a single declarative configuration file instead of deploying and configuring multiple servers. This section explains the configuration and deployment of a MongoDB replica set using the MongoDB Kubernetes Operator.
Create a new file named replica-set.yaml.
# nano replica-set.yaml
Add the following contents to the file.
apiVersion: v1
kind: Secret
metadata:
name: demo-admin-password
type: Opaque
stringData:
password: 'your_password'
---
apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
name: mongodb
spec:
members: 3
type: ReplicaSet
version: "5.0.5"
replicaSetHorizons:
- horizon: mongodb-0.example.com:27017
- horizon: mongodb-1.example.com:27017
- horizon: mongodb-2.example.com:27017
security:
authentication:
modes: ["SCRAM"]
tls:
enabled: true
certificateKeySecretRef:
name: mongodb-cert-key-pair
caCertificateSecretRef:
name: mongodb-cert-key-pair
users:
- name: demo-admin
db: admin
passwordSecretRef:
name: demo-admin-password
roles:
- name: clusterAdmin
db: admin
- name: userAdminAnyDatabase
db: admin
scramCredentialsSecretName: my-scram
statefulSet:
spec:
volumeClaimTemplates:
- metadata:
name: data-volume
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: vultr-block-storage
resources:
requests:
storage: 20Gi
- metadata:
name: logs-volume
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: vultr-block-storage-hdd
resources:
requests:
storage: 40Gi
The above configuration creates a MongoDBCommunity resource in the cluster and a Kubernetes secret containing a password for the administrator user. When you apply the configuration, the MongoDB Kubernetes Operator creates a Kubernetes StatefulSet to provision the MongoDB replica set. Each replica set member is a pod with two containers: a container for the mongod process binary and a container for the MongoDB agent, which handles the provisioning of the mongod process. Each pod consists of 2 volumes: a volume for storing the database and a volume for storing logs.
The following are the highlights of the configuration:
metadata.name - The name of the MongoDBCommunity resource, used as prefix of related resources.
spec.version - The version of MongoDB community server installed in the replica set.
spec.members - The number of replica set members.
spec.replicaSetHorizons - The DNS hostnames used for external access. They should match the number of replica set members. Ensure that the TLS certificate includes these hostnames.
spec.security.tls - Enable TLS security and define the Kubernetes secret created by cert-manager containing the TLS certificate key-pair.
spec.users - The name of the administrator user, the Kubernetes secret referring to the administrator user's password, and the name of the collection for storing authentication information.
spec.statefulSet.spec.volumeClaimTemplates - The storage class and the storage size for volumes created for each member in the replica set. The above configuration uses the storage classes defined by Vultr Container Storage Interface (CSI) to provision Vultr Block Storage for each volume. Refer to the Vultr Container Storage Interface documentation for more information.
If you want to set non-default values for the Kubernetes StatefulSet deployed by the MongoDB Kubernetes Operator, you can use the spec.statefulSet attribute. It provides direct access to the Kubernetes Staefulset configuration. Refer to the configuration samples for more information.
Apply the configuration.
# kubectl apply -f replica-set.yaml -n mongodb
Follow the replica set deployment progress.
# watch -n 0.1 kubectl get pods -n mongodb
It takes around 5 to 10 minutes to finish the deployment.
Output.
Every 0.1s: kubectl get pods -n mongodb
NAME READY STATUS RESTARTS AGE
mongodb-0 2/2 Running 0 3m6s
mongodb-1 2/2 Running 0 102s
mongodb-2 2/2 Running 0 55s
mongodb-kubernetes-operator-648bf8cc59-mc5sm 1/1 Running 0 10m
Verify the deployment.
# kubectl get mongodbcommunity -n mongodb
Output.
NAME PHASE VERSION
mongodb Running 5.0.5
If the deployment stops making any progress, you can use the following commands to troubleshoot and find the issue causing the delay.
# kubectl describe mongodbcommunity mongo -n mongodb
# kubectl describe pod mongo-0 -n mongodb
# kubectl get pvc -n mongodb
# kubectl get pv -n mongodb
The following are the issues you might face during the deployment of the replica set:
Not having enough CPU.
Not having enough Memory.
Vultr Block Storage limit exceeded.
Ensure that your VKE cluster has enough resources available and you do not exceed the total number of Vultr Block Storage volumes limit.
You can fetch the connection string and the credentials to connect to the MongoDB replica set inside the cluster with the credentials stored by the MongoDB Kubernetes Operator in the Kubernetes secrets.
This is the syntax for the name of the Kubernetes secrets resource that contains the credentials:
<metadata.name>-<auth-db>-<username>
Fetch the connection string.
# kubectl get secret mongodb-admin-demo-admin -n mongodb -o json
The above command returns JSON response with BASE64 encoded values.
Install the jq package.
# apt install jq
The above package lets you decode the BASE64 values in the JSON response.
Fetch the usable connection string.
# kubectl get secret mongodb-admin-demo-admin -n mongodb -o json | jq -r '.data | with_entries(.value |= @base64d)'
Output.
{
"connectionString.standard": "mongodb://demo-admin:[email protected]:27017,mongodb-1.mongodb-svc.mongodb.svc.cluster.local:27017,mongodb-2.mongodb-svc.mongodb.svc.cluster.local:27017/admin?ssl=true",
"connectionString.standardSrv": "mongodb+srv://demo-admin:[email protected]/admin?ssl=true",
"password": "your_password",
"username": "demo-admin"
}
You use the connectionString.standardSrv value and the TLS certificate to connect to the MongoDB replica set inside the cluster.
Enter the mongod container.
# kubectl exec -it mongodb-0 -c mongod -n mongodb -- /bin/bash
Connect to the MongoDB replica set using mongosh.
$ mongosh --tls --tlsCAFile /var/lib/tls/ca/*.pem --tlsCertificateKeyFile /var/lib/tls/server/*.pem "mongodb+srv://demo-admin:[email protected]/admin?ssl=true"
Fetch all databases.
mongodb [primary] admin> show dbs
Output.
admin 176 kB
config 172 kB
local 545 kB
Exit the mongosh session.
mongodb [primary] admin> quit()
Exit the mongod container.
$ exit
You secured the MongoDB replica set deployed in the cluster using the TLS certificate key pair. You need that certificate and the private key to connect to the MongoDB replica set. Every pod provisioned by the MongoDB Kubernetes Operator contains the certificate and the private key in /var/lib/tls in the mongod container.
You can manually enter the mongod container of the first pod using kubectl exec -it mongodb-0 -c mongod -n mongodb -- /bin/bash to copy and paste the certificate and the private key files or use the following shortcuts to fetch them.
Fetch the certificate file.
# kubectl exec mongodb-0 -c mongod -n mongodb -- /bin/bash -c "cat /var/lib/tls/ca/*.pem" > crt.pem
The above command creates a file named crt.pem in the working directory containing the certificate contents.
Fetch the private key file.
# kubectl exec mongodb-0 -c mongod -n mongodb -- /bin/bash -c "cat /var/lib/tls/server/*.pem" > key.pem
The above command creates a file named key.pem in the working directory containing the private key contents.
The Vultr Cloud Controller Manager provides the ability to provision Vultr Load Balancer with Kubernetes. It binds the LoadBalancer service type with the Vultr Load Balancer service. When you create a new LoadBalancer service, the Vultr Cloud Controller Manager provisions a new Vultr Load Balancer in your account and configures it according to the object specifications and the annotations. This section explains how to set up external access for the MongoDB replica using load balancers and DNS records.
Create a new file named external-access.yaml.
# nano external-access.yaml
Add the following contents to the file.
apiVersion: v1
kind: Service
metadata:
name: mongodb-lb-0
namespace: mongodb
annotations:
service.beta.kubernetes.io/vultr-loadbalancer-protocol: "tcp"
spec:
type: LoadBalancer
ports:
- name: mongodb
port: 27017
protocol: TCP
selector:
app: mongodb-svc
statefulset.kubernetes.io/pod-name: mongodb-0
---
apiVersion: v1
kind: Service
metadata:
name: mongodb-lb-1
namespace: mongodb
annotations:
service.beta.kubernetes.io/vultr-loadbalancer-protocol: "tcp"
spec:
type: LoadBalancer
ports:
- name: mongodb
port: 27017
protocol: TCP
selector:
app: mongodb-svc
statefulset.kubernetes.io/pod-name: mongodb-1
---
apiVersion: v1
kind: Service
metadata:
name: mongodb-lb-2
namespace: mongodb
annotations:
service.beta.kubernetes.io/vultr-loadbalancer-protocol: "tcp"
spec:
type: LoadBalancer
ports:
- name: mongodb
port: 27017
protocol: TCP
selector:
app: mongodb-svc
statefulset.kubernetes.io/pod-name: mongodb-2
The above configuration creates three LoadBalancer service objects in the cluster. You can adjust the number of load balancers according to the number of members in the replica set. Each LoadBalancer service object provisions a Vultr Load Balancer, which forwards TCP traffic on port 27017 (default MongoDB port) to individual members in the replica set.
Apply the configuration.
# kubectl apply -f external-access.yaml -n mongodb
Follow the load balancer deployment progress.
# watch -n 0.1 kubectl get service -n mongodb
It takes around 5 minutes to finish the deployment.
Output.
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
mongodb-arb-svc ClusterIP None <none> 27017/TCP 155m
mongodb-lb-0 LoadBalancer 10.110.133.138 104.207.153.207 27017:31808/TCP 2m7s
mongodb-lb-1 LoadBalancer 10.98.170.249 45.32.70.189 27017:32151/TCP 2m7s
mongodb-lb-2 LoadBalancer 10.97.161.16 66.42.101.239 27017:30036/TCP 2m7s
mongodb-svc ClusterIP None <none> 27017/TCP 155m
Configure the following DNS records in your domain to set up the A records and the Service (SRV) records for each load balancer.
Type
Name
Content
A
mongodb-0.example.com
LB-0-IP
A
mongodb-1.example.com
LB-1-IP
A
mongodb-2.example.com
LB-2-IP
SRV
_mongodb._tcp.mongodb.example.com
0 50 27017 mongodb-0.example.com
SRV
_mongodb._tcp.mongodb.example.com
0 50 27017 mongodb-1.example.com
SRV
_mongodb._tcp.mongodb.example.com
0 50 27017 mongodb-2.example.com
Verify DNS records.
# nslookup mongodb-0.example.com
# nslookup mongodb-1.example.com
# nslookup mongodb-2.example.com
# nslookup -q=SRV _mongodb._tcp.mongodb.example.com
The following is the connection string for external access.
mongodb+srv://demo-admin:[email protected]/admin
You must pass the TLS certificate fetched in the previous section along with the connection string.
Verify the connection using the MongoDB Shell tool on your local system.
mongosh --tls --tlsCAFile /path/to/certificate/file --tlsCertificateKeyFile /path/to/private_key/file "mongodb+srv://demo-admin:[email protected]/admin"
Output.
Current Mongosh Log ID: 62a71d81196f9e1982eefd7a
Connecting to: mongodb+srv://<credentials>@mongodb.example.com/admin?appName=mongosh+1.3.1
Using MongoDB: 5.0.5
Using Mongosh: 1.3.1
For mongosh info see: https://docs.mongodb.com/mongodb-shell/
mongodb [primary] admin>
You deployed a MongoDB replica set on a Vultr Kubernetes Engine (VKE) cluster, which stores data on the Vultr Block Storage service using the MongoDB Kubernetes Operator. You secured the replica set using a TLS certificate created using the cert-manager add-on. You used the Vultr Load Balancer service for setting up external access to the MongoDB replica set.