How To Deploy a Fault-Tolerant MongoDB Sharded Cluster on Ubuntu 18/19/20

Note: This guide is specifically written for Ubuntu 18.04, 19.04, 19.10, 20.04 and Debian 9, 10.

Prerequisites

You can set hostname on your servers with below command:

sudo hostnamectl set-hostname server-name.domain

You can set timezone on your servers with below command:

sudo timedatectl set-timezone US/New_York

Update Hosts Files

Login to your first server (cfg1.example.pk) in our case, edit /etc/hosts file:

sudo nano /etc/hosts

# Config Server

192.168.10.1 cfg1.example.pk
192.168.10.2 cfg2.example.pk
192.168.10.3 cfg3.example.pk

# Shard Server (rs0)
192.168.10.4 shrd1.example.pk
192.168.10.5 shrd2.example.pk
192.168.10.6 shrd3.example.pk

# Shard Server (rs1)
192.168.10.7 shrd4.example.pk
192.168.10.8 shrd5.example.pk
192.168.10.9 shrd6.example.pk

# Query Router
192.168.10.10 qrt1.example.pk

Save and close file when you are finished.

Repeat the same on remaining servers before proceeding to next step.

Add MongoDB Source

Type below to add mongo official repository:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 4b7c549a058f8b6b
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.2.list

Make sure you repeat the same steps on remaining servers before proceeding to next step.

Installing MongoDB

Type below command to install mongodb version v4.2.3:

sudo apt update
sudo apt -y install mongodb-org

Repeat the same on remaining servers as well.

Create an Administrative User

Start mongodb service with below command:

sudo systemctl start mongod.service
sudo systemctl enable mongod.service

Access mongo shell with below command:

mongo

On mongo shell, type below to switch to the default admin database:

use admin

db.createUser({user: "administrator", pwd: "password", roles:[{role: "root", db: "admin"}]})

This will return similar to the following output:

Successfully added user: {
"user" : "administrator",
"roles" : [
                {
"role" : "root",
"db" : "admin"
                }
        ]
}

Type below to exit from mongo shell:

quit()

Set Up MongoDB Authentication

On (cfg1.example.pk) in our case, type below command to generate a key file:

openssl rand -base64 756 > ~/mongodb_key.pem

For this guide, we generated key file under the home directory of user.

sudo cp ~/mongodb_key.pem /var/lib/mongodb/

sudo chown -R mongodb:mongodb /var/lib/mongodb/mongodb_key.pem
sudo chmod -R 400 /var/lib/mongodb/mongodb_key.pem

sudo scp -p /var/lib/mongodb/mongodb_key.pem cfg2.example.pk:~/

sudo scp -p /var/lib/mongodb/mongodb_key.pem cfg3.example.pk:~/

sudo scp -p /var/lib/mongodb/mongodb_key.pem shrd1.example.pk:~/
sudo scp -p /var/lib/mongodb/mongodb_key.pem shrd2.example.pk:~/
sudo scp -p /var/lib/mongodb/mongodb_key.pem shrd3.example.pk:~/
sudo scp -p /var/lib/mongodb/mongodb_key.pem shrd4.example.pk:~/
sudo scp -p /var/lib/mongodb/mongodb_key.pem shrd5.example.pk:~/
sudo scp -p /var/lib/mongodb/mongodb_key.pem shrd6.example.pk:~/

sudo scp -p /var/lib/mongodb/mongodb_key.pem qrt1.example.pk:~/

We are performing following steps from (cfg1.example.pk) using ssh on each server:

ssh cfg2.example.pk sudo mv ~/mongodb_key.pem /var/lib/mongodb/
ssh cfg2.example.pk sudo chown -R mongodb:mongodb /var/lib/mongodb/mongodb_key.pem
ssh cfg2.example.pk sudo chmod -R 400 /var/lib/mongodb/mongodb_key.pem

ssh cfg3.example.pk sudo mv ~/mongodb_key.pem /var/lib/mongodb/
ssh cfg3.example.pk sudo chown -R mongodb:mongodb /var/lib/mongodb/mongodb_key.pem
ssh cfg3.example.pk sudo chmod -R 400 /var/lib/mongodb/mongodb_key.pem

ssh shrd1.example.pk sudo mv ~/mongodb_key.pem /var/lib/mongodb/
ssh shrd1.example.pk sudo chown -R mongodb:mongodb /var/lib/mongodb/mongodb_key.pem
ssh shrd1.example.pk sudo chmod -R 400 /var/lib/mongodb/mongodb_key.pem

ssh shrd2.example.pk sudo mv ~/mongodb_key.pem /var/lib/mongodb/
ssh shrd2.example.pk sudo chown -R mongodb:mongodb /var/lib/mongodb/mongodb_key.pem
ssh shrd2.example.pk sudo chmod -R 400 /var/lib/mongodb/mongodb_key.pem

ssh shrd3.example.pk sudo mv ~/mongodb_key.pem /var/lib/mongodb/
ssh shrd3.example.pk sudo chown -R mongodb:mongodb /var/lib/mongodb/mongodb_key.pem
ssh shrd3.example.pk sudo chmod -R 400 /var/lib/mongodb/mongodb_key.pem

ssh shrd4.example.pk sudo mv ~/mongodb_key.pem /var/lib/mongodb/
ssh shrd4.example.pk sudo chown -R mongodb:mongodb /var/lib/mongodb/mongodb_key.pem
ssh shrd4.example.pk sudo chmod -R 400 /var/lib/mongodb/mongodb_key.pem

ssh shrd5.example.pk sudo mv ~/mongodb_key.pem /var/lib/mongodb/
ssh shrd5.example.pk sudo chown -R mongodb:mongodb /var/lib/mongodb/mongodb_key.pem
ssh shrd5.example.pk sudo chmod -R 400 /var/lib/mongodb/mongodb_key.pem

ssh shrd6.example.pk sudo mv ~/mongodb_key.pem /var/lib/mongodb/
ssh shrd6.example.pk sudo chown -R mongodb:mongodb /var/lib/mongodb/mongodb_key.pem
ssh shrd6.example.pk sudo chmod -R 400 /var/lib/mongodb/mongodb_key.pem

ssh qrt1.example.pk sudo mv ~/mongodb_key.pem /var/lib/mongodb/
ssh qrt1.example.pk sudo chown -R mongodb:mongodb /var/lib/mongodb/mongodb_key.pem
ssh qrt1.example.pk sudo chmod -R 400 /var/lib/mongodb/mongodb_key.pem

Create the Config Server Replica Set

Login to your first config server (cfg1.example.pk) in our case, edit /etc/mongod.conf file:

sudo nano /etc/mongod.conf

Add, update the following highlighted values, make sure you replace default port with 27019

port: 27019
  bindIp: cfg1.example.pk

security:
  keyFile: /var/lib/mongodb/mongodb_key.pe2cm

replication:
  replSetName: configReplSet

sharding:
  clusterRole: configsvr

Save and close file when you are finished.

Repeat the same on remaining config servers (cfg2.example.pk, cfg3.example.pk) in our case.

For reference, you can see below mongod.conf from cfg1.example.pk in our case.

This mongod.conf from cfg2.example.pk in our case.

and lastly mongod.conf from cfg3.example.pk in our case.

sudo systemctl start mongod.service

sudo systemctl enable mongod.service

Confirm that the mongodb service is active and running:

sudo systemctl status mongod.service

screenshot of cfg1.example.pk

screenshot of cfg2.example.pk

screenshot of cfg3.example.pk

Initiate the config replica set.

mongo --host cfg1.example.pk --port 27019 -u administrator --authenticationDatabase admin

This will prompt you for password:

MongoDB shell version v4.2.3
Enter password:

From the mongo shell, initiate the config server's replica set like below:

rs.initiate({ _id: "configReplSet", configsvr: true, members: [{ _id : 0, host : "cfg1.example.pk:27019"},{ _id : 1, host : "cfg2.example.pk:27019"},{ _id : 2, host : "cfg3.example.pk:27019"}]})

You will see a message similar to below indicating the operation succeeded:


To make sure that each config server has been added to the replica set, type below on mongo shell:

rs.status()

If the replica set has been configured properly, you’ll see output similar to the following:


Now exit from mongo shell with below command:

quit()

Create the Shard Replica Sets

In this section we will configure shard replica set (rs0) on (shrd1, shrd2, shrd3) servers.

sudo nano /etc/mongod.conf

Add, update the following values, make sure you replace default port with 27018:

net:
  port: 27018
  bindIp: shrd1.example.pk

security:
  keyFile: /var/lib/mongodb/mongodb_key.pem

replication:
  replSetName: rs0

sharding:
  clusterRole: shardsvr

Save and close file when you are finished.

Make sure you repeat the same on (shrd2.example.pk, shrd3.example.pk) as well.


Below screenshot of mongod.conf from shrd2.example.pk

and below screenshot of mongod.conf from shrd3.example.pk

sudo systemctl start mongod

sudo systemctl enable mongod
sudo systemctl status mongod

screenshot of mongod service status from shrd1.example.pk

screenshot of mongod service status from shrd2.example.pk

screenshot of mongod service status from shrd3.example.pk

When you are finished with the above step on (shrd1, shrd2, shrd3), proceed with the below.

In this step, we will configure shard replica set (rs1) on (shrd4, shrd5, shrd6) servers.

Login to (shrd4.example.pk) in our case, edit /etc/mongod.conf file:

sudo nano /etc/mongod.conf

Add, update the following values, make sure you replace default port with 27018:

net:
  port: 27018
  bindIp: shrd4.example.pk

security:
  keyFile: /var/lib/mongodb/mongodb_key.pem

replication:
  replSetName: rs1

sharding:
  clusterRole: shardsvr

Save and close file when you are finished.

Make sure you repeat the same on (shrd5.example.pk, shrd6.example.pk) as well.


Below screenshot of mongod.conf from shrd5.example.pk

and below screenshot of mongod.conf from shrd6.example.pk

sudo systemctl start mongod

sudo systemctl enable mongod
sudo systemctl status mongod

mongod service status screenshot from shrd4.example.pk

mongod service status screenshot from shrd5.example.pk

mongod service status screenshot from shrd6.example.pk

Initiate the shard replica set.

mongo --host shrd1.example.pk --port 27018 -u administrator --authenticationDatabase admin

Type below on mongo shell to initiate shard replica set (rs0):

rs.initiate({ _id : "rs0", members:[{ _id : 0, host : "shrd1.example.pk:27018" },{ _id : 1, host : "shrd2.example.pk:27018" },{ _id : 2, host : "shrd3.example.pk:27018" }]})

This will return { "ok" : 1 } indicating that shard replica set rs0 is initiated successfully.

Now exit from the mongo shell with below command:

quit()

mongo --host shrd4.example.pk --port 27018 -u administrator --authenticationDatabase admin

Type below on mongo shell to initiate shard replica set (rs1):

rs.initiate({ _id : "rs1", members:[{ _id : 0, host : "shrd4.example.pk:27018" },{ _id : 1, host : "shrd5.example.pk:27018" },{ _id : 2, host : "shrd6.example.pk:27018" }]})

This will return { "ok" : 1 } indicating that shard replica set rs1 is initiated successfully.

Now exit from the mongo shell with below command:

quit()

Configure Mongos (Query Router)

Login to your query router server (qrt1.example.pk) in our case and perform the following.

Type below command to deactivate mongod service:

sudo systemctl stop mongod.service
sudo systemctl disable mongod.service

sudo systemctl status mongod.service

As you can see mongod.service has been stopped on query router.

Type below to create /etc/mongos.conf file:

sudo nano /etc/mongos.conf

Add the following configuration directives:

# Where write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongos.log

# network interfaces
net:
  port: 27017
  bindIp: qrt1.example.pk

security:
  keyFile: /var/lib/mongodb/mongodb_key.pem

sharding:
  configDB: configReplSet/cfg1.example.pk:27019,cfg2.example.pk:27019,cfg3.example.pk:27019

Save and close file when you are finished.

You can see final mongos.conf file in below screenshot.

Next, create a systemd service unit file for mongos like below:

sudo nano /lib/systemd/system/mongos.service

Add the following parameters:

[Unit]
Description=Mongo Cluster Router
After=network.target

[Service]
User=mongodb
Group=mongodb

ExecStart=/usr/bin/mongos --config /etc/mongos.conf

LimitFSIZE=infinity
LimitCPU=infinity
LimitAS=infinity

LimitNOFILE=64000
LimitNPROC=64000

TasksMax=infinity
TasksAccounting=false

[Install]
WantedBy=multi-user.target

Save and close file when you are finished.

You can see final mongos.service file in below screenshot.

Next, start mongos.service with below command to activate query router.

sudo systemctl start mongos.service
sudo systemctl enable mongos.service

Confirm that mongos.service is running with below command:

sudo systemctl status mongos.service

You can see below screenshot of mongos.service status

Add Shards to the Cluster

mongo --host qrt1.example.pk --port 27017 -u administrator --authenticationDatabase admin

On mongo shell, type below to add shard replica set (rs0) in the cluster:

sh.addShard( "rs0/shrd1.example.pk:27018,shrd2.example.pk:27018,shrd2.example.pk:27018")

Type below to add shard replica set (rs1) in the cluster:

sh.addShard( "rs1/shrd4.example.pk:27018,shrd5.example.pk:27018,shrd6.example.pk:27018")

You will see the similar output indicating that shard replica sets added successfully.

At this stage, your MongoDB sharded cluster is active and running.

• Databases - The broadest data structure in MongoDB, used to hold groups of related data.
• Collections - Analogous to tables in traditional relational database systems, collections are the data structures that comprise databases
• Documents - The most basic unit of data storage in MongoDB. Documents use JSON format for storing data using key-value pairs that can be queried by applications

Sharding Strategy

• Range-based sharding divides your data based on specific ranges of values in the shard key. 
• Hash-based sharding distributes data by using a hash function on your shard key for a more even distribution of data among the shards.

Enable Sharding for a Database

mongo --host qrt1.example.pk --port 27017 -u administrator --authenticationDatabase admin

From the mongos shell, type below to create a test database called testDB

use testDB

db.testCollection.ensureIndex( { _id : "hashed" } )

Type below to enable sharding for newly created database:

sh.enableSharding("testDB")
sh.shardCollection( "testDB.testCollection", { "_id" : "hashed" } )

This enables sharding across any shards that you added to your cluster in the Add Shards to the Cluster step.



To verify that the sharding was successfully enabled, type below to switch to the config database:

use config

Next, run below method:

db.databases.find()

This will return a list of all databases with useful information similar to like below:


In the above output, you can see only one entry for the testDB database we just created.

Verify Shard Cluster

Connect to the mongos shell on your query router:

mongo --host qrt1.example.pk --port 27017 -u administrator -p --authenticationDatabase admin

Switch to your testDB database:

use testDB

Type the following code in the mongo shell to generate 10000 simple documents and insert them into testCollection:

for (var i = 1; i <= 10000; i++) db.testCollection.insert( { x : i } )

Check your data distribution:

db.testCollection.getShardDistribution()

This will return information similar to the following:

When you’re finished, we recommend you to delete the testDB (because it has no use) with below command:

db.dropDatabase()

Wrapping up