Improving the local development setup for running MongoDB Replica Set with Docker

NOTE: All associated source code used in this blog can be found at the companion Github repository at this link.

In my previous post , we created a docker based development setup for running MongoDB Replica Set.

A flaw in that setup, was that although clients could connect to stand-alone nodes of the replica set, they could not connect to the replica set as a whole, as reported by this issue on the github repository.

In this blog post, we will see what can be done, to overcome this problem.

Root Cause

As per the MongoDB specifications:

the hostnames used in a replica set config are reachable from the client

In our previous blog, we initialized the following replica set configuration:

MONGODB1=mongo1
MONGODB2=mongo2
MONGODB3=mongo3
:
var cfg = {
    "_id": "rs0",
    ...
    "members": [
        {
            ...
            "host": "${MONGODB1}:27017",
            ...
        },
        {
            ...
            "host": "${MONGODB2}:27017",
            ...
        },
        {
            ...
            "host": "${MONGODB3}:27017",
            ...
        }
    ]
};

In essence, this would result in clients being able to reach the following hosts: mongo1:27017, mongo2:27017, mongo3:27017.

We can update /etc/hosts file to point 127.0.0.1 to mongo1, mongo2, mongo3 hostnames. But the portnumber 27017 would still cause the problem - we can't have 3 nodes (services) running on the same port.

Therefore, solution to the problem would be the following:

Update hosts file in client machines, to map hostnames mongo1, mongo2, mongo3 to 127.0.0.1
Run mongod service on the 3 nodes of the replica set. in different port numbers

Implementing the solution

Implementing the 1st step is trivial. One would need to update /etc/hosts file in Linux, or C:\Windows\System32\drivers\etc\hosts in Windows:

127.0.0.1 localhost mongo1 mongo2 mongo3

Let's go through the steps to implement the second step.

Changes to the docker compose file

...
mongo1:
  hostname: 'mongo1'
  container_name: 'mongo1'
  ...
  command: ["-f", "/etc/mongod.conf", "--port", "30001", "--keyFile", "/auth/file.key", "--replSet", "${MONGO_REPLICA_SET_NAME}", "--bind_ip_all"]
  expose: 
    - 30001
  ports: 
    - 30001:30001 
  ...

mongo2:
  hostname: 'mongo2'
  container_name: 'mongo2'
  ...
  command: ["-f", "/etc/mongod.conf", "--port", "30002", "--keyFile", "/auth/file.key", "--replSet", "${MONGO_REPLICA_SET_NAME}", "--bind_ip_all"]
  expose: 
    - 30002
  ports: 
    - 30002:30002 
  ...

mongo3:
  hostname: 'mongo3'
  container_name: 'mongo3'
  ...
  command: ["-f", "/etc/mongod.conf", "--port", "30003", "--keyFile", "/auth/file.key", "--replSet", "${MONGO_REPLICA_SET_NAME}", "--bind_ip_all"]
  expose: 
    - 30003
  ports: 
    - 30003:30003 
  ...

This will bring up mongod services in the 3 nodes in 3 different ports.

Next, we modify the replica set configuration, by changing mongosetup.sh file:

mongo --host ${MONGODB1}:30001 -u ${MONGO_INITDB_ROOT_USERNAME} -p ${MONGO_INITDB_ROOT_PASSWORD} <<EOF
var cfg = {
    "_id": "rs0",
    ...
    "members": [
        {
            ...
            "host": "${MONGODB1}:30001",
            ...
        },
        {
            ...
            "host": "${MONGODB2}:30002",
            ...
        },
        {
            ...
            "host": "${MONGODB3}:30003",
            ...
        }
    ]
};

These are essentially all the necessary changes needed to be done, to solve the problem.

Final Result

Once we make these changes, we could connect to the replica set using mongo client:

$ mongo "mongodb://localhost:30001,localhost:30002,localhost:30003/<MONGO_INITDB_DATABASE>?replicaSet=rs0" -u <MONGO_INITDB_USERNAME> --authenticationDatabase admin

$ mongo "mongodb://mongo1:30001,mongo2:30002,mongo3:30003/<MONGO_INITDB_DATABASE>?replicaSet=rs0" -u <MONGO_INITDB_USERNAME> --authenticationDatabase admin

We can also view result of db.isMaster() command to debug details like hosts of the replicaset a client must connect to, if the node is primary, etc.

Santanu Paul's Digital Garden