Subdots.

As we have shown in a previous blog post - dotmesh enables you to commit stateful container data and push it to a dotmesh remote. The remote could be the dothub or your own hosted dotmesh remote.

In recent years, the software industry has gravitated towards the microservices approach to writing software. As part of this (and for good reasons), we are encouraged to deploy a datastore for each of the different services.

This leads to the following situation:

Multiple dots.

One approach would be to create a seperate dot for each database - for example. you could a different dot for each of:

• MySQL
• Mongo
• RabbitMQ

Then - you could use dm commit 3 times. However - there are a couple of downsides to this approach:

1. Each dm operation will require 3 seperate commands - this is made worse as we add more stateful services to our stack
2. There is the danger of inconsistency for the snapshot as we have a small amount of time between each dm commit command. The state of one datastore might be 10 seconds behind the state of another.

Subdots.

Dotmesh introduces a subdot feature that can help with this very situation. In essence, a subdot offers a filesystem to a container just like a normal dot. You can then combine multiple subdots into an overall data dot and use dm commit and dm push on the whole collection of volumes in a single command.

The keyword here is atomicity - using the subdot abstraction, you can think of your datadot as capturing: the state of your entire stack

And as each subdot as capturing: the state of a single datastore. We can see this by the following illustration:

As you can see, the mongo and mysql each have their own subdot inside an encompassing datadot. We can add commits to the overall dot as normal using dm commit -m "..."

We can also branch the dot and that means we have a branch for each of our subdots:

Basically, anything you can do with a normal datadot, you can do with a dot that contains subdots too!

Sockshop.

Sock Shop is an excellent project by Weave Works and Container Solutions that provides us with a modern stack that contains various microservices.

The architecture diagram can be seen here:

In this stack, we have the following stateful services:

• order-db (Mongo)
• user-db (Mongo)
• catalogue-db (MySQL)
• cart-db (Mongo)
• shipping queue (RabbitMQ)

So, we have a stack with Mongo x 3, MySQL and RabbitMQ. The following guide will show you how to use dotmesh and subdots to capture the state of this entire stack with one command!

Setup dotmesh.

First - you will need a dotmesh cluster running locally. To do this - you can follow our install on docker documentation.

Clone repo.

Next - we clone the repo setup for this exercise:

git clone https://github.com/dotmesh-io/blogpost-examples
cd blogpost-examples/subdots

Study docker-compose.yaml.

You will notice we have modified the docker-compose.yaml slightly - the original lives here. All we have done is add dotmesh subdots for each of the stateful services - you can see the volume definitions at the top of the file:

volumes:
  sockshop.catalogue:
    driver: dm
  sockshop.carts:
    driver: dm
  sockshop.orders:
    driver: dm
  sockshop.user:
    driver: dm
  sockshop.shippingqueue:
    driver: dm

And then in each of the stateful services - we have added a dm volume like so:

catalogue-db:
  image: weaveworksdemos/catalogue-db:0.3.0
  hostname: catalogue-db
  restart: always
  environment:
    - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
    - MYSQL_ALLOW_EMPTY_PASSWORD=true
    - MYSQL_DATABASE=socksdb
  volumes:
   - "sockshop.catalogue:/var/lib/mysql"

Start stack.

Let’s boot the stack:

docker-compose up -d

This will take a short while the first time as it downloads the images from the registry. Once the stack has booted - you should see the following output from dm list:

dm list
Current remote: local (use 'dm remote -v' to list and 'dm remote switch' to switch)

  DOT               BRANCH  SERVER            CONTAINERS                                      SIZE        COMMITS  DIRTY
* subdots_sockshop  master  be3e4bdd17d569ab  /subdots_catalogue-db_1,/subdots_user-db_1,/subdots_rabbitmq_1,/subdots_orders-db_1,/subdots_carts-db_1  810.63 MiB  0        810.63 MiB

This shows that our stack has consumed 5 subdots all under the main data dot called subdots_sockshop

Commit.

Now we can commit the state of our whole stack. First - let’s switch to our subdots_sockshop data dot:

dm switch subdots_sockshop

Now - let’s commit an “empty state”:

dm commit -m "empty state"

dm list should now show zero bytes of dirty data:

$ dm list
Current remote: local (use 'dm remote -v' to list and 'dm remote switch' to switch)

  DOT               BRANCH  SERVER            CONTAINERS                                      SIZE        COMMITS  DIRTY
* subdots_sockshop  master  be3e4bdd17d569ab  /subdots_catalogue-db_1,/subdots_user-db_1,/subdots_rabbitmq_1,/subdots_orders-db_1,/subdots_carts-db_1  810.67 MiB  1        -

Create some state.

The sockshop application is now running locally on your machine - you should be able to see it running by visiting localhost in your browser. Register a new user - add some items to your cart and otherwise create some data within the application.

Doing a dm list should now show you some dirty data because we just wrote to disk after having comitted:

$ dm list
Current remote: local (use 'dm remote -v' to list and 'dm remote switch' to switch)

  DOT               BRANCH  SERVER            CONTAINERS                                      SIZE        COMMITS  DIRTY
* subdots_sockshop  master  be3e4bdd17d569ab  /subdots_catalogue-db_1,/subdots_user-db_1,/subdots_rabbitmq_1,/subdots_orders-db_1,/subdots_carts-db_1  810.99 MiB  1        329.00 kiB

From this - we can see that 329.00 kiB of new data has been written to our dot. Let’s capture this new data with a commit:

dm commit -m "some data"

Push.

Now we can push the state of these 5 databases all within a single dm push command.

First, if you haven’t already - sign up for an account on the dothub and add it as a remote.

We can now use the following single command to push the state of 5 databases remotely so anyone else (human or CI) can pull that data into it’s own stack:

dm push hub subdots_sockshop
Pushing admin/subdots_sockshop to hub:binocarlos/subdots_sockshop
Calculating...
finished 811.84 MB / 811.84 MB [======================] 100.00% 5.17 MiB/s (1/1)
Done!