Creating Instant Clones of Production Data for Containerized Test and Development

Share:

Using containers with FlexClones is a powerful combination which increases the efficiency of the test and development process beyond what containers alone can provide. FlexCloning a volume containing production data allows your developers to test with the real thing, without incurring performance penalties or using any additional storage capacity, all in the span of milliseconds.

One of our customers recently asked about the possibility of naturally integrating this functionality with MongoDB running on a Mesos cluster, with container scheduling managed by Marathon. NetApp’s containers team, internally known as BARNACLE, responded by crafting a solution that involved a number of different integrations, many of which are available today and require simple extensions to provide basic awareness of containerized applications.

Creating the Database and Protecting It

The first step is to simply create a containerized MongoDB instance which has been deployed by Marathon and takes advantage of NetApp storage. This can be done manually, or by using Mesos’ extensibility, with the overall goal of simply connecting the container to the NetApp volume, whether using block or file storage.

Containerized MongoDB using NetApp storage and Snap Creator

After the instance has been created, we need to register it with NetApp Snap Creator. This gives us manageability over quiescing the instance and taking snapshots. This can be done manually, or using the automatic registration that was discussed in this blog post.

MongoDB registered with NetApp Snap Creator

Quiesce the Database and FlexClone It

At this point we have a standard MongoDB deployment, which just happens to be using containers instead of a more traditional native rpm or apt package install method. From Snap Creator, we can manage it just like any other MongoDB instance as well, including taking snapshots, reverting snapshots, and creating FlexClones of a volume snapshot.

Using the Snap Creator interface, we now want to create a snapshot of the database and create a FlexClone of that volume in one simple step.

Using Snap Creator to create a FlexClone of the volume

And finally, we mount the newly FlexCloned volume to an available host.

The FlexCloned volume has been created in only a few seconds with the exact same data as the original

Start a New Instance Using the Original Data

The data, in a quiesced state, has now been made available to the hosts in our Mesos cluster, so all that’s left is to start a new MongoDB container which uses the newly created and exported volume.

An updated Marathon definition which takes advantage of the FlexCloned volume

A second MongoDB container running using the FlexCloned volume

Extending Dev and Test Operations

This post highlights how you can use NetApp FlexClone technology, combined with containers and Mesos + Marathon, to quickly instantiate dev and test instances of an application. Using FlexClone technology native to Data ONTAP means there are no capacity or performance penalties, and the cloned data is available instantly. Enabling development teams to work with real data means they have a more accurate view of what the production application works with and find more real issues in a manner that’s even faster than generating a much smaller set of synthetic data.

Docker is beloved by development teams for good reason: it makes things easier for them. Ensuring the infrastructure can support their operations by seamlessly providing environments with the appropriate resources, and near-real-time data, makes development even easier. Extending this paradigm into other phases of the development and deployment lifecycle potentially increases the ability to detect and prevent invalid, or buggy, code from going forward. NetApp is the storage platform of choice for providing persistent storage to your containerized application, with robust protection and unmatched data management features. Reach out to us at opensource@netapp.com, and we will be happy to discuss the best way to leverage NetApp and container technology together.

Related Content