Volumes are useful for storing large data objects so the docker image for the channel can be kept small. When using volumes, the channel’s volume data is transferred to the platform separate from the docker image. The volume needs to be created and the data staged on the volume before a channel is deployed.

This document will share how to use volumes and the SDK calls necessary to stage data before channel deployment.

The /data Directory

When you run your channel locally, we recommend you keep your data in an ana/data directory and reference this data from your package in the packages/<package-name>/config/package.yml file. For example, in the Rendered.ai Example channel the package.yml file looks like this:

# This file contains package-specific configuration information
# The name of an object is the name of the collection in the blender file

volumes:
  example: volumes/example

objects:
  YoYo:
    filename: example:LowPoly.blend
CODE

When we load these objects into a scene, the .blend file is found by joining the path given in the --data ana parameter and the path defined in the package.yml file with information about which volume it belongs to and where in that volume it is located.

Volume Management

Creating the Volume

Volumes are used to store this data in the cloud rather than building it directly into the docker container. To create a volume, use the create_managed_volume() call from the anatools SDK:

create_managed_volume(name, organizationId=None)
CODE

This will create a volume in the specified organization and return a volume id. If no organizationId is specified it will create a volume in your current organization. This volume can then be used to store data that the channel can access later during ana execution.

Name the volume the same as what it is called in the local data/volumes directory.

Retrieving Volume Information

To verify that the volume is created, we can call get_managed_volumes() to retrieve volume information.

get_managed_volumes(organizationId=None)
CODE

This will retrieve information about the volume, when it was created.

Editing a Volume

You can change the name of a volume using the edit_managed_volume() call from the anatools SDK:

edit_managed_volumes(volumeId, name)
CODE

This command will return a boolean to determine if the call was successful. You can verify the name change by calling get_managed_volumes() and inspecting the name field for the volumeId.

Deleting a Volume

To delete a volume, call delete_managed_volume() call from the anatools SDK:

delete_managed_volumes(volumeId)
CODE

Again, this command will return a boolean to determine if the call was successful. You can verify the volume status by calling get_managed_volumes() and inspecting the name field for the volumeId.

Data Management

Retrieving Volume Data Information

To retrieve information about what is currently in the volume, use the get_volume_data() SDK call:

get_volume_data(volumeId, files=[])
CODE

The first parameter here, the volumeId, is created and returned during the create_managed_volume() call but you can also retrieve the volumeId using the get_managed_volumes() call as mentioned above. The files parameter can either be kept as an empty list to retrieve information about all files in a volume or specify specific files in a volume.

Uploading Data to the Volume

To push data to the volume, use the upload_volume_data() call from the anatools SDK:

upload_volume_data(volumeId, files=[], localDir=None)
CODE

Like above, the volumeId is required to specify which volume to upload to. The second parameter, files, is a list of files to upload. These files will be in reference from the localDir parameter, so if a file is located at '/home/user/data/volumes/example/file1.blend' you could specify files as ['file1.blend'] and directory as '/home/user/data/volumes/example'. If the file list is left empty, the SDK will upload all files in the specified directory. If the localDir parameter is left empty, the current directory will be used as the parent directory, this is determined by os.getcwd().

This function can take a long time to complete.

Downloading Data from a Volume

To download data from a volume, use the download_volume_data() call from the anatools SDK:

download_volume_data(volumeId, files=[], localDir=None)
CODE

Just like the upload command, this SDK call uses the volumeId to specify which volume to download from. The second parameter, files, specifies which files to download, again if the files list is empty it will download all files from the volume. The localDir parameter specifies the file download directory.

This function can take a long time to complete.

Deleting Data from a Volume

To delete data from a volume, use the delete_volume_data() call from the anatools SDK:

delete_volume_data(volumeId, files=[])
CODE

If the files list is left empty, this will not delete all files in the volume. You will need to specify the files that you want deleted from the volume.