JDMA library

To enable the JDMA to be used programatically in automated workflows, a Python library is provided to expose the functionality of the JDMA as an API. The functions of the API are outlined below.

User functions

jdma_client.jdma_lib.create_user(name, email=None, workspace='default')

Create a user with the username: name

Parameters:

  • name (string) – (required) name of the user to create

  • email (string) – (optional) email address of user for notifications

  • workspace (string) – (optional) default workspace of the user

Returns:

A HTTP Response object. The two most important elements of this object are:

  • status_code (integer): the HTTP status code:

    • 200 OK: Request was successful

    • 403 FORBIDDEN: User already initialized

    • 404 NOT FOUND: Name not supplied in POST request

    • 404 NOT FOUND: Email not supplied in POST request

  • json() (Dictionary): information about the user, the possible keys are:

    • name (string): the name of the user

    • email (string): the email address of the user

    • error (string): information about the error, if one occurred

Return type:

requests.Response

jdma_client.jdma_lib.update_user(name, email=None, notify=None)

Update a user’s information with the username: name

Parameters:

  • name (string) – (required) name of the user to modify

  • email (string) – (optional) email address of user for notifications

  • notify (bool) – (optional) whether the user should be notified, via email, of their requests completing.

Returns:

A HTTP Response object. The two most important elements of this object are:

  • status_code (integer): the HTTP status code:

    • 200 OK: Request was successful

    • 403 FORBIDDEN: User not initialised yet

    • 404 NOT FOUND: name not supplied in POST request

    • 404 NOT FOUND: email not supplied

  • json() (Dictionary): information about the user, the possible keys are:

    • name (string): the name of the user

    • email (string): the email address of the user

    • notify (bool): whether to notify the user

    • error (string): information about the error, if one occurred

Return type:

requests.Response

jdma_client.jdma_lib.info_user(name)

Get a user’s information with the username: name

Parameters:

  • name (string) – (required) name of the user

  • email (string) – (optional) email address of user for notifications

  • notify (bool) – (optional) whether the user should be notified, via email, of their requests completing.

Returns:

A HTTP Response object. The two most important elements of this object are:

  • status_code (integer): the HTTP status code:

    • 200 OK: Request was successful

    • 403 FORBIDDEN: User not initialized yet

    • 404 NOT FOUND: Name not supplied in GET request

    • 404 NOT FOUND: User not found

  • json() (Dictionary): information about the user, the possible keys are:

    • name (string): the name of the user

    • email (string): the email address of the user

    • notify (bool): whether to notify the user

    • error (string): information about the error, if one occurred

Return type:

requests.Response

Query functions

jdma_client.jdma_lib.get_storage()

Get a list of storage backends.

Returns:

A HTTP Response object. The two most important elements of this object are:

  • status_code (integer): the HTTP status code:

    • 200 OK: Request was successful

    • 404 NOT FOUND: URL not found

  • json() (Dictionary): a Dictionary of backends, the format is:

    • key (integer) the numeric id of the storage backend

    • value (string) the name of the backend

Return type:

requests.Response

jdma_client.jdma_lib.get_request(name, req_id=None, workspace=None, ffilter=None)

Get a list of a user’s requests or the details of a single request.

Parameters:

  • name (string) – (required) name of the user.

  • req_id (integer) – (optional) request id to list. If none then get all of the user’s requests.

  • ffilter (string) – (optional) filter the results on user name or workspace

Returns:

A HTTP Response object. The two most important elements of this object are:

  • status_code (integer): the HTTP status code:

    • 200 OK: Request was successful

    • 404 NOT FOUND: name not supplied in GET request

    • 404 NOT FOUND: request id was not found (for the user)

  • json() (Dictionary): information about the request (or a list of Dictionaries), the possible keys are:

    • request_id (integer): the unique request id

    • user (string): the name of the user that the request belongs to

    • request_type (string): one of GET|PUT|MIGRATE|DELETE

    • migration_id (integer): the batch id that the request refers to

    • label (string): the label of the batch

    • workspace (string): the workspace that the batch belongs to

    • storage (string): the storage system the batch resides on

    • stage (string): the stage of the request, see documentation

    • failure_reason (string): why the request failed if stage==FAILED

    • date (date): the time and date the request was made

    • filelist (filelist): the list of files in the request for GET|PUT|MIGRATE

    • error (string): information about the error, if one occurred

Return type:

requests.Response

jdma_client.jdma_lib.get_batch(name, batch_id=None, workspace=None, label=None, ffilter=None)

Get a list of a user’s batches or the details of a single batch.

Parameters:

  • name (string) – (required) name of the user.

  • batch_id (integer) – (optional) batch id to list. If none then get all of the users’ batches.

  • workspace (string) – (optional) workspace to list batches for. If none then list batch(es) for all of the users’ workspaces.

  • ffilter (string) – (optional) filter the results on user name or workspace

Returns:

A HTTP Response object. The two most important elements of this object are:

  • status_code (integer): the HTTP status code:

    • 200 OK: Request was successful

    • 404 NOT FOUND: batch id was not found (for the user)

    • 404 NOT FOUND: workspace was not found (for the user)

  • json() (Dictionary): information about the batch (or a list of Dictionaries), the possible keys are:

    • migration_id (integer) batch id.

    • user (string) the name of the user that the batch belongs to.

    • workspace (string) the workspace the batch belongs to

    • label (string) the label for the batch

    • stage (string) the stage of the batch

    • storage (string) the external storage the batch is on

    • external_id (string) the unique id of the batch on the external storage

    • registered_date (string) the date the batch was uploaded

Return type:

requests.Response

jdma_client.jdma_lib.get_files(name, batch_id=None, workspace=None, limit=0, digest=0, ffilter=None)

Get a list of files that belong to a batch.

Parameters:

  • name (string) – (required) name of the user to get files for.

  • batch_id (integer) – (optional) batch id to list files for. If none then get all of the users’ files.

  • workspace (string) – (optional) workspace to list files for. If none then list files for all of the users’ workspaces.

  • limit (integer) – (optional) limit the number of files returned.

  • digest (integer) – (optional) output the digest (checksum) for each file.

Returns:

A HTTP Response object. The two most important elements of this object are:

  • status_code (integer): the HTTP status code:

    • 200 OK: Request was successful

    • 404 NOT FOUND: name not supplied

    • 404 NOT FOUND: batch id was not found (for the user)

  • json() (List): a list of files, each one containing a Dictionary, the format is:

    • migration_id (integer) the unique id of the batch

    • user (string) the user the batch belongs to

    • workspace (string) the workspace the batch resides in

    • label (string) the batch label

    • storage (string) the external storage the batch resides on

    • archives (list of dictionaries) a list of archives that makes up the batch, each entry in the list is a dictionary, the format of which is:

      • archive_id (string) the unique ID of the archive

      • pk (integer) the numeric ID of the archive

      • size (integer) the total size of the archive in bytes

      • limit (integer) the limit of the returned number of files

      • digest (string) the digest (checksum) of the total archive (if packed)

      • digest_format (string) the algorithm of the digest

      • files (list of dictionaries) a list of files that make up the archive. Each entry is a dictionary, the format of which is:

        • pk (integer) the numeric ID of the file

        • path (string) the full, original path of the file

        • size (integer) the size of the file, in bytes

        • digest (string) the digest (checksum) of the file

        • digest_format (string) the algorithm of the digest

Return type:

requests.Response

jdma_client.jdma_lib.get_archives(name, batch_id=None, workspace=None, limit=0, digest=0, ffilter=None)

Get a list of archives that are in a batch.

Parameters:

  • name (string) – (required) name of the user to get archives for.

  • batch_id (integer) – (optional) batch id to list archives for. If none then get all of the users’ archives.

  • workspace (string) – (optional) workspace to list archives for. If none then list archives for all of the users’ workspaces.

  • limit (integer) – (optional) limit the number of archives returned.

  • digest (integer) – (optional) output the digest (checksum) for each archive.

Returns:

A HTTP Response object. The two most important elementsof this object are:

  • status_code (integer): the HTTP status code:

    • 200 OK: Request was successful

    • 404 NOT FOUND: name was not supplied

    • 404 NOT FOUND: batch id was not found (for the user)

  • json() (List): a list of archives, each one containing a Dictionary, the format is:

    • migration_id (integer) the unique id of the batch

    • user (string) the user the batch belongs to

    • workspace (string) the workspace the batch resides in

    • label (string) the batch label

    • storage (string) the external storage the batch resides on

    • archives (list of dictionaries) a list of archives that makes up the batch, each entry in the list is a dictionary, the format of which is:

      • archive_id (string) the unique ID of the archive

      • pk (integer) the numeric ID of the archive

      • size (integer) the total size of the archive in bytes

      • limit (integer) the limit of the returned number of files

      • digest (string) the digest (checksum) of the total archive (if packed)

      • digest_format (string) the algorithm of the digest

Return type:

requests.Response

File transfer functions

jdma_client.jdma_lib.upload_files(name, workspace=None, filelist=[], label=None, request_type=None, storage=None, credentials=None)

Put a list of files to a storage backend.

Parameters:

  • name (string) – (required) name of the user to get archives for.

  • workspace (string) – (optional) workspace to put files to.

  • filelist (list[string]) – (optional) list of files to put to storage. Absolute paths must be used.

  • label (string) – (optional) user label to give to the batch. If omitted a default will be used.

  • request_type (string) – (optional) request type for putting files to storage. Can be either PUT, which is non-destructive, or MIGRATE which will delete the source files after a successful upload.

  • storage (string) – (optional) the storage backend to put the files to. e.g. objectstore or elastictape.

  • credentials (Dictionary[string]) – (optional) value:key pairs of credentials required by backend and groupworkspace.

Returns:

A HTTP Response object. The two most important elements of this object are:

  • status_code (integer): the HTTP status code:

    • 200 OK: Request was successful

    • 403 FORBIDDEN: User not initialised yet

    • 403 FORBIDDEN: User does not have write permissions or the workspace does not exist for external storage

    • 403 FORBIDDEN: User does not have write permission for file/directory

    • 403 FORBIDDEN: User has insufficient quota remaining for workspace for external storage

    • 403 FORBIDDEN: Filelist or directory is already in a batch

    • 404 NOT FOUND: Name not supplied

    • 404 NOT FOUND: No request type supplied

    • 404 NOT FOUND: Invalid request method

    • 404 NOT FOUND: No workspace supplied

    • 404 NOT FOUND: Workspace has no associated groupworkspace quota set

    • 404 NOT FOUND: No directory path or filelist supplied

    • 404 NOT FOUND: Filelist is empty

    • 404 NOT FOUND: External storage not specified in PUT / MIGRATE request

    • 404 NOT FOUND: Path does not exist

    • 404 NOT FOUND: External storage has not been attached to the groupworkspace

  • json() (List): a Dictionary containing information about the uploaded batch, the format is:

    • name (string) user name for the request (input data)

    • batch_id (integer) the assigned batch id

    • workspace (string) workspace (input data)

    • filelist (string) filelist (input data)

    • label (string) label (input data)

    • request_id (integer) assigned id of the request

    • request_type request_type = PUT | MIGRATE (input data)

    • storage (string) storage (input data)

    • filelist (List) the list of files in the request (input data but may be modified by request)

    • request_id (integer) assigned id of the request

    • stage (string) the stage that the request is at

    • registered_date (string) the date the request was made

    • error (string) the error description if an error occurred

Return type:

requests.Response

jdma_client.jdma_lib.delete_batch(name, batch_id=None, storage=None, credentials=None)

Delete a single batch from a storage backend.

Parameters:

  • name (string) – (required) name of the user to get archives for.

  • batch_id (integer) – (optional) unique id of the batch

  • credentials (Dictionary[string]) – (optional) value:key pairs of credentials required by backend and groupworkspace.

Returns:

A HTTP Response object. The two most important elements of this object are:

  • status_code (integer): the HTTP status code:

    • 200 OK: Request was successful

    • 403 FORBIDDEN User does not have permission to delete batch

    • 403 FORBIDDEN Duplicate DELETE request made

    • 404 NOT FOUND No batch id supplied

    • 404 NOT FOUND Batch not found

    • 404 NOT FOUND: batch id was not found (for the user)

  • json() (Dictionary): a Dictionary containing information about the deleted batch, the format is:

    • name (string) user name for the request (input data)

    • batch_id (integer) batch id (input data)

    • storage (string) storage the batch is stored on (input data)

    • request_id (integer) assigned id of the request

    • request_type (string) = DELETE

    • workspace (string) workspace that the batch is stored in

    • stage (string) the stage that the request is at

    • registered_date (string) the date the request was made

    • label (string) label of the batch

    • error (string) the error description if an error occurred

Return type:

requests.Response

jdma_client.jdma_lib.download_files(name, batch_id=None, filelist=[], target_dir=None, credentials=None)

Download files from a storage backend.

Parameters:

  • name (string) – (required) name of the user to get archives for.

  • batch_id (integer) – (optional) unique id of the batch

  • filelist (list[string]) – (optional) list of files to put to storage. Absolute paths must be used.

  • target_dir (string) – (optional) path to download the files to.

  • credentials (Dictionary[string]) – (optional) value:key pairs of credentials required by backend and groupworkspace.

Returns:

A HTTP Response object. The two most important elements of this object are:

  • status_code (integer): the HTTP status code:

    • 200 OK: Request was successful

    • 403 FORBIDDEN: User not initialised yet

    • 403 FORBIDDEN: User does not have permission to request batch

    • 403 FORBIDDEN: Batch stage is: (~ON_STORAGE). Cannot retrieve (GET) until stage is ON_STORAGE

    • 403 FORBIDDEN: Duplicate GET request made

    • 403 FORBIDDEN: User does not have write permission to the directory

    • 403 FORBIDDEN: Insufficient diskspace for the retrieval (GET)

    • 404 NOT FOUND: No request type supplied

    • 404 NOT FOUND: Invalid request method

    • 404 NOT FOUND: No batch id supplied

    • 404 NOT FOUND: Batch not found

    • 404 NOT FOUND: Target path not supplied

    • 404 NOT FOUND: Parent of target path does not exist

    • 404 NOT FOUND: Requested file list contains zero files that belong to batch

  • json() (List): a Dictionary containing information about the downloaded batch, the format is:

    • name (string) user name for the request (input data)

    • batch_id (integer) batch id (input data)

    • storage (string) storage the batch is stored on (input data)

    • filelist (List) the list of files downloaded (input data but may be modified by the request)

    • target_path (string) the directory the files have been downloaded to

    • request_id (integer) assigned id of the request

    • request_type (string) = GET

    • workspace (string) workspace that the batch is stored in

    • stage (string) the stage that the request is at

    • registered_date (string) the date the request was made

    • label (string) label of the batch

    • error (string) the error description if an error occurred

Return type:

requests.Response

jdma_client.jdma_lib.modify_batch(name, batch_id=None, label=None)

Modify the details of a batch. Currently limited to changing the label

Parameters:

  • name (string) – (required) name of the user to get archives for.

  • batch_id (integer) – (required) unique id of the batch

  • label (string) – (optional) new batch label

Returns:

A HTTP Response object. The two most important elements of this object are:

  • status_code (integer): the HTTP status code:

    • 200 OK: Request was successful

    • 403 FORBIDDEN: Cannot edit this batch as they do not own it

    • 404 NOT FOUND: No name supplied

    • 404 NOT FOUND: No batch id supplied

    • 404 NOT FOUND: No label supplied

    • 404 NOT FOUND: Batch id not found

  • json() (Dictionary): a Dictionary containing information about the modified batch, the format is:

    • name (string) user name for the request (input data)

    • batch_id (integer) batch id (input data)

    • label (string) new label of the batch (input data)

    • error (string) the error description if an error occurred

Return type:

requests.Response