JDMA library

To enable the JDMA to be used programatically in automated workflows, a Python library is provided to expose the functionality of the JDMA as an API. The functions of the API are outlined below.

User functions

jdma_client.jdma_lib.create_user(name, email=None, workspace='default')

Create a user with the username: name

Parameters:
  • name (string) – (required) name of the user to create
  • email (string) – (optional) email address of user for notifications
  • workspace (string) – (optional) default workspace of the user
Returns:

A HTTP Response object. The two most important elements of this object are:

  • status_code (integer): the HTTP status code:

    • 200 OK: Request was successful
    • 403 FORBIDDEN: User already initialized
    • 404 NOT FOUND: Name not supplied in POST request
    • 404 NOT FOUND: Email not supplied in POST request

  • json() (Dictionary): information about the user, the possible keys are:

    • name (string): the name of the user
    • email (string): the email address of the user
    • error (string): information about the error, if one occurred
Return type:

requests.Response
jdma_client.jdma_lib.update_user(name, email=None, notify=None)

Update a user’s information with the username: name

Parameters:
  • name (string) – (required) name of the user to modify
  • email (string) – (optional) email address of user for notifications
  • notify (bool) – (optional) whether the user should be notified, via email, of their requests completing.
Returns:

A HTTP Response object. The two most important elements of this object are:

  • status_code (integer): the HTTP status code:

    • 200 OK: Request was successful
    • 403 FORBIDDEN: User not initialised yet
    • 404 NOT FOUND: name not supplied in POST request
    • 404 NOT FOUND: email not supplied

  • json() (Dictionary): information about the user, the possible keys are:

    • name (string): the name of the user
    • email (string): the email address of the user
    • notify (bool): whether to notify the user
    • error (string): information about the error, if one occurred
Return type:

requests.Response
jdma_client.jdma_lib.info_user(name)

Get a user’s information with the username: name

Parameters:
  • name (string) – (required) name of the user
  • email (string) – (optional) email address of user for notifications
  • notify (bool) – (optional) whether the user should be notified, via email, of their requests completing.
Returns:

A HTTP Response object. The two most important elements of this object are:

  • status_code (integer): the HTTP status code:

    • 200 OK: Request was successful
    • 403 FORBIDDEN: User not initialized yet
    • 404 NOT FOUND: Name not supplied in GET request
    • 404 NOT FOUND: User not found

  • json() (Dictionary): information about the user, the possible keys are:

    • name (string): the name of the user
    • email (string): the email address of the user
    • notify (bool): whether to notify the user
    • error (string): information about the error, if one occurred
Return type:

requests.Response

Query functions

jdma_client.jdma_lib.get_storage()

Get a list of storage backends.

Returns:A HTTP Response object. The two most important elements of this object are:
  • status_code (integer): the HTTP status code:
    • 200 OK: Request was successful
    • 404 NOT FOUND: URL not found
  • json() (Dictionary): a Dictionary of backends, the format is:
    • key (integer) the numeric id of the storage backend
    • value (string) the name of the backend
Return type:requests.Response
jdma_client.jdma_lib.get_request(name, req_id=None, workspace=None, ffilter=None)

Get a list of a user’s requests or the details of a single request.

Parameters:
  • name (string) – (required) name of the user.
  • req_id (integer) – (optional) request id to list. If none then get all of the user’s requests.
  • ffilter (string) – (optional) filter the results on user name or workspace
Returns:

A HTTP Response object. The two most important elements of this object are:

  • status_code (integer): the HTTP status code:

    • 200 OK: Request was successful
    • 404 NOT FOUND: name not supplied in GET request
    • 404 NOT FOUND: request id was not found (for the user)

  • json() (Dictionary): information about the request (or a list of Dictionaries), the possible keys are:

    • request_id (integer): the unique request id
    • user (string): the name of the user that the request belongs to
    • request_type (string): one of GET|PUT|MIGRATE|DELETE
    • migration_id (integer): the batch id that the request refers to
    • label (string): the label of the batch
    • workspace (string): the workspace that the batch belongs to
    • storage (string): the storage system the batch resides on
    • stage (string): the stage of the request, see documentation
    • failure_reason (string): why the request failed if stage==FAILED
    • date (date): the time and date the request was made
    • filelist (filelist): the list of files in the request for GET|PUT|MIGRATE
    • error (string): information about the error, if one occurred
Return type:

requests.Response
jdma_client.jdma_lib.get_batch(name, batch_id=None, workspace=None, label=None, ffilter=None)

Get a list of a user’s batches or the details of a single batch.

Parameters:
  • name (string) – (required) name of the user.
  • batch_id (integer) – (optional) batch id to list. If none then get all of the users’ batches.
  • workspace (string) – (optional) workspace to list batches for. If none then list batch(es) for all of the users’ workspaces.
  • ffilter (string) – (optional) filter the results on user name or workspace
Returns:

A HTTP Response object. The two most important elements of this object are:

  • status_code (integer): the HTTP status code:

    • 200 OK: Request was successful
    • 404 NOT FOUND: batch id was not found (for the user)
    • 404 NOT FOUND: workspace was not found (for the user)

  • json() (Dictionary): information about the batch (or a list of Dictionaries), the possible keys are:

    • migration_id (integer) batch id.
    • user (string) the name of the user that the batch belongs to.
    • workspace (string) the workspace the batch belongs to
    • label (string) the label for the batch
    • stage (string) the stage of the batch
    • storage (string) the external storage the batch is on
    • external_id (string) the unique id of the batch on the external storage
    • registered_date (string) the date the batch was uploaded
Return type:

requests.Response
jdma_client.jdma_lib.get_files(name, batch_id=None, workspace=None, limit=0, digest=0, ffilter=None)

Get a list of files that belong to a batch.

Parameters:
  • name (string) – (required) name of the user to get files for.
  • batch_id (integer) – (optional) batch id to list files for. If none then get all of the users’ files.
  • workspace (string) – (optional) workspace to list files for. If none then list files for all of the users’ workspaces.
  • limit (integer) – (optional) limit the number of files returned.
  • digest (integer) – (optional) output the digest (checksum) for each file.
Returns:

A HTTP Response object. The two most important elements of this object are:

  • status_code (integer): the HTTP status code:

    • 200 OK: Request was successful
    • 404 NOT FOUND: name not supplied
    • 404 NOT FOUND: batch id was not found (for the user)

  • json() (List): a list of files, each one containing a Dictionary, the format is:

    • migration_id (integer) the unique id of the batch

    • user (string) the user the batch belongs to

    • workspace (string) the workspace the batch resides in

    • label (string) the batch label

    • storage (string) the external storage the batch resides on

    • archives (list of dictionaries) a list of archives that makes up the batch, each entry in the list is a dictionary, the format of which is:

      • archive_id (string) the unique ID of the archive

      • pk (integer) the numeric ID of the archive

      • size (integer) the total size of the archive in bytes

      • limit (integer) the limit of the returned number of files

      • digest (string) the SHA256 digest (checksum) of the total archive (if packed)

      • files (list of dictionaries) a list of files that make up the archive. Each entry is a dictionary, the format of which is:

        • pk (integer) the numeric ID of the file
        • path (string) the full, original path of the file
        • size (integer) the size of the file, in bytes
        • digest (string) the SH256 digest (checksum) of the file
Return type:

requests.Response
jdma_client.jdma_lib.get_archives(name, batch_id=None, workspace=None, limit=0, digest=0, ffilter=None)

Get a list of archives that are in a batch.

Parameters:
  • name (string) – (required) name of the user to get archives for.
  • batch_id (integer) – (optional) batch id to list archives for. If none then get all of the users’ archives.
  • workspace (string) – (optional) workspace to list archives for. If none then list archives for all of the users’ workspaces.
  • limit (integer) – (optional) limit the number of archives returned.
  • digest (integer) – (optional) output the digest (checksum) for each archive.
Returns:

A HTTP Response object. The two most important elementsof this object are:

  • status_code (integer): the HTTP status code:

    • 200 OK: Request was successful
    • 404 NOT FOUND: name was not supplied
    • 404 NOT FOUND: batch id was not found (for the user)

  • json() (List): a list of archives, each one containing a Dictionary, the format is:

    • migration_id (integer) the unique id of the batch

    • user (string) the user the batch belongs to

    • workspace (string) the workspace the batch resides in

    • label (string) the batch label

    • storage (string) the external storage the batch resides on

    • archives (list of dictionaries) a list of archives that makes up the batch, each entry in the list is a dictionary, the format of which is:

      • archive_id (string) the unique ID of the archive
      • pk (integer) the numeric ID of the archive
      • size (integer) the total size of the archive in bytes
      • limit (integer) the limit of the returned number of files
      • digest (string) the SHA256 digest (checksum) of the total archive (if packed)
Return type:

requests.Response

File transfer functions

jdma_client.jdma_lib.upload_files(name, workspace=None, filelist=[], label=None, request_type=None, storage=None, credentials=None)

Put a list of files to a storage backend.

Parameters:
  • name (string) – (required) name of the user to get archives for.
  • workspace (string) – (optional) workspace to put files to.
  • filelist (list[string]) – (optional) list of files to put to storage. Absolute paths must be used.
  • label (string) – (optional) user label to give to the batch. If omitted a default will be used.
  • request_type (string) – (optional) request type for putting files to storage. Can be either PUT, which is non-destructive, or MIGRATE which will delete the source files after a successful upload.
  • storage (string) – (optional) the storage backend to put the files to. e.g. objectstore or elastictape.
  • credentials (Dictionary[string]) – (optional) value:key pairs of credentials required by backend and groupworkspace.
Returns:

A HTTP Response object. The two most important elements of this object are:

  • status_code (integer): the HTTP status code:

    • 200 OK: Request was successful
    • 403 FORBIDDEN: User not initialised yet
    • 403 FORBIDDEN: User does not have write permissions or the workspace does not exist for external storage
    • 403 FORBIDDEN: User does not have write permission for file/directory
    • 403 FORBIDDEN: User has insufficient quota remaining for workspace for external storage
    • 403 FORBIDDEN: Filelist or directory is already in a batch
    • 404 NOT FOUND: Name not supplied
    • 404 NOT FOUND: No request type supplied
    • 404 NOT FOUND: Invalid request method
    • 404 NOT FOUND: No workspace supplied
    • 404 NOT FOUND: Workspace has no associated groupworkspace quota set
    • 404 NOT FOUND: No directory path or filelist supplied
    • 404 NOT FOUND: Filelist is empty
    • 404 NOT FOUND: External storage not specified in PUT / MIGRATE request
    • 404 NOT FOUND: Path does not exist
    • 404 NOT FOUND: External storage has not been attached to the groupworkspace

  • json() (List): a Dictionary containing information about the uploaded batch, the format is:

    • name (string) user name for the request (input data)
    • batch_id (integer) the assigned batch id
    • workspace (string) workspace (input data)
    • filelist (string) filelist (input data)
    • label (string) label (input data)
    • request_id (integer) assigned id of the request
    • request_type request_type = PUT | MIGRATE (input data)
    • storage (string) storage (input data)
    • filelist (List) the list of files in the request (input data but may be modified by request)
    • request_id (integer) assigned id of the request
    • stage (string) the stage that the request is at
    • registered_date (string) the date the request was made
    • error (string) the error description if an error occurred
Return type:

requests.Response
jdma_client.jdma_lib.delete_batch(name, batch_id=None, storage=None, credentials=None)

Delete a single batch from a storage backend.

Parameters:
  • name (string) – (required) name of the user to get archives for.
  • batch_id (integer) – (optional) unique id of the batch
  • credentials (Dictionary[string]) – (optional) value:key pairs of credentials required by backend and groupworkspace.
Returns:

A HTTP Response object. The two most important elements of this object are:

  • status_code (integer): the HTTP status code:

    • 200 OK: Request was successful
    • 403 FORBIDDEN User does not have permission to delete batch
    • 403 FORBIDDEN Duplicate DELETE request made
    • 404 NOT FOUND No batch id supplied
    • 404 NOT FOUND Batch not found
    • 404 NOT FOUND: batch id was not found (for the user)

  • json() (Dictionary): a Dictionary containing information about the deleted batch, the format is:

    • name (string) user name for the request (input data)
    • batch_id (integer) batch id (input data)
    • storage (string) storage the batch is stored on (input data)
    • request_id (integer) assigned id of the request
    • request_type (string) = DELETE
    • workspace (string) workspace that the batch is stored in
    • stage (string) the stage that the request is at
    • registered_date (string) the date the request was made
    • label (string) label of the batch
    • error (string) the error description if an error occurred
Return type:

requests.Response
jdma_client.jdma_lib.download_files(name, batch_id=None, filelist=[], target_dir=None, credentials=None)

Download files from a storage backend.

Parameters:
  • name (string) – (required) name of the user to get archives for.
  • batch_id (integer) – (optional) unique id of the batch
  • filelist (list[string]) – (optional) list of files to put to storage. Absolute paths must be used.
  • target_dir (string) – (optional) path to download the files to.
  • credentials (Dictionary[string]) – (optional) value:key pairs of credentials required by backend and groupworkspace.
Returns:

A HTTP Response object. The two most important elements of this object are:

  • status_code (integer): the HTTP status code:

    • 200 OK: Request was successful
    • 403 FORBIDDEN: User not initialised yet
    • 403 FORBIDDEN: User does not have permission to request batch
    • 403 FORBIDDEN: Batch stage is: (~ON_STORAGE). Cannot retrieve (GET) until stage is ON_STORAGE
    • 403 FORBIDDEN: Duplicate GET request made
    • 403 FORBIDDEN: User does not have write permission to the directory
    • 403 FORBIDDEN: Insufficient diskspace for the retrieval (GET)
    • 404 NOT FOUND: No request type supplied
    • 404 NOT FOUND: Invalid request method
    • 404 NOT FOUND: No batch id supplied
    • 404 NOT FOUND: Batch not found
    • 404 NOT FOUND: Target path not supplied
    • 404 NOT FOUND: Parent of target path does not exist
    • 404 NOT FOUND: Requested file list contains zero files that belong to batch

  • json() (List): a Dictionary containing information about the downloaded batch, the format is:

    • name (string) user name for the request (input data)
    • batch_id (integer) batch id (input data)
    • storage (string) storage the batch is stored on (input data)
    • filelist (List) the list of files downloaded (input data but may be modified by the request)
    • target_path (string) the directory the files have been downloaded to
    • request_id (integer) assigned id of the request
    • request_type (string) = GET
    • workspace (string) workspace that the batch is stored in
    • stage (string) the stage that the request is at
    • registered_date (string) the date the request was made
    • label (string) label of the batch
    • error (string) the error description if an error occurred
Return type:

requests.Response
jdma_client.jdma_lib.modify_batch(name, batch_id=None, label=None)

Modify the details of a batch. Currently limited to changing the label

Parameters:
  • name (string) – (required) name of the user to get archives for.
  • batch_id (integer) – (required) unique id of the batch
  • label (string) – (optional) new batch label
Returns:

A HTTP Response object. The two most important elements of this object are:

  • status_code (integer): the HTTP status code:

    • 200 OK: Request was successful
    • 403 FORBIDDEN: Cannot edit this batch as they do not own it
    • 404 NOT FOUND: No name supplied
    • 404 NOT FOUND: No batch id supplied
    • 404 NOT FOUND: No label supplied
    • 404 NOT FOUND: Batch id not found

  • json() (Dictionary): a Dictionary containing information about the modified batch, the format is:

    • name (string) user name for the request (input data)
    • batch_id (integer) batch id (input data)
    • label (string) new label of the batch (input data)
    • error (string) the error description if an error occurred
Return type:

requests.Response