Tutorial

This page is a tutorial on NLDS covering:

Introduction to the NLDS

The Near-Line Data Store (NLDS) is a multi-tiered storage solution that uses Object Storage as a front end cache to a tape library. Interaction with NLDS is via a HTTP API, with a Python library and command-line client provided to users for programmatic or interactive use.

When a user ingests files to the NLDS, the files are first written to the cache, and a backup is made to tape. When the cache is approaching capacity, a set of policies is interrogated to determine which files will be removed from the cache. When a user GETs a file, the NLDS may have to first transfer the file from tape to the cache, if it has been deleted by the policies. This, in effect, creates a multi-tiered system of hot (disk), warm (cache) and cold (Tape) storage, with a common interface to all. Additionally, and more importantly, it takes the decision of storage location out of the user’s hands; the user only needs to interact with the service itself rather than manually specifying the movement of data between disk, cache, and tape.

The NLDS also provides a searchable catalog of files, with the user able to assign a label and tags to a collection of files.

_images/userview.png

Figure 1: User view of the Near-Line Data Store

Setting up the NLDS client

Before using the NLDS, a user must install and set-up the NLDS client. For the NLDS installation on JASMIN, the user must also:

  1. Have access to the Object Storage tenancy (cache) they wish to write files to

  2. Be a member of, and have read/write access to, the GroupWorkspace they wish to use for writing to tape

  3. The NLDS client is a command-line program with the name nlds, which requires installation into the users homespace, preferably into a Python virtual-environment. To install nlds, refer to the page installation.

The nlds client program relies on a configuration file stored in the user’s home directory with the path ~/.nlds-config. This contains information needed to connect the client to the NLDS server. For more in-depth information, refer to the page configuration.

The configuration file has the contents:

{
    "user" : {
        "default_user" : "{{ user_name }}",
        "default_group" : "{{ user_gws }}"
    },
    "server" : {
        "url" : "{{ nlds_api_url }}",
        "api" : "{{ nlds_api_version }}"
    },
    "authentication" : {
        "oauth_client_id" : "{{ oauth_client_id }}",
        "oauth_client_secret" : "{{ oauth_client_secret }}",
        "oauth_token_url" : "{{ oauth_token_url }}",
        "oauth_scopes" : "{{ oauth_scopes }}"",
        "oauth_token_file_location" : "~/.nlds-token"
    },
    "object_storage" : {
        "tenancy"    : "{{ object_store_tenancy }}",
        "access_key" : "{{ object_store_access_key }}",
        "secret_key" : "{{ object_store_secret_key }}"

    }
}

After installing the client, the nlds init command can be used to fill in the majority of this file. The init command will fill in:

  1. The server section.

  2. The authentication section.

The user will have to fill in the following sections:

  1. The user_name.

  2. The user_gws.

  3. The value of the oauth_token_file_location field in the authentication section can be changed to a location of the user’s choosing.

  4. Fields in the object_storage section are generated by the user interacting with the object storage portal to generate the access_key and secret_key values.

Running the NLDS client for the first time

Once the NLDS client is successfully installed, the user can run it. It is recommended that you run the init command first, which will fill in the url and authentication sections of the .nlds-config file for you - see the relevant seciton for details.

Once initialised, the first time nlds is run on JASMIN the user will be asked for their username and password. These are the JASMIN username and password, which the user usually uses to login into JASMIN via one of the login servers.

The following message will be displayed:

This application uses OAuth2 to authenticate with the server on your behalf.
To do this it needs your username and password.
Your password is not stored.  It is used to obtain an access token, which is stored in the file: ~/.nlds-token
Username:
Password:

After entering your JASMIN username and password, the OAuth2 token is created at the path pointed to by oauth_token_file_location in the ~/.nlds-config file. The NLDS authorisation system uses refresh tokens to fetch a new token when the current one has expired

How the NLDS data catalog is organised

When a user PUTs files into the NLDS, a record of those files is kept in a data catalog. This organises the files into Holdings, Transactions and Files, with the ability to add metadata to the Holdings in the form of Tags and Labels.

Figure 2 shows a simplified view of the NLDS data-catalog. This is covered in much more depth in the Catalog Organisation section.

_images/simple_catalog.png

Figure 2: Simplified view of the NLDS data-catalog

Getting help on the NLDS commands

The --help option can be used in conjunction with any command to get help on that specific command. To get a list of commands, just use the --help option.

> nlds --help
Usage: nlds [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  find     Find and list files.
  get      Get a single file.
  getlist  Get a number of files specified in a list.
  list     List holdings.
  meta     Alter metadata for a holding.
  put      Put a single file.
  putlist  Put a number of files specified in a list.
  stat     List transactions.
> nlds find --help

Usage: nlds find [OPTIONS]

  Find and list files. If no user or group is given then these values will
  default to the ``user:default_user`` and ``user:default values`` in the
  ``~/.nlds-config file``.

Options:
  -u, --user TEXT            The username to find files for.
  -g, --group TEXT           The group to find files for.
  -l, --label TEXT           The label of the holding which the files belong
                              to.  This can be a regular expression (regex).
  -i, --holding_id INTEGER   The numeric id of the holding which the files
                              belong to.
  -n, --transaction_id TEXT  The UUID transaction id of the transaction to
                              list.
  -p, --path TEXT            The path of the files to find.  This can be a
                              regular expression (regex)
  -t, --tag TAG              The tag(s) of the holding(s) to find files
                              within.
  -j, --json                 Output the result as JSON.
  --help                     Show this message and exit.

Copying a single file (PUT) to the NLDS

To put a single file into the NLDS use the command:

> nlds put <filepath>

This is the simplest form of the command and will PUT the file into a holding with a seemingly random label. It will also use the values of user and group from the Configuration File.

To specify the user and group:

> nlds put -u <name> -g <group> <filepath>

To give the newly created holding a sensible label:

> nlds put -l <label> <filepath>

(-u and -g can also be used at the same time as -l).

To add tags to the holding while PUTting a file to the NLDS:

> nlds put -l <label> -t <key:value> <filepath>

(-l is optional here).

To specify a job label use -b <job_label>. This is a convenience function for the user to allow them to group multiple transactions under a single job_label.

To get the return output from the put command in JSON format, specify the -j option.

When a command is invoked, NLDS will return a summary of the command, including the transaction_id.

> nlds put -l SheepPen /Users/frjohn/sheep.txt -u frjohn -g farmers -b SheepHerding -t sheepdog:skye
PUT transaction accepted for processing.
    user            : frjohn
    group           : farmers
    action          : put
    job label       : SheepHerding
    transaction id  : c31abca8-2bc4-43dc-ac64-7d28359d6828
    label           : SheepPen
    tags            : sheepdog : skye

Copying a list of files (PUTLIST) to the NLDS

To PUT more than one file into the NLDS, use the putlist command. This takes the same options as the put command but, rather than taking the path of a single file as an argument, it takes the name of a plain text file which contains the paths of the files to PUT into the NLDS. This filelist file must be in plain text format, with the path of a single file per line, for example:

/Users/frjohn/albatross.txt
/Users/frjohn/rabbit.txt

The command and response then becomes (where test_putlist is the name of the above file):

> nlds putlist -l Zoo test_list -u frjohn -g farmers -b test_putlist -t zoo:Bristol
PUT transaction accepted for processing.
    user            : frjohn
    group           : farmers
    action          : put
    job label       : test_putlist
    transaction id  : 41d412e2-1c1b-4d59-943a-40d9e717a0a1
    label           : Zoo
    tags            : zoo : Bristol

Querying the status of a transaction (STAT)

The put and putlist commands above create transactions. These are actions in the NLDS that carry out a specific task, usually either a put, putlist, get or getlist command. To view the status of a transaction, use the stat command. Invoke the stat command on its own to view the state of all the transactions for a user:

> nlds stat
State of transactions for user:frjohn, group:farmers
    id    action          job label       label           state                  last update
    1     put             SheepHerding    SheepPen        COMPLETE               2023-04-18 15:21:41
    2     put             test_putlist    Zoo             COMPLETE               2023-04-18 15:28:53

(the user_name and group have not been specified with the -u and -g arguments here, and so the defaults are read from the Configuration File.)

This table shows the numeric id, the action which created the transaction, the user-specified job label, the holding label, the last time the state was updated and the state of the transaction.

As the transaction is processed by the NLDS, it goes through a number of states and will end with a terminating state. All possible states are listed on the Status Codes page. The terminating states are:

  • COMPLETE

  • FAILED

  • COMPLETE_WITH_ERRORS

  • COMPLETE_WITH_WARNINGS

Results for the stat command can be filtered using the following options:

  • -b : filter on the user-specified job label.

  • -s : filter on the state. See Status Codes for a list of possible values.

  • -a : filter on the api action. Options are get, put getlist and putlist.

> nlds stat -s COMPLETE
State of transactions for user:frjohn, group:farmers, state:COMPLETE
    id    action          job label       label           state                  last update
    1     put             SheepHerding    SheepPen        COMPLETE               2023-04-18 15:21:41
    2     put             test_putlist    Zoo             COMPLETE               2023-04-18 15:28:53

> nlds stat -a put
State of transactions for user:frjohn, group:farmers, api-action:put
    id    action          job label       label           state                  last update
    1     put             SheepHerding    SheepPen        COMPLETE               2023-04-18 15:21:41
    2     put             test_putlist    Zoo             COMPLETE               2023-04-18 15:28:53

> nlds stat -b SheepHerding
State of transaction for user:frjohn, group:farmers
    id              : 1
    user            : frjohn
    group           : farmers
    action          : put
    transaction id  : ebb89e7d-5671-41f9-9f42-968fa69b0c87
    label           : SheepPen
    creation time   : 2023-04-18 15:21:36
    state           : COMPLETE
    warnings        :
    sub records     ->
    +   id           : 1
        sub_id       : 19ccd443-e269-465d-b0ed-51c5e98b8fad
        state        : COMPLETE
        retries      : 0
        last update  : 2023-04-18 15:21:41

In the last example, only one transaction was found, and so the entire details for a single transaction was returned. If more than one transaction had the same job_label, then the list format would be returned.

To guarantee to get the full information for a single transaction, the -i option can be used with the numeric id of the transaction. The -n option can also be used with the transaction id, if you know it.

> nlds stat -i 2
State of transaction for user:frjohn, group:farmers, id:2
    id              : 2
    user            : frjohn
    group           : farmers
    action          : put
    transaction id  : 41d412e2-1c1b-4d59-943a-40d9e717a0a1
    label           : Zoo
    creation time   : 2023-04-18 15:28:48
    state           : COMPLETE
    warnings        :
    sub records     ->
    +   id           : 2
        sub_id       : 8d457f6c-f26c-43db-98ff-7c2efe7ff695
        state        : COMPLETE
        retries      : 0
        last update  : 2023-04-18 15:28:53

> nlds stat -n 41d412e2-1c1b-4d59-943a-40d9e717a0a1
State of transaction for user:frjohn, group:farmers, transaction_id:41d412e2-1c1b-4d59-943a-40d9e717a0a1
    id              : 2
    user            : frjohn
    group           : farmers
    action          : put
    transaction id  : 41d412e2-1c1b-4d59-943a-40d9e717a0a1
    label           : Zoo
    creation time   : 2023-04-18 15:28:48
    state           : COMPLETE
    warnings        :
    sub records     ->
    +   id           : 2
        sub_id       : 8d457f6c-f26c-43db-98ff-7c2efe7ff695
        state        : COMPLETE
        retries      : 0
        last update  : 2023-04-18 15:28:53

In these last examples, the sub records are present as the NLDS will split large transactions (with many files, or large files) into smaller units of work, and create a sub record for each one of them.

Being able to easily check the progress of transactions in the NLDS is a key design idea. To enable a program to check the progress, the -j option can be used to return a JSON formatted version of the status.

Querying the file collections the user holds on the NLDS (LIST)

The put and putlist commands above create holdings in the NLDS catalog. Holdings can be thought of as collections of transactions which, in themselves, are collections of files Therefore, a holding can also be thought of as a collection of files.

To see the holdings that are assigned to a user in NLDS, use the list command. Invoke the list command on its own to see all of the holdings that a user has:

> nlds list
Listing holdings for user:frjohn, group:farmers
    id    label           ingest time
    1     SheepPen        2023-04-18 15:21:37
    2     Zoo             2023-04-18 15:28:48

(the user_name and group have not been specified with the -u and -g arguments here, and so the defaults are read from the Configuration File.)

This table shows the numeric id, label and latest ingest time for the holding. To examine the holding in more detail, the -i option can be used with the id, or the -l option can be used with the label.

> nlds list -l SheepPen
    id              : 1
    label           : SheepPen
    ingest time     : 2023-04-18 15:21:37
    transaction id  : ebb89e7d-5671-41f9-9f42-968fa69b0c87
    tags            : sheepdog : skye

> nlds list -i 2 -u frjohn -g farmers
    id              : 2
    label           : Zoo
    ingest time     : 2023-04-18 15:28:48
    transaction id  : 41d412e2-1c1b-4d59-943a-40d9e717a0a1
    tags            : zoo : Bristol

Finally, tags can be used to seach for a holding:

> nlds list  -t sheepdog:skye
    Listing holding for user:frjohn, group:farmers, tag:{'sheepdog': 'skye'}
        id              : 1
        label           : SheepPen
        ingest time     : 2023-04-18 15:21:37
        transaction id  : ebb89e7d-5671-41f9-9f42-968fa69b0c87
        tags            : sheepdog : skye

Querying the files the user holds on the NLDS (FIND)

To view which files the user holds in the NLDS, use the find command:

> nlds find
Listing files for holdings for user:frjohn, group:farmers
    h-id  h-label         path                              size    time
    1     SheepPen        /Users/frjohn/sheep.txt         49.0B   2023-04-18 15:21:37
    2     Zoo             /Users/frjohn/albatross.txt     96.0B   2023-04-18 15:28:48
    2     Zoo             /Users/frjohn/rabbit.txt        50.0B   2023-04-18 15:28:48

Warning : issuing the find command like this, with no filters, will make an attempt to list all of a user’s files. When a user has many files in the NLDS, this is likely to end in a gateway timeout, as the request will take too long to process. It is much better to use the options to the find command to limit the number of files that will be returned. This can be done in a number of ways, which will be illustrated below.

To list the files in a holding, use -i with the holding id (h-id) or -l with the holding label (h-label).

> nlds find -i 1
Listing files for holding for user:frjohn, group:farmers, holding_id:1
    path            : /Users/frjohn/sheep.txt
    type            : FILE
    size            : 49.0B
    user uid        : 0
    group gid       : 20
    permissions     : rw-r--r--
    ingest time     : 2023-04-18 15:21:37
    storage location: OBJECT_STORAGE

> nlds find -l Zoo
Listing files for holding for user:frjohn, group:farmers, label:Zoo
    h-id  h-label         path                              size    time
    2     Zoo             /Users/frjohn/albatross.txt     96.0B   2023-04-18 15:28:48
    2     Zoo             /Users/frjohn/rabbit.txt        50.0B   2023-04-18 15:28:48

In the first example, only one file is returned, so the full details are shown. To view the particular details of a file in the second example, the filepath of the file can be used with the -p argument.

> nlds find -l Zoo -p /Users/frjohn/rabbit.txt
Listing files for holding for user:frjohn, group:farmers, label:Zoo
    path            : /Users/frjohn/rabbit.txt
    type            : FILE
    size            : 50.0B
    user uid        : 504
    group gid       : 20
    permissions     : rw-r--r--
    ingest time     : 2023-04-18 15:28:48
    storage location: OBJECT_STORAGE

The filepath argument can be a regular expression:

> nlds find -l Zoo -p /Users/frjohn/a.*
Listing files for holding for user:frjohn, group:farmers, label:Zoo
    path            : /Users/frjohn/albatross.txt
    type            : FILE
    size            : 96.0B
    user uid        : 504
    group gid       : 20
    permissions     : rw-r--r--
    ingest time     : 2023-04-18 15:28:48
    storage location: OBJECT_STORAGE

Finally, tags can be used to list files from holdings that contain those tags:

> nlds find  -t zoo:Bristol
Listing files for holding for user:frjohn, group:farmers, tag:{'zoo': 'Bristol'}
    h-id  h-label         path                              size    time
    2     Zoo             /Users/frjohn/albatross.txt     96.0B   2023-04-18 15:28:48
    2     Zoo             /Users/frjohn/rabbit.txt        50.0B   2023-04-18 15:28:48

Retrieving a single file from the NLDS (GET)

After the put and putlist commands above, there are three files in two holdings in the NLDS, which can be seen by issuing the find command.

> nlds find
Listing files for holdings for user:frjohn, group:farmers
    h-id  h-label         path                              size    time
    1     SheepPen        /Users/frjohn/sheep.txt         49.0B   2023-04-18 15:21:37
    2     Zoo             /Users/frjohn/albatross.txt     96.0B   2023-04-18 15:28:48
    2     Zoo             /Users/frjohn/rabbit.txt        50.0B   2023-04-18 15:28:48

NLDS supports ways five of retrieving these files, by using the get command in conjunction with:

  1. The full file path.

  2. A regular expression that evaluates to more than one file path.

  3. A holding id.

  4. A holding label.

  5. A holding tag.

By default the NLDS will try to retrieve the file to the original location indicated by the path field above. If the user does not wish to overwrite any file that might now have that filepath, or they do not have permission to access that filepath, then the optional target directory can be specified with the option -r <target_path>

1. To GET a file using the fully qualified filepath, and write it to the current directory, invoke the command:

> nlds get /Users/frjohn/sheep.txt -r ./
GETLIST transaction accepted for processing.
    user            : frjohn
    group           : farmers
    action          : getlist
    job label       : 14bc9846
    transaction id  : 14bc9846-9d45-440a-af6c-dfcb5cb9dcae

This will also recreate the directory structure in the current directory, i.e. the file will be written to <current_directory>/Users/frjohn/sheep.txt.

Using this method of retrieval, no holding is specified. If more than one file with the filepath is held in the NLDS (in different holdings), then the latest ingested file will be returned. See 3. A holding id or 4. A holding label to guarantee that a specific file, ingested at a specific time is returned.

2. NLDS understands regular expressions (regex). This is a useful tool as it allows a user to get files depending on a pattern. One use case would be to get all of the files beneath a certain directory. To get files using regular expressions, use the command:

> nlds get "/Users/frjohn/.*" -r ./
GETLIST transaction accepted for processing.
    user            : frjohn
    group           : farmers
    action          : getlist
    job label       : 016ae5b7
    transaction id  : 016ae5b7-3b0d-485c-8812-7b3fa1b003de

(The Users/frjohn/.* has to be in "" so as to not confuse the shell, which will see the * as a wildcard and try to expand it to all files and directories in the current path. Enclosing it in "" prevents this.)

3. To ensure that the file at a filepath retrieved from the NLDS is a particular version, ingested on a particular day, the holding id can be specified in the get command using the option -i.

> nlds get -i 2 /Users/frjohn/albatross.txt -r ./
GETLIST transaction accepted for processing.
    id              : 2
    user            : frjohn
    group           : farmers
    action          : getlist
    job label       : fb130e43
    transaction id  : fb130e43-25a3-4cef-9eeb-b31e72a1f808

If the user attempts to get a file from a holding that does not contain it, then an error will be returned when a stat command is used to check the status of the transaction.

> nlds get -i 1 /Users/frjohn/albatross.txt
GETLIST transaction accepted for processing.
    id              : 1
    user            : frjohn
    group           : farmers
    action          : getlist
    job label       : 5842d371
    transaction id  : 5842d371-bf07-4ad4-a6ff-c46876a84ca6

> nlds stat -i 13
State of transaction for user:frjohn, group:farmers, id:13
    id              : 13
    user            : frjohn
    group           : farmers
    action          : getlist
    transaction id  : 5842d371-bf07-4ad4-a6ff-c46876a84ca6
    label           :
    creation time   : 2023-04-24 16:01:05
    state           : FAILED
    warnings        :
    sub records     ->
    +   id           : 13
        sub_id       : 85e36999-fc80-4784-acbd-dcc2c9977bbd
        state        : FAILED
        retries      : 6
        last update  : 2023-04-24 16:01:06
        failed files ->
        +    filepath : /Users/frjohn/albatross.txt
            reason   : File:/Users/frjohn/albatross.txt not found in holding:SheepPen for user:frjohn in group:farmers.

4. Similary to 3, a label can be used to fetch a particular file from a holding.

> nlds get -l SheepPen /Users/frjohn/sheep.txt -r ./
GETLIST transaction accepted for processing.
    user            : frjohn
    group           : farmers
    action          : getlist
    job label       : SheepPen
    transaction id  : b0544301-aadb-4f26-b2e0-b4643b0adfee
    label           : SheepPen

Again, specifying a label that doesn’t exist or a file that does not exist in the holding will return an error:

> nlds get -l sheeppen /Users/frjohn/sheep.txt -r ./
GETLIST transaction accepted for processing.
    user            : frjohn
    group           : farmers
    action          : getlist
    job label       : sheeppen
    transaction id  : c07dd223-88cb-41ec-a1e2-00f31c162116
    label           : sheeppen

> nlds stat -i 15
State of transaction for user:frjohn, group:farmers, id:15
    id              : 15
    user            : frjohn
    group           : farmers
    action          : getlist
    transaction id  : c07dd223-88cb-41ec-a1e2-00f31c162116
    label           :
    creation time   : 2023-04-24 16:11:03
    state           : FAILED
    warnings        :
    sub records     ->
    +    id           : 15
        sub_id       : 16759739-bc53-44be-b678-3999d0f7b76b
        state        : FAILED
        retries      : 0
        last update  : 2023-04-24 16:11:04
        failed files ->
        +    filepath : /Users/frjohn/sheep.txt
            reason   : Exception during callback: Could not find record of requested holding: label: sheeppen, id: None

5. Finally, a tag can be specified when retrieving files:

> nlds get -t sheepdog:skye /Users/frjohn/sheep.txt -r ./
GETLIST transaction accepted for processing.
    user            : frjohn
    group           : farmers
    action          : getlist
    job label       : 23db226f
    transaction id  : 23db226f-d377-4511-8cc8-656b006c4f1e
    tags            : sheepdog : skye

Retrieving a list of files from the NLDS (GETLIST)

Similarly to how the user can PUT a list of files to the NLDS, the user can also GET files specified as a list in a text file. As before, this filelist file must be in plain text format, with the path of a single file per line, for example:

/Users/frjohn/albatross.txt
/Users/frjohn/rabbit.txt

The command and response then becomes (where test_getlist is the name of the above file). Here give the retrieval a job_label of getlisttest:

> nlds getlist test_list -r ./ -b getlisttest
GETLIST transaction accepted for processing.
    user            : frjohn
    group           : farmers
    action          : getlist
    job label       : getlisttest
    transaction id  : d83d8c2d-7ba8-4be8-b8dd-c643e4bfba49

The stat command can then be used to check on the progress of the retrieval.

> nlds stat -b getlisttest
State of transaction for user:frjohn, group:farmers
    id              : 18
    user            : frjohn
    group           : farmers
    action          : getlist
    transaction id  : d83d8c2d-7ba8-4be8-b8dd-c643e4bfba49
    label           :
    creation time   : 2023-04-25 14:34:37
    state           : COMPLETE
    warnings        :
    sub records     ->
    +    id           : 18
        sub_id       : 419d164a-119a-4e6e-b919-fef154902066
        state        : COMPLETE
        retries      : 0
        last update  : 2023-04-25 14:34:38

A holding id or label can be used in conjunction with getlist to make sure that the correct version of the filepaths in the filelist are retrieved.

> nlds getlist -i 2 test_list -r ./ -b getlist2
GETLIST transaction accepted for processing.
    id              : 2
    user            : frjohn
    group           : farmers
    action          : getlist
    job label       : getlist2
    transaction id  : 441065cb-fc14-4260-9661-2a3b163a5dce

However, if the filelist contains files that are not in the holding id, a different terminating state will be produced for the transaction. If the filelist now contains:

/Users/frjohn/albatross.txt
/Users/frjohn/rabbit.txt
/Users/frjohn/sheep.txt

and the command is issued:

> nlds getlist -i 1 test_list -r ./ -b getlist3
GETLIST transaction accepted for processing.
    id              : 2
    user            : frjohn
    group           : farmers
    action          : getlist
    job label       : getlist3
    transaction id  : c4c520a1-fd2b-42cd-9a36-32ba94f3b562

then the terminating state is COMPLETE_WITH_ERRORS because the /Users/frjohn/sheep.txt is not a part of the Zoo holding:

> nlds stat -i 22
State of transaction for user:frjohn, group:farmers, id:22
    id              : 22
    user            : frjohn
    group           : farmers
    action          : getlist
    transaction id  : 0f1d711e-6227-4bd9-bd0b-d1b7ed47220f
    label           :
    creation time   : 2023-04-25 14:44:52
    state           : COMPLETE_WITH_ERRORS
    warnings        :
    sub records     ->
    +   id           : 22
        sub_id       : 4c38aab3-16ae-4c21-bce6-9171e15fc231
        state        : COMPLETE
        retries      : 0
        last update  : 2023-04-25 14:44:54
    +   id           : 23
        sub_id       : 668de6ae-7c0e-4763-82bc-ebf7135f4420
        state        : FAILED
        retries      : 6
        last update  : 2023-04-25 14:44:53
        failed files ->
        +   filepath : /Users/frjohn/sheep.txt
            reason   : File:/Users/frjohn/sheep.txt not found in holding:Zoo for user:frjohn in group:farmers.

Changing the label of a file collection (META)

The meta command allows the user to change the label of a holding or alter the holding’s tags. Changing the label is particularly useful if an automatically, seemingly random label has been assigned to the holding:

> nlds list
Listing holdings for user:frjohn, group:farmers
    id    label           ingest time
    1     SheepPen        2023-04-18 15:21:37
    2     Zoo             2023-04-18 15:28:48
    3     e4c00744        2023-04-25 15:07:18

Here the holding with id 3 has the automatically generated label of e4c00744. This can be changed to something more rememberable by using the meta command with the -L (for new label) option. To specify which holding to change the label for the -i (holding id) or -l (existing label) options can be used.

> nlds meta -i 3 -L Farm
Changed metadata for holding for user:frjohn, group:farmers, holding_id:3
    id  : 3
        old metadata:
            label   : e4c00744
            tags    : {}
        new metadata:
            label   : Farm
            tags    : {}

> nlds meta -l Farm -L SmallHolding
Changed metadata for holding for user:frjohn, group:farmers, label:Farm
    id  : 3
        old metadata:
            label   : Farm
            tags    : {}
        new metadata:
            label   : SmallHolding
            tags    : {}

Adding tags to a file collection (META)

In addition to allowing the user to change the label of a holding, the meta command also allows the user to add or alter the holding’s tags. Tags are specified as key:value pairs, and using the -T (new tags) option. For a holding with no tags, specifying tags will add them to the holding:

> nlds meta -l SmallHolding -T type:arable
Changed metadata for holding for user:frjohn, group:farmers, label:SmallHolding
    id  : 3
        old metadata:
            label   : SmallHolding
            tags    : {}
        new metadata:
            label   : SmallHolding
            tags    : {'type': 'arable'}

Extra tags can then also be added:

> nlds meta -l SmallHolding -T address:"1 Cow Lane"
Changed metadata for holding for user:frjohn, group:farmers, label:SmallHolding
    id  : 3
        old metadata:
            label   : SmallHolding
            tags    : {'type': 'arable'}
        new metadata:
            label   : SmallHolding
            tags    : {'type': 'arable', 'address', '1CowLane'}

Note that adding tags strips the spaces from any keys or values. If a holding already has a tag with a key, and the user specifies the key with a new value, then the tag is modified:

> nlds meta -l SmallHolding -T type:animal
Changed metadata for holding for user:frjohn, group:farmers, label:SmallHolding
    id  : 3
        old metadata:
            label   : SmallHolding
            tags    : {'type': 'arable', 'address': '1CowLane'}
        new metadata:
            label   : SmallHolding
            tags    : {'type': 'animal', 'address': '1CowLane'}

It is also possible to delete a tag. Here the -t option is used to specify the holding(s) to delete the tag from, and the -D option is used to name the tag that is to be deleted:

> nlds meta -t type:animal -D type:animal
Changed metadata for holding for user:frjohn, group:farmers, tag:{'type': 'animal'}
    id  : 3
        old metadata:
            label   : SmallHolding
            tags    : {'type': 'animal', 'address': '1CowLane'}
        new metadata:
            label   : SmallHolding
            tags    : {'address': '1CowLane'}