CERN Tape Archive Set Up

As part of the development of the NLDS, a tape emulator was set up to better understand how to interact with xrootd and the CERN tape archive. The following instructions detail how to set up such a tape emulator and are adapted from the instructions on [the CTA repo] (https://gitlab.cern.ch/cta/CTA/-/tree/main/continuousintegration/buildtree_runner) and those provided by STFC’s Scientific Computing Department at the Rutherford Appleton Laboratory. There are two major parts: commissioning the virtual machine, and setting it up appropriately according to the CERN instructions.

Commissioning the VM

These instructions are specifically for commissioning a VM on the STFC openstack cloud interface. For other machines, slightly different instructions will need to be followed.

After you have logged in to the openstack interface, you click on the “Launch Instance” button on the top to create the VM and then:

1. In the “Details” tab, give your VM a suitable name

2. In the “Source” tab, select scientificlinux-7-aq as a source image

3. In the “Flavour” tab, select a VM type depending on how many VCPUs, RAM and disk size you need

4. In the “Networks” tab, select “Internal”

5. In the “Key Pair” tab, upload your public rsa ssh key so you can login to the VM once it is created.

6. In the “Metadata” tab, click on the pull-down menu “Aquilon Image Properties” and then set it to “Aquilon Archetype”, specifying ral-tier1, and also “Aquilon Personality” specifying it as eoscta_ci_cd. Note that you will have to manually write these values out so it’s worth copy pasting to avoid typos!

7. Press the “Launch Instance” button and the VM will be created. Give it some time so that quattor runs - quattor being a vm management tool like Puppet. It may also need a reboot at some point.

This setup produces a vm which requires logging in as your openstack username - in most cases this will be your STFC federal ID. You will be able to sudo assuming the machine remains in the correct configuration.

Note: The above setup is one that theoretically works. However, the machine which – after some attempts – successfully had CTA installed on it had to be commissioned manually by SCD so that

1. quattor could be made sure to have run successfully and then subsequently disabled

2. I would be able to log in as root thus negating the need to edit the sudoers file after each reboot.

I would strongly recommend this approach if SCD are agreeable to commissioning your vm for you.

Setting up CTA on the VM

The following are the working set of instructions at time of writing, provided by SCD at RAL.

• Ensure quattor is not running

  There should be an empty file at /etc/noquattor, if there is not one then create it with

  sudo touch /etc/noquattor

• Clone the CTA gitlab repo

  git clone https://gitlab.cern.ch/cta/CTA.git

• User environment

  As per instructions

  cd ./CTA/continuousintegration/buildtree_runner/vmBootstrap

  BUT in bootstrapSystem.sh, delete/comment line 46 and then

  ./bootstrapSystem.sh

  When prompted for password, press return (i.e don’t give one). This creates the cta user and adds them to sudoers

• CTA build tree

  As per instructions

  su - cta cd ~/CTA/continuousintegration/buildtree_runner/vmBootstrap

  BUT edit lines 54,55 in bootstrapCTA.sh to look like

  sudo wget https://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol7 -O /etc/pki/rpm-gpg/RPM-GPG-KEY-oracle --no-check-certificate sudo wget https://download.ceph.com/keys/release.asc -O /etc/pki/rpm-gpg/RPM-ASC-KEY-ceph --no-check-certificate

  Note the change in the URL on line 55 from git.ceph.com to download.ceph.com, as well as the addition of the --no-check-certificate flag.

  Then run bootstrapCTA.sh (without any args)

  ./bootstrapCTA.sh

• Install MHVTL

  As per instructions

  cd ~/CTA/continuousintegration/buildtree_runner/vmBootstrap ./bootstrapMHVTL.sh

• Kubernetes setup

  As per instructions

  cd ~/CTA/continuousintegration/buildtree_runner/vmBootstrap ./bootstrapKubernetes.sh

  and reboot host

  sudo reboot

• Docker image

  Depending on how your machine was set up you may now need to ensure that quattor is still disabled (i.e. that the /etc/noquattor file still exists) and that the cta user is still in the sudoers file. This will not be necessary if you are running as root.

  Then, as per instructions

  su - cta cd ~/CTA/continuousintegration/buildtree_runner

  BUT edit lines 38,39 in /home/cta/CTA/continuousintegration/docker/ctafrontend/cc7/buildtree-stage1-rpms-public/Dockerfile to look like

  RUN wget https://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol7 -O /etc/pki/rpm-gpg/RPM-GPG-KEY-oracle --no-check-certificate RUN wget https://download.ceph.com/keys/release.asc -O /etc/pki/rpm-gpg/RPM-ASC-KEY-ceph --no-check-certificate

  then run the master script to prepare all the Docker images.

  ./prepareImage.sh

• Preparing the environment (MHVTL, kubernetes volumes…)

  As per instructions

  cd ~/CTA/continuousintegration/buildtree_runner sudo ./recreate_buildtree_running_environment.sh

• Preparing the CTA instance

  As per instructions

  cd ~/CTA/continuousintegration/orchestration sudo ./create_instance.sh  -n cta -b ~ -B CTA-build -O -D -d internal_postgres.yaml

  This may work first time but it never did for me, so the fix is to then run

  ./delete_instance.sh -n cta

  To remove the instance and then re-create it with the same command as above sudo ./create_instance.sh  -n cta -b ~ -B CTA-build -O -D -d internal_postgres.yaml

  This can be verified to be working with a call to

  kubectl -n cta get pods

  which should return a list of the working pods, looking something like:

  NAME	READY	STATUS	RESTARTS	AGE
  client	1/1	Running	0	35m
  ctacli	1/1	Running	0	35m
  ctaeos	1/1	Running	0	35m
  ctafrontend	1/1	Running	0	35m
  kdc	1/1	Running	0	35m
  postgres	1/1	Running	0	36m
  tpsrv01	2/2	Running	0	35m
  tpsrv02	2/2	Running	0	35m

Additional steps

The above steps will get you 90% of the way there but there’s some extra steps required get xrd- commands running, and more for getting it running with pyxrootd.

There’re 3 main pods you’ll want/need to use: 1. client - where you actually run the xrootd commands (as user1) and will eventually run the pyxrootd commands and the archiver 2. ctaeos - where you run the eos commands, which are basically filesystem admin commands, making directories, setting attributes of directories etc. 3. ctacli - where you run cta-admin commands which, as far as I can tell, control the tape system metadata.

For any of the above to work you’ll first need to run the preparation script /home/cta/CTA/continuousintegration/orchestration/tests/prepare_tests.sh to fill the cta database with the appropriate test metadata. This may or may not work straight up, but you may also need to run the command

kubectl --namespace cta exec ctafrontend -- cta-catalogue-admin-user-create /etc/cta/cta-catalogue.conf --username ctaadmin1 -m "docker cli"

which recreates the ctaadmin1 user on the ctafrontend pod, a potentially necessary step after you’ve recreated the ctaeos instance - which you will probably have to to get it working (as per the “Preparing the CTA instance” section above).

XRootD Python Bindings

The python xrootd bindings are a bit of a pain to get to work on centos images, but after some (read as lots) of trial and error I managed to compile it using newer versions of a few packages and have baked it into a docker container. The image is on the [ceda registry under the NLDS consumers project] (registry.ceda.ac.uk/nlds-consumers/archiver). The difficulty is getting these onto the client pod to be useful in developing against, but this was achieved by taking a snapshot image of the running client pod’s container and then running a multi-stage build with both this image and the archiver image. [This image is now available on the registry] (registry.ceda.ac.uk/nlds-consumers/ctaeos-client). You should be able to deploy this using the same pod.yaml that client uses, which you can steal by running:

kubectl get pod -n cta client -o yaml

and then save to a file, change out the image for your new image and then redeploy with

kubectl apply -f pod.yaml

There are many ways to get images to the local repository so I will leave that as an exercise to the reader.