Cheatsheet for test stands (serenitybutler etc.)

The Serenity butler can be used to perform the configuration/monitoring procedures that are required in test stands most often, as outlined in the sections below. For some more advanced procedures that don’t commonly take place in test stands you may need to use other tools than serenitybutler. Internally the butler invokes the required sequences of SMASH functions and/or measurements, and it supports a minimal set of command-line options that should cover most of the typical test stand procedures.

Note

Before running any of the commands listed on this page, you should set up your environment variables as follows:

source /opt/cactus/bin/serenity/serenity_env.sh

You should also also either set SMASH_DEFAULT_CONFIG to the path of the SMASH config file that describes your board’s daugther card & firefly configuration, or reference that config file from the smash entry in your Serenity butler config file (whose path should be specified using the SERENITY_CONFIG variable).

Version checks

To check the versions of software packages are installed, and what versions of firmware are loaded onto the FPGAs:

serenitybutler info

Power control

To turn on power to X0, X1, or to both sites:

serenitybutler power on (x0|x1|both)

To power cycle the X0 site, X1 site, or both of them:

serenitybutler power cycle (x0|x1|both)

To turn off power to X0, X1 or to both sites:

serenitybutler power off (x0|x1|both)

To check the status of the power modules:

serenitybutler power status (x0|x1|both)

Reference clock and firefly configuration

Note

All firefly modules and clock synthesisers associated with a site need to be be reconfigured after powering on that site.

To configure the clock synthesisers that provide the MGT reference clocks:

serenitybutler clocks configure (x0|x1|both)

By default, the clock synthesisers will be configured to output 320MHz clocks, which is suitable for operating both 16 and 25Gbps links in EMP-based firmware. However, if you need to have a different reference clock frequency, you can specify a different clock configuration file using the --config option.

To configure all Firefly optical modules on the board:

serenitybutler fireflys configure

To then check the status of the Firefly modules:

serenitybutler fireflys status

Programming the daughter card FPGAs

To program the daughter card FPGA(s) at X0, X1 or both sites:

serenitybutler program (x0|x1|both) <path_to_bitfile/bitfile_name.bit>

You can also add the -r flag to the end of this command, in order to power cycle the site(s) and reconfigure the clock synthesisers before loading the bitfile.

Crosspoint switch

On the Serenity Z1.2 board, there is a high-speed switch that routes the TCDS signals between backplane as the daughter card that is currently in use. The crosspoint switch can be configured to route the TCDS signals appropriately as follows:

serenitybutler xpoint-switch configure (x0|x1)

XVC server

To run an XVC server (port 5000) connected to the FPGA at either one or both of the sites:

serenitybutler jtag xvc (x0|x1|both)

Just leave this command running while you want the XVC server, and stop the script with Ctrl+C whenever you want to stop the server. To change the port number to a non-default value, just add one extra argument - e.g. to run the XVC server on port 9999:

serenitybutler jtag xvc (x0|x1|both) 9999

Or to run XVC servers for both FPGAs, but on separate ports, e.g. 5000 for X0 and 5001 for X1:

serenitbutler jtag xvc both 5000 5001

Artix TTC

Note

In order for the commands in this section to work, the Artix address table must be specified in the Serenity butler config file. The format of that file is discussed in the Artix TTC page, along with a more in-depth summary of the functionality of the Artix TTC firmware.

To configure the TTC logic to distribute internally-generated clock and TTC signals:

serenitybutler artix reset internal

To configure the TTC logic to use an external TTC source instead:

serenitybutler artix reset external

Test suite

The serenitybutler test suite (implemented in Python, using pytest) validates that the firmware and software are functioning as expected. The tests are organised into many test cases, each of which tests a specific workflow or area of functionality with a specific set of parameters. These test cases are grouped into three test modules:

  • test_i2c - Power cycle daughter card and configure clock synths

  • test_jtag - Program daughter card with valid and invalid bitfiles

  • test_ttc - Reset Artix TTC controller using both internal and external TTC clock and data sources

The test suite can be run as follows:

python -m serenity.tests --site [site] --repeat=[N] --config [serenitybutler config file] --tarball [firmware tarball]

For example:

python -m serenity.tests --site x0 --repeat=2 --config serenity.yml --tarball bitfiles/serenity_vu13p_so2.tgz --external-ttc-clock-only

Warning

If external TTC sources are present, use flag --external-ttc (both clock and data sources) or --external-ttc-clock-only (clock source only).

If the board PROM is programmed with a bitfile, use flag --prom-programmed.

Correctly setting these flags is important so that the test suite knows which tests should be run and which ones should pass and fail.

Note

If you want to only run a subset of tests, you can specify the name(s) or part of the name(s) of the test module or test cases using the -k flag. For example, to only run the I2C and TTC tests:

python -m serenity.tests -k 'test_i2c or test_ttc' ...

Or to skip the firefly tests:

python -m serenity.tests -k 'not firefly' ...

Running docker containers

If you are running docker containers that need access to the board’s FPGAs, we strongly recommend starting those containers using the /opt/cactus/bin/serenity/docker-run.sh script rather than invoking docker run directly. It is a wrapper script that invokes docker run internally, passes all user-supplied arguments onto that command, and automatically adds other arguments that are required in order to allow applications running in the containers to communicate with the FPGAs.

For example, with the wrapper script, you can run the ‘Serenity watch’ container in the background (listening on TCP port 8000, and bind mounting a SMASH config to /board.smash) as follows:

/opt/cactus/bin/serenity/docker-run.sh -d -p 8000:8000 -v /path/to/myBoard.smash:/board.smash gitlab-registry.cern.ch/p2-xware/software/serenity-watch/serenity-watch:master

Note: If there are problems in accessing files in certain cirumstances (in particular, PCIe device files after programming the FPGA), then make sure that the PCP SELinux RPM is updated to pcp-selinux-4.3.2-13.el7_9.x86_64 (e.g. by running sudo yum update pcp-selinux-4.3.2-13.el7_9.x86_64) - there appears to be a bug in pcp-selinux-4.3.2-12.el7.

Changing the CPU frequency governor

On Intel Atom COM express modules, it’s possible to select from one of two CPU frequency ‘governor’ settings, that provide a different balance between power usage and performance:

  • powersave: This governor forces the CPU to the lowest possible frequency

  • performance: Forces the CPU to use the highest possible frequency

We typically recommend running with the performance governor as this setting doesn’t pose any problems with cooling or power usage on the Serenity.

The governor can be set at runtime with the following command:

sudo cpupower frequency-set --governor GOVERNOR_NAME

And you can find out which governor is currently active as follows:

sudo cpupower frequency-info --policy