First Steps¶

[edit this page] pdf version

It is assumed that you have a pre-configured SERENITY prototype board and access to Vivado software with a connection to the JTAG cable connection on the SERENITY board. Delivered systems will have a “cmx” user account with sudo access, therefore the following commands should work on that given system. Hostnames and specific commands are given as an example.

The SERENITY Prototype board is shown below. One daughter card (DC) site is populated (X1) with a the black heat-sink mounted on the top. The JTAG interface with connections is visible to the top. The ETHERNET connector and ATCA backplane adaptor (power) is mounted on the top right.

Accessing the Daughter Card Sites:¶

There are three methods of accessing the daughter card sites via JTAG:
- directly from the COM Express (serenitybutler program...) - fastest method
- via a cable connected to the jtag header (labelled “IPMC”)
- or remotely through the COM Express (XVC)

Below we describe installation on a standard SERENITY board with embedded system as delivered. There is also a troubleshooting guide which will be expanded as the prototype systems are deployed amongst users.

Most of the configuration is handled by the serenitybutler which is explained in more extensive detail in a cheatsheet reference. For more complete and advanced usage see the SMASH documentation.

Installation (SMASH & EMP)¶

(optional) open a vivado session (e.g. on greg-special)
- navigate to Flow -> Open Hardware Manager specific carrier card (jtag end-point)
- Daughter Card (DC) FPGA should be powered off, therefore no DC connection is visible
Log into the SERENITY host (COM Express)
- e.g. ssh -X cmx@serenity-2368-04 (also need sudo permission)

Remove existing (legacy) packages in case they exist:

sudo yum remove "cactuscore-uhal*" "cactusboards-mp7*" "smash*" "cactusboards-emp*" "cactusboards-serenity*"
sudo yum groups mark remove smash
sudo yum groups mark remove emp
sudo yum groups mark remove uhal

Download the appropriate YUM repo files, and copy under /etc/yum.repos.d (Requires sudo permission for the account on the SERENITY board and first time needs password):

sudo curl https://serenity.web.cern.ch/serenity/board-setup/_downloads/ipbus-sw.centos7.repo -o /etc/yum.repos.d/ipbus-sw.repo
sudo curl https://serenity.web.cern.ch/serenity/board-setup/_downloads/smash.repo -o /etc/yum.repos.d/smash.repo
sudo curl https://serenity.web.cern.ch/serenity/board-setup/_downloads/emp.repo -o /etc/yum.repos.d/emp.repo
sudo curl https://serenity.web.cern.ch/serenity/board-setup/_downloads/serenity.repo -o /etc/yum.repos.d/serenity.repo
sudo curl http://serenity.web.cern.ch/serenity/emp-fwk/_downloads/cactus-build-utils.repo -o /etc/yum.repos.d/cactus-build-utils.repo

Install the software (again with sudo rights):

sudo python3.6 -m pip install --upgrade click==8.0.4 click_didyoumean==0.3.0 pexpect==4.8.0 pytest==7.0.1 pexpect==4.8.0  pyyaml==6.0 python-dateutil==2.8.2 colorama==0.4.5
sudo yum clean all
sudo yum groupinstall smash emp uhal
sudo yum install cactuscore-uhal-python36 smash-python36 "cactusboards-serenity*" python36-rpm cactuscore-build-utils-0.2.9 prometheus-node_exporter

The yum install command will check for dependencies so more than three packages will be installed. After checking a confirmation is required (y) to proceed with the installation.

The output looks like this (and choose y if asked):

Ends with the satisfying Complete!

You can compare against the latest versions by following the reference Current software and firmware versions, which also describes way to fix any problems.

On the SERENITY baseboard ComX there is typically a cmx shared account and it is important that you create your own filespace under this area. Therefore we suggest a directory using your name or your CERN login (MY_USERNAME):

cd /home/cmx
mkdir MY_USERNAME
cd MY_USERNAME

Power up!¶

This procedure will switch on a daughter card site.

Environment

source /opt/cactus/bin/serenity/serenity_env.sh

Power on Daughter Card sites using SMASH:

If the default SMASH config file variable hasn’t already been set:
export SMASH_DEFAULT_CONFIG=/path/to/someBoardConfig.smash
For X0 DC:
serenitybutler power on x0
For X1 DC:
serenitybutler power on x1
Here the relevant SMASH commands are issued by the butler script internally. Note that in order for SMASH to function correctly someBoardConfig.smash should point to a configuration file that actually represents the hardware configuration you have on your board. It is suggested to look at the packaged configuration files within /opt/smash/etc/serenity/configuration and adapt accordingly. More details on SMASH are available within the documentation.

Uploading firmware to the daughter cards¶

If you’re loading a bitfile onto a Serenity daughter card FPGA for the first time, we recommend that you use one of the EMP null algo builds. The EMP framework provides common infrastructural firmware - as well as corresponding control and monitoring software - to support arbitrary user-created payload firmware, e.g. the reconstruction algorithm in a level-1 track finder or level-1 trigger board. In the null algo builds, the data received on each input channel is fed into the corresponding output channel, after several clock cycles. The null algo bitfiles for the latest EMP release can be found here.

The recommended method for uploading firmware to the daughter card FPGAs is to load the bitfile directly from an application running on the COM express, via the Artix-7 FPGA.

Alternately, the daughter card FPGAs can be programmed via a SMASH XVC (Xilinx Virtual Cable) server, which creates a network-based interface for Vivado to access the JTAG connection via the COM express. Finally, if a JTAG programmer cable is connected to the board, that can be used to load bitfiles instead of the ‘direct’ and ‘XVC server’ methods.

Each of these procedures are described below.

After the bitfile has been copied onto the COM express, it can be loaded onto the FPGA using the following Serenity butler command:

export SMASH_DEFAULT_CONFIG=/path/to/someBoardConfig.smash
serenitybutler program SITE /path/to/top.bit

where SITE should be changed to x0 for the FPGA on the X0 site, x1 for X1, or both to program both sites.

By adding a -r flag to this command, i.e:

serenitybutler program SITE /path/to/top.bit -r

it will power cycle the site(s) and reconfigure the clock synthesisers before loading the bitfile.

In this method, the SMASH XVC server mediates a ‘logical’ JTAG connection between the FPGA and Vivado software via the ComExpress and ultimately Ethernet. The only requirement is to have the machine hosting Vivado software running on the same subnet as the SERENITY. Tunnelling could also be used to link Vivado with the hardware.

With the FPGA powered start smash and instantiate the XVC:

smash.exe -c /path/to/someBoardConfig.smash "DC0:FPGA Decorate XVC 5000" -b

This should then start the virtual cable and print some lines of code (with the correct addresses). NB: the -b argument results in keeping XVC running always.
The commands printed to the screen above need to be run within the Vivado session using the TCL command-line or as a script:
```
open_hw
connect_hw_server
open_hw_target -xvc_url 128.141.YYY.XXX:5000
set current_hw_jtag [get_property hw_jtag [get_hw_target localhost:3121/xilinx_tcf/Xilinx/128.141.YYY.XXX:5000]]
```
where XXX and YYY are your own unique subnet address. The 3121 is a Vivado default and the 5000 is the port which needs to be open for communication to work.
If the XVC fails to connect then there is a chance that the firewall on the ComExpress is somehow blocking it. In which case first check that 5000 is open:
```
sudo firewall-cmd --zone=public --list-ports
```
If port 5000 is not listed then you can first try adding it:
```
sudo firewall-cmd --zone=public --add-port=5000/tcp --permanent
```
But if connection still fails (timeouts) then it is suggested to disable the firewall completely:
```
sudo systemctl stop firewalld
```
Then restart “XVC Run” and re-run the TCL script within Vivado.
The FPGA should now be visible for programming within Vivado.

More details about SMASH and the associated commands for the virtual JTAG can be found here

Connect the FGPA to the JTAG header:

For X0 DC:

smash.exe -c /path/to/someBoardConfig.smash -c /opt/smash/etc/serenity/examples/jtag_header_x0.smash

For X1 DC site:

smash.exe -c /path/to/someBoardConfig.smash -c /opt/smash/etc/serenity/examples/jtag_header_x1.smash

This tells the ARTIX FPGA to direct the jtag header to that daughter card site.

Connect the JTAG to the header marked IPMC on the carrier card. Prior to issuing the second “power-up” command the header would be directed to the IPMC, hence the label. But if the above power-up instructions were completed successfully the correct daughter end-point should be visible from Vivado.

If an unpopulated DC site is chosen then following abort error will be printed to the screen:

image of warning screen output when powering on the FPGA DC site

In the above a path should be provided for the configuration file for SMASH. Examples of pre-existing configurations can be found under /opt/smash/etc/serenity/configuration/ however it is important to have the correct configuration for your actual hardware implementation (e.g. FPGA voltages). In addition it is possible to remove the -c config_file construction from smash.exe commands if you have already set the SMASH_DEFAULT_CONFIG environment variable.

Checking communication with the loaded firmware¶

In order to check that the firmware was loaded onto the daughter cards successfully, you can run the serenitybutler info command which will print the EMP firmware version registers for each daugther card site, along with version information for the software and the Artix.

If you were loading firmware onto the X1 site, but the X0 site is empty, the output of serenitybutler info should look like:

Software versions
  uHAL:     2.7.9
  EMP:      0.7.2
  SMASH:    0.6.0
  Serenity: 0.3.16

Artix
  FW version: 0.3.2

X0
  No daughter card present
  Power: Disabled

X1
  Power: Enabled
  FPGA: Programmed
  Framework version: 0.3.2 (design 0x42)
  Payload version:   0x12345678

Next Steps:¶

You can now proceed to understanding the EMP framework and developing an algorithm within the firmware:

EMP Framework and the walkthroughs described therein.
In addition if you are using a facility test stand then we have created a helpful cheatsheet which includes concise commands for common operations - e.g. power cycling, configuring clock synthesisers and fireflies.