Overview

SERENITY is a carrier card for the phase-2 upgrade of the CMS experiment at the LHC at CERN. It has been designed in a way which means it can be used in different capacities across many sub-detectors. It is primarily a data engine allowing high data throughput (up to 5TBps per carrier board), utilising either electrical or optical SAMTEC Firefly connectors and coupled with a flexible firmware stack (EMP) which allows rapid algorithm development independent of the underlying infrastructure or even choice of FPGA.

More details about the technical specifications can be found under short summary and getting started documentation can be found under hardware first steps. Information for connections to initially program the FPGA daughter cards can be found here.

A technical paper describing the current status of SERENITY for CMS the phase-2 upgrade was presented at TWEPP’18:

• Serenity: An ATCA prototyping platform for CMS Phase-2

SERENITY and associated components

SERENITY itself is a PCB carrier card with two daughter card sites (interposers X0/X1), a small CPU (COM Express mini) a small service FPGA (artix) based around an ATCA form factor (power, ethernet, clock, cooling) with the possibility of a micro-backplane adaptor for benchtop usage.

The SERENITY design takes advantage of a generic approach to the firm/software benefiting from the previous EMP developments as part of the MP7 trigger board for the phase-1 CMS upgrade. The following bullet points try to clarify the interconnectivity between the components:

• SERENITY hardware:

  • carrier card - the main board which when populated contains a CPU, power adaptors, ETHERNET, two daughter card sites, a service FPGA, IPMC card, various LED indicators along with other headers for programming and debugging.

  • Daughter card - the two interposers which contain the main FPGAs for data processing. As these are mezzanines these can come in different varieties and be customised according to need/use case.

  • CPU - this is a small COM Express processor running centos 7. The SSD hard-drive is located on the underside of the board. There are connections for USB3 (keyboard, mouse, exernal drive/ethernet) and displayport (switchable to VGA/HDMI). The COM Express can access multiple locations on the board and would be the normal access point to the FPGAs. There is a UART connection to the IPMC.

  • ATCA socket - the card is designed to be mounted in a full ATCA crate, however, a micro-backplane can also be provided to allow benchtop running with an external power supply (+48V) and RJ45 ethernet connection (two sockets, mirrored).

  • IPMC - a small MCU which handles slow control and some ATCA fabric (e.g. temperature monitoring and associated fan control). IPMC is part of the ATCA standard which connects to the IPMB ultimately providing an Intelligent Platform Management Interface. A specification and mechanism for providing inventory management, monitoring, logging, and control for elements of a computer system. As defined in Intelligent Platform Management Interface Specification. The IPMC also offers a backdoor method to program the artix FPGA. There is also a UART connection to the CPU.

  • Clock - this can come through the ATCA pins (or a micro-backplane SMA), an onboard crystal or an MMCX differential pair. The clock distribution is flexible and potentially complex. Functionally it is described in more detail below (“Clock Distribution”).

• Associated firm/software:

  • EMP is the main firmware development framework which allows the main FPGA infrastructure to be decoupled from algorithmic development and deployment (“payload”).

  • IPBB is an associated tool which manages firmware repositories generation of projects for synthesis and building using the Vivado tool.

  • IPMC has an associated slow control protocol and software for monitoring and controlling aspects of the board, e.g. temperature.

  • SMASH is a framework attached to the slow control and diagnostics.

Clock Distribution:

There are two reference clock inputs on the ATCA backplane (can also be applied on the SMA connectors on the micro-backplane adaptor) in addition there is an MMCX coax connector on the carrier board (differential using two MMCXs) and an internal 40MHz crystal. Each of these four clocks can be distributed to the fan out and selected in the artix or at a daughter card site. For the interposers there are two jitter correctors which then allow two reference clocks to be distributed to the FPGAs (multi-gigabit-transceivers). This then allows a choice of reference clock depending upon the requirements - for example front-end board input would be phase locked to the LHC clock, whereas the output could be an asynchronous link. A simpler way to consider the clock is that there are four input sources: 2xATCA, MMCX-coax, crystal and two inputs for each transceiver (ref-clock-0 and ref-clock-1). The distribution is carried out by the global fan-out and then the local jitter correctors for each daughter card site. The control of the clock selection is carried out by EMP control.

The LHC will provide two potential reference clocks: 40 MHz (LHC) and a higher frequency more precise 320 MHz (LHCP) cleaner one.

SERENITY Logical Connectivity

The SERENITY carrier card is designed around a flexible connectivity paradigm. This means that there are a number of ways in which to communicate with the carrier card and the attached daughter boards.

The following is a logical diagram derived from the board schematic incorporating all the connection possibilities:

A version showing it as physically represented on the board can be at the bottom.

Connectivity: Use cases

The diagram above displays the complexity in the ways in which communication can be carried out.

Initially the recommended method of communication is:

• Ethernet to the ComExpress CPU (through the ATCA backplane)

• JTAG for direct programming of the FPGA Daughter Card sites

• PCIe to “talk” to the daughter cards through the CPU (empbutler etc…)

The main in-use connetivity would be the following:

Slow Control Connectivity:

The connections for slow-control (configuration, temperature read-out, diagnostics, fan-control) and also ETHERNET communication allows connection between the ATCA, IPMC and COMExpress. Optionally it could also communicate with the ARTIX (not foreseen). A standard configuration method would involve communicating through the ATCA to the ETHERNET-switch and then routed to the target of either the IPMC or the ComExpress:

High Speed I/O:

The main IO of the board is carried out through the SAMTEC firefly transceivers (72-bi-directional optical/electrical connectors per daughter card site. In addition there are 64 uni-directional lanes between the two daughter card sites. For DAQ purposes there are a further 2 bi-directional links per interposer connected to a Quad SFP mounted next to the ComExpress at the rear (“east”) of the board:

TTC (Trigger):

The TTC and TTS allow connectivity between the ATCA backplane and the interposers via the artix service FPGA:

JTAG:

There are three main methods to program the FPGA daughter cards:

1. ATCA → ETHERNET-switch → ComExpress → PCIe → artix → JTAG → daughter-card.

2. ComExpress → PCIe → artix → JTAG → daughter-card.

3. JTAG directly → artix → JTAG → daughter-card.

The first would be an external method, the second would be loading internall from e.g. an SSD and the third is the process if connecting vivado directly to the carrier card.

For direct connection using Vivado the JTAG interface should be connected to the header marked “IPMC” on the SERENITY carrier card. This routes the JTAG information through the artix FPGA to the chosen end-point (within Vivado).

Logical-Diagram

As it is laid out on the board the connection options look like this:

edit this page