SERENITY-Z-V1.2 Technical Specifications¶
A data-stream processing platform and framework
Overview¶
Serenity is a 10Tbps multi-purpose data-processing platform which can be configured in multiple configurations with flexible I/O between topologies. It comprises three broad elements:
A generically applicable hardware platform targeting data-stream processing applications
A firmware framework which abstracts away specific details of the hardware
A general purpose software toolkit to facilitate firmware development as well as full system control and running.
The next iteration of the card (version 1.2) is currently in production.
Hardware¶
The Serenity hardware platform comprises a carrier-card (for common board services) and two sites for the mounting of FPGA-hosting daughter-card sites (for data-processing).
The carrier-card is in an ATCA form-factor providing power, clocking, optical interfaces, electrical interconnections between daughter-cards, and all management infrastructure for monitoring and controlling the platform. The daughter cards are free to simply process data.
The IPMC functionality, as required for the ATCA, is provided by the commercial PigeonPoint software, guaranteeing compliance.
Board control is provided by an industry-standard ComExpress type-10 computer-on-module, although there is provision for a custom CPU card (e.g. Zynq based). The use of a standard CPU facilitates a Linux-based operating-system, simplifying and guaranteeing long-term maintenance. Each daughter card is connected to the CPU by a PCIe Gen2/Gen3 ×1 connection.
Fig. 1 The SERENITY carrier card¶
The Serenity hardware was designed to meet the needs of several systems, each having very different requirements in terms of logic resources, cost considerations and connectivity. The chosen solution was to provide a standardized interface to a very large number of connectivity options, which could be used if required or simply ignored.
By placing the FPGAs on daughter-cards, FPGAs in any package, from any family or generation, and possibly even any vendor, can be mounted. The design of the daughter-card determines to which carrier resources the FPGA is connected, and thus tailors the generic board to a specific application. The pinouts of the two daughter-card sites are identical.
The connectivity from the carrier to the daughter-cards is provided via a factory-customized 1990 pin Samtec Z-RAY one-piece connector. A range of daughter-card designs already exist or the user can design their own if required.
Between the two daughter-card sites run 16 high-speed differential pairs whose direction is unspecified, and which may be used for unidirectional or bidirectional communications between sites if required.
Each daughter-card connects to 16 multi-purpose Samtec Firefly connectors, providing up to 192 serial links capable of operating in excess of 25Gbps. Unidirectional and bidirectional flavours of optical firefly modules and passive electrical firefly modules are commercially available. In addition, each daughter card has 8 birectional links via 2 firefly modules.
Technical Specification¶
Form-factor
Advanced Tele-Communications Architecture (ATCA) blade
Adapter for desktop operation
Processors
Two daughter card sites hosting the user’s preferred generation, family or package of FPGA
A selection of daughter-card designs with Xilinx Kintex Ultrascale, Kintex Ultrascale+ and Virtex Ultrascale+ FPGAs. Custom options also possible.
Control
Industry-standard ComExpress type-10 computer-on-module. Typically an i5-8365UE CPU, with 8/16GB of LPDDR3 RAM, although others are available.
Standard Linux operating-system
USB3, DisplayPort & UART connectors for local direct access
PCIe Gen2/3 ×1 connection to each daughter-card
Full software control and monitoring of all voltage regulators, clock-network and optical modules
Commercial IPMC implementation for ATCA operation
Inter daughter-card connectivity
16 differential pairs
Proven to 25Gbps
Directionality and occupancy is user-defined (as required)
Standard off-board connectivity
16 Samtec Firefly modules per daughter-card
Proven to 25Gbps
Copper and optical cables available (both up to 25Gbps)
Up to 192 serial links per daughter-card when fitted with unidirectional Firefly modules.
Up to 64 Tx + 64 Rx serial links per daughter-card when fitted with bidirectional Firefly modules.
Any combination of electrical, unidirectional and bidirectional optical modules supported, dependent only on the design of the daughter-card.
Additional 8 bidirectional links to each daughter card from 2 bidirectional Firefly modules.
Support for the 10G CMS Timing Control & Distribution System
Up to 10Tbps processing capability (uni-directional), or 5Tb/s bidirectional
The previous version of the card, v1.1, was very similar but with the following limitations (due to increased inter-site bus between the two daughter cards):
Inter daughter-card connectivity via 64 differential pairs
12 Samtec Firefly modules per daughter-card
Up to 144 serial links per daughter-card when fitted with unidirectional Firefly modules.
Up to 48 Tx + 48 Rx serial links per daughter-card when fitted with bidirectional Firefly modules.
5Tbps
For more details contact Geoff Hall at Imperial College London: g.hall@imperial.ac.uk
A summary paper was also presented at the TWEPP’18 conference: