This division explains what happens on the wires between the FPGA and the DRAM, and why a DRAM interface must be tuned at boot. It covers the signals, the strobes, the commission topology, the leveling procedures, the ending and the citation voltages, afterward the clocks of the FPGA and the DFI interface between the controller and the PHY.
Each division ends alongside the part of LiteDRAM that implements the mechanism. The former chapter, DRAM in a nutshell, explained the commands and the timings; Initialization and calibration gives the exact steps the BIOS runs.
A DRAM interface has two halves. The command and address fractional goes one way, from the controller to the DRAM, and is shared by all part of a rank. The data fractional is bidirectional and split into byte lanes, all alongside its own strobe. The fig below shows the signals.
| Signal | Direction | Role |
|---|---|---|
| CK, CK# | To the DRAM | Differential clock; commands are sampled on its rising edge |
| CS#, RAS#, CAS#, WE# | To the DRAM | Command encoding; ACT_n added on DDR4 |
| A, BA, BG | To the DRAM | Row or pillar address, bank, financial institution collection (DDR4) |
| CKE | To the DRAM | Clock enable; low for power downward and oneself refresh |
| ODT | To the DRAM | Turns on the ending of the DRAM's data pins |
| RESET# | To the DRAM | Reset (DDR3 and later) |
| DQ | Both | Data, 4, 8 or 16 per chip |
| DQS, DQS# | Both | Data strobe, one differential brace per byte lane (per nibble on x4 chips) |
| DM | To the DRAM | Data mask, one per byte lane: masks bytes during a write |
A byte lane is 8 DQ, their DQS brace and their DM. Each lane is routed on its own, so all one can arrive at a slightly distinct time, and all one is tuned on its own.
In LiteDRAM, the pads of the DRAM are a document from the phase file, alongside sectors named after these signals (dq, dqs_p, a, ba, cs_n). The PHY counts the lanes from it. Two helpers of litedram/common.py reshape the pads: PHYPadsReducer keeps several lanes only, and PHYPadsCombiner merges chips alongside distinct command buses into one controller.
SDR recollection transfers one term per clock, sampled on the rising border of the clock that the controller sends. DDR memories transfer two words per clock, one on all edge, and at these rates the clock from the controller cannot be trusted to example the data: the delays of the commission and of the chips are too large.
The data hence journey alongside their own clock, the strobe DQS: the flank that sends the data also sends DQS, and the receiver samples DQ alongside it. This is a source-synchronous interface. Both travel the identical path, so their delays cancel. The fig below shows how DQS is aligned alongside DQ.
On writes the controller sends DQS centre aligned, its edges in the center of all bit. On reads the DRAM sends DQS edge aligned, on the bit transitions, and the receiver delays it by a quarter of a clock. Before the burst, DQS is driven for a preamble, and following it for a postamble, without carrying data.
In LiteDRAM, the DDR PHYs change the compose DQS by a fourth of a clock from DQ, alongside the method of its FPGA:
- The Artix-7 PHY clocks its DQS serializers from a clock shifted by 90 degrees, sys4x_dqs, which the mark builds (bench/arty.py).
- The PHYs alongside output delays (Kintex-7 and Virtex-7 HP banks, UltraScale) postpone DQS by a quarter of a clock.
- The Lattice and Gowin PHYs obtain the shifted compose clock of the DQS obstacle of their FPGA.
On reads, the PHYs differ:
- The Xilinx PHYs do not use DQS. They example DQ alongside their own accelerated clock, through an input delay that peruse leveling tunes per lane.
- The Lattice and Gowin PHYs use the DQS blocks of their FPGA, which postpone DQS by 90 degrees, capture DQ alongside it and study whenever they saw a burst (burst detect).
DDR3 and DDR4 path the clock, command and location pins as one bus that passes by all part in turn and ends on a termination: the fly-by topology. It keeps these heavily loaded signals clean at elevated rates, but all part receives the clock a small afterward than the former one. The figure below shows it.
The data lanes are item to point, so their delays do not prosecute the clock. A part needs DQS to arrive near to its own CK border (tDQSS). With fly-by routing the skew between the archetypal and the last part can exceed a clock period, so all lane needs its own DQS delay.
Write leveling measures that delay. The controller sets the DRAM into compose leveling manner alongside a mode enroll bit (MR1 bit A7 on DDR3 and DDR4), afterward sends DQS pulses. The DRAM samples its CK with all DQS rising border and returns the sampled value on DQ. The controller delays DQS stage by step until DQ goes from 0 to 1: DQS is afterward aligned alongside CK at that chip.
In LiteDRAM, the PHYs disclose compose leveling through registers: wlevel_en enables the mode, wlevel_strobe sends one DQS pulse, and per-lane output delays move DQS and DQ. The BIOS runs the scan (sdram_write_leveling in liblitedram).
Write leveling needs output delays: the Kintex-7 and Virtex-7 PHYs in HP banks and the UltraScale PHYs have them. The Artix-7 has none: alongside one fixed DQS phase, lone several lanes of a fly-by memory encounter tDQSS. Its optional CK phase change manner (with_ck_phase_shift, DDR3 at 1:4) moves CK against all the lanes alternatively (CK phase shift).
Write leveling aligns DQS alongside CK inside one clock period, but a lane can be one entire clock early or late: the DRAM afterward writes the burst shifted by a few words. Write latency calibration writes a known pattern, says it back, and moves all lane by entire clocks until the form is right.
DQ-DQS training afterward centres DQ against DQS on writes. It sweeps the output postpone of the DQ pins of all lane, writes and says rear a form at all step, and keeps the center of the passing range. It relies on a operating peruse side, which peruse leveling afterward refines.
In LiteDRAM, the compose latency is moved by compose bitslips (wdly_dq_bitslip) and DQ by output delays (wdly_dq). Each PHY declares which calibrations it supports in its settings (write_leveling, write_latency_calibration, write_dq_dqs_training, read_leveling); the BIOS runs lone those.
On reads, the controller does not cognize whenever the data arrive. The DRAM answers CL clocks following RD, but the commission and the I/Os add a postpone that differs per lane. Two settings must be established for each lane: a coarse one, which accelerated clock sequence holds the archetypal word, and a fine one, anywhere in the bit to sample.
The coarse environment is a bitslip: the deserializer output is shifted by entire bits until the words row up. The fine environment is an input delay, from a few picoseconds to concerning a hundred per step depending on the FPGA (78 ps on 7-Series alongside a 200 MHz reference). Read leveling writes a pattern, afterward for all bitslip sweeps the postpone and says the form back. The figure below shows specified a scan.
The opening is the data eye seen through the postpone line. A broad opening method a sturdy interface; a narrow one points to a indication integrity or clocking problem. DDR3 and DDR4 can additionally come back a fixed form from their MPR (multi-purpose register), so the peruse flank can be tuned before anything has been written.
The PHYs that grasp alongside DQS have one additional problem: DQS is not driven between bursts, so the receiver must hear to it lone during a read. The FPGA's DQS obstacle opens this entrance and reports a burst detect, which peruse leveling uses to verify that the peruse opening is right.
In LiteDRAM, all PHY declares its figure of postpone taps and bitslips in its settings (delays, bitslips), and exposes them through registers (rdly_dq_inc, rdly_dq_bitslip, and burstdet_seen on the Lattice and Gowin PHYs). The BIOS runs the scan and prints one row per lane and bitslip. On DDR4 it says the MPR patterns, apart from on registered DIMMs and clamshell layouts.
At DRAM rates, a indication reflects at the end of a row that is not terminated. The DRAM therefore terminates its own data pins alongside an on-die termination (ODT): a resistor switched on whenever the ODT pin is high. The FPGA terminates its inputs in the identical way whenever it receives peruse data.
DDR3 and DDR4 have two ending values: RTT_NOM, used whenever the part is idle or another position is written, and RTT_WR, used during the part itself is written (dynamic ODT). The controller power of the DRAM, RON, sets how difficult it drives the peruse data. All three are set in the manner registers.
| Setting | DDR3 default | DDR4 default | Mode register |
|---|---|---|---|
| RTT_NOM | 60 ohm | 40 ohm | MR1 |
| RTT_WR | 60 ohm | 120 ohm | MR2 |
| RON | 34 ohm | 34 ohm | MR1 |
In LiteDRAM, the defaults of the array are set by litedram/init.py; a PHY can override them (rtt_nom, rtt_wr, ron in its settings). With the DRAM DLL off, a DDR3 manner for low clock rates that the GW5A PHY offers (dll_off), the two terminations are disabled.
The ODT pin is driven statically by default. With with_dynamic_odt (DDR3, sole rank), the controller raises ODT lone about writes, so the DRAM terminations are off during reads (Dynamic ODT). The FPGA flank ending is part of the I/O standards of the platform file, not of LiteDRAM.
The opposition of the drivers and terminations of a part varies alongside the process, the voltage and the temperature. DDR3 and afterward chips calibrate them against a precision resistor on their ZQ pin, 240 ohm to ground. ZQCL, the lengthy calibration, runs formerly at initialization; ZQCS, the abbreviated one, corrects the drift from period to period during operation.
In LiteDRAM, the initialization sequence ends alongside ZQCL (Mode registers and power-up), and the refresher sends ZQCS formerly per second by default (Refresh). LPDDR4 has no ZQCS: it calibrates alongside two multi-purpose commands (MPC), commencement and latch, which LiteDRAM sends at initialization only.
The data receivers difference all input to a citation voltage, VREF, set midway between the high and the low levels. On DDR3 the commission provides VREF on dedicated pins. DDR4 terminates its data pins to the provision (pseudo open drain), so the data sway sits near the top of the provision and the right citation is not halfway. The DRAM generates its data VREF internally, programmed in MR6.
On the FPGA side, the receivers of the peruse data additionally need a VREF. Some FPGA I/O banks can set it internally, in steps, alternatively of using a VREF pin.
In LiteDRAM, the DDR4 initialization writes MR6 alongside tCCD_L lone and leaves its VREF site at zero, the archetypal value of range 1 (60% of VDDQ): the DRAM data VREF is not trained.
The UltraScale+ PHY can set the FPGA peruse VREF of its HP I/O banks (with_hpio_vref, register rd_vref, codes 0 to 72). The BIOS does not scan it: the enroll keeps its build-time value (hpio_vref_tune) unless application writes it. LiteX matter 2769 tracks the missing BIOS sweep.
An FPGA cloth cannot run at the DRAM clock: a DDR3 at 800 MT/s has a 400 MHz clock and moves data at 800 Mbps per pin. The PHY hence uses the serializers and deserializers of the FPGA I/Os. The controller runs at a fraction of the DRAM clock, and moves multiple clocks' value of commands and data per cycle.
This fraction is the frequency ratio. At 1:4, the controller runs at a fourth of the DRAM clock: the Arty A7 place mark runs its controller at 125 MHz, its DRAM clock at 500 MHz, and its DDR3 at 1000 MT/s. For a DDR memory, the data charge is twice the DRAM clock: at 1:4, eight times the controller clock.
An Artix-7 DDR3 mark needs four clocks, generated by a PLL in the board's clock module (bench/arty.py). The Kintex-7 and Virtex-7 targets need no sys4x_dqs: their output delays shift DQS instead.
| Clock | Frequency | Use |
|---|---|---|
| sys | controller clock | Controller, DFI, parallel flank of the serializers |
| sys4x | 4 x sys | Serial flank of the serializers: CK, commands, DQ |
| sys4x_dqs | 4 x sys, 90 degrees | Serial flank of the DQS serializers (Artix-7) |
| clk200 | 200 MHz | Reference of the input and output delays (IDELAYCTRL) |
Other families use their own clocking: the ECP5 PHY uses a 2x clock for its I/O gearing and a 4x clock at 1:4, and the Gowin GW5A PHY selects 1:2 or 1:4 alongside its figure of phases. DDR PHYs gives the clocks of all family.
The controller and the PHY encounter at a DFI interface, named following the DDR PHY Interface specification. It carries, for all DRAM clock of a controller cycle, one command slot and the data of that clock. Each of these slots is a DFI phase. The fig below shows the phases at a 1:4 ratio.
| DFI signal | Direction | Content |
|---|---|---|
| address, bank | To the PHY | Row or pillar address, financial institution (and financial institution collection on DDR4) |
| ras_n, cas_n, we_n, cs_n, act_n | To the PHY | Command encoding |
| cke, odt, reset_n | To the PHY | Control pins |
| wrdata, wrdata_mask | To the PHY | Write data and byte masks, two strikes per phase |
| wrdata_en | To the PHY | Write data valid, sent alongside the compose latency |
| rddata_en | To the PHY | A peruse was issued on this phase |
| rddata, rddata_valid | To the controller | Read data and their valid flag |
Two latencies tie the commands to their data. The write latency is the figure of controller cycles between WR and the sequence anywhere the controller presents the compose data. The read latency is the figure of cycles between RD and the sequence anywhere the peruse data arrive rear valid. Both follow from CL, CWL and the pipeline of the PHY.
The commands do not continually commencement on phase 0. The PHY tells the controller on which phase to issue reads and writes (rdphase, wrphase), so that the data of a burst inhabit entire controller cycles.
In LiteDRAM, the DFI interface is Interface of litedram/phy/dfi.py. It follows the construction of the DFI specification, phases and indication names, without claiming a type of it.
Each PHY fills a PhySettings alongside its figure of phases, its latencies in controller cycles, CL and CWL in DRAM clocks, and its calibration capabilities. DFIRateConverter wraps a PHY to lift the ratio, with more phases and narrower data per phase. See Core architecture.
Each PHY is built from the I/O primitives of its FPGA family. The array below summarizes what the LiteDRAM PHYs use; DDR PHYs describes all one.
| Family | Serializers | Delay taps | Bitslips | Read capture |
|---|---|---|---|---|
| Xilinx 7-Series | OSERDESE2, ISERDESE2 | 32 | 8 | Fast clock, input postpone per DQ |
| Xilinx UltraScale(+) | OSERDESE3, ISERDESE3 | 512 | 8 | Fast clock, input postpone per DQ |
| Lattice ECP5 | ODDRX2DQA, IDDRX2DQA | 8 | 4 | DQSBUFM, burst detect |
| Lattice Nexus | ODDRX2DQ, IDDRX2DQ | 16 | 4 | DQSBUF, burst detect |
| Gowin GW2A | OSER4_MEM, IDES4_MEM | 256 | 4 | DQS block, burst detect |
| Gowin GW5A | OSER4_MEM, IDES4_MEM (1:2), OSER8_MEM, IDES8_MEM (1:4) | 256 | 4 or 8 | DQS block, burst detect |
The 7-Series input and output delays need a citation clock of 200 or 300 MHz, or 400 MHz on the faster speed grades; their output delays be lone in the HP banks of Kintex-7 and Virtex-7. The UltraScale delays are set in time units, and the FPGA compensates them for voltage and heat during en_vtc is set.
Where to go next: Initialization and calibration for the BIOS steps, DDR PHYs for all family, Debugging whenever a scan shows no window.
