There are six components in this sample EPICS package for LLRF4 board.
These are:

1. Verilog bus master for USB interfacing (verilog/fx2.v)
2. FX2 firmware for enumeration, FIFO config, and vendor request servicing
   (src/fx2_firmware).
3. USBIO library and command-line tools. The library includes features for
   downloading the firmware to the FX2, EEPROM reading/writing, and the FPGA
   high-speed interface access (src/fx2_usbio).
4. Customized xc3sprog tool for downloading JTAG bitstreams via FX2
   (src/xc3sprog).
5. EPICS softIOC (src/epics).
6. EDM panels for communicating with the softIOC (src/dl).

=== Environment and OS setup ===

For all the tools to run properly and the compiles to work, a number of
environment variables has to be set correctly. Most of these are standard
EPICS development variables (my version of epics.sh from 3.14.11 is included,
EPICS base and extensions under /opt/epics). Source shell script envSet.sh
to get the LLRF-specific variables exported.

In order to compile the tools, the following RPMS had to be installed on
freshly installed CentOS 6.3 (~= RHEL6, configured as "Software Development
Workstation"):

1. YUM repository EPEL (Extra Packages for Enterprise Linux), installed as
follows:
sudo yum install http://mirrors.servercentral.net/fedora/epel/6/i386/epel-release-6-7.noarch.rpm

2. Packages

sdcc		Small device C compiler, FX2 firmware
libusb-devel	USB I/O for FX2, FTDI devices
libusb-static	Static library
libftdi		FTDI library needed to compile xc3sprog
libftdi-devel	Development files for libftdi
fftw		FFT library
fftw-devel	Development files for fftw
fftw-static	Static library

You need, of course, pre-compiled EPICS base to compile the IOC.

On the client machine EDM is needed to bring up the display panels.

=== Verilog code ===

The critical component of the host-FPGA interface chain is the fx2.v FPGA
bus master. Together with the FX2 firmware and the USBIO library it
implements a memory-mapped bus abstraction within the FPGA. That is, USBIO
library includes functions for read and write accesses to internal locations
in the FPGA that have 32-bit address and data. The library converts these to
appropriate USB endpoint requests which are then received by fx2.v from the
slave FIFO interface. Logic in fx2.v reconstructs and executes the bus
transactions.

The local bus interface of fx2.v includes the following signals:

// Local interface
output [31:0] data_in,	// Local data in, from USB!!!
input [31:0]  data_out,	// Local data out, to USB!!!
output 	      strobe,	// Local bus strobe
output 	      rd,	// Local bus read enable
output [31:0] addr,	// Local bus address
input         ack,	// Local bus ack

Bus transactions are synchronous to the local 48 MHz clock, received from
FX2. Thus, even if the signal processing clock is absent, the local bus
functions correctly. A transaction is initiated by the bus master by
asserting strobe output. State of the rd signal defines a read or write
transaction (1 - read). The master sets 32-bit addr output to the desired
access location. On writes, data from the host (coming from the USB) is
presented on data_in. On reads, the master clocks in the data on data_out
and sends it to the host.

Bus transactions are moderated by the ack signal, monitored by the fx2 bus
master. After strobe is asserted, bus master waits for the ack signal.
Strobe is reset on the next bus clock edge. Fastest transaction rate is two
clock cycles, if ack is combinatorially derived from strobe signal. It is
certainly possible to optimize this protocol, but at 48 MHz bus clock the
maximum transfer rate is 768 Mbps, above the USB high-speed bandwidth.

Local bus structure uses two more standard modules: address decoder
(addrdec.v) and the bus multiplexer (busmux.v). The decoder is self
explanatory. The multiplexer receives data outputs and ack signals from
multiple bus slaves and multiplexes them appropriately as defined by CE
vector (generated by address decoder). Two special conditions conditions are
of interest: no address decoded (CE==32'b0) and multiple addresses decoded.
In the first case the mux routes strobe back as ack and asserts 32'hdeadbeef
on the bus. In the second case it produces 32'hbad0bad and sets ack to 0,
effectively hanging the bus. The master has no timeout on local bus. FX2 can
drive FPGA reset line to abort the hanged cycle (use "usbio_utils if_reset"
command). Since such collision should not happen in a properly working
system, I view this as a feature, not a bug.

A demo application in ./verilog includes DCM control, AD9512 interface, ADC
waveform acquisition, DAC drive using DDS and RAM-based waveform generators,
ADC signal downconversion, decimation and readout, ADC and DAC power
control, drivers for AD5644 DACs, MCP3208 ADC, DS1822 temperature sensor. 
Second-order IIR signal processing is implemented at the full ADC rate. 
Code for ADC channel RMS monitoring is also included, but is disabled in
this version (XC3S1000 has 24 multipliers: 4 per DDC = 16, 5 in IIR, 2 in
DDS interpolator = 23, leaving us 3 short for RMS). Xilinx ISE project is
in ./xil.  A ready-to-go bit file is included (./xil/llrf.bit).

=== FX2 firmware ===

FX2 firmware compiles to an Intel HEX file (LLRF4.ihx). It can be
loaded by usbio_utils command line tool (or other FX2 firmware loaders).
Firmware handles renumeration, slave FIFO and endpoint configuration and
processes service requests for:

a) JTAG access to the FPGA
b) EEPROM reading and writing
c) FPGA reset line (should not be needed in normal operation)
d) Reading out USB error counter within FX2

Firmware compiles with no warnings with SDCC (Fedora 17,
sdcc-3.0.0-2.fc17.x86_64 RPM).

-- USB IDs --
Vendor/product/device IDs for LLRF4 is defined in src/include/llrf41_usb.h
and are chosen as 0xfffe:0x0020:0x0101. Before firmware load device ID
should be 0x0001, so that udev can automatically process firmware download.

Copy files misc/fx2.rules and misc/llrf4.rules to /etc/udev/rules.d, so that
LLRF4 USB device is automatically made world-readable and writable on
connection.

=== USBIO library and tools ===

-- Library --

The main goal of the USBIO library is to present memory-mapped abstraction
of the FPGA local bus to the Linux program (command line, EPICS, etc.).

The interface is initialized with usbio_open() and released with
usbio_close(). The single argument to usbio_open() right now is a bit flag
that selects whether FPGA interface is initialized or not. With flag set to
0, normal FPGA interfacing mode is configured. With flag set to 1, FX2 is
not expected to run the proper firmware. This mode is used by command-line
tools to load the firmware, read FX2 memory, control the 8051 run enable,
etc. - all service tasks.

Three functions are used for all FPGA local bus interfacing:

usbio_block_write()
usbio_block_read()
usbio_mult_write()

The first two functions are quite straightforward - they write or read a
block of data from some start address. Data sizes can range from 1 word to
2^32-1.

The third function allows to write to an arbitrary list of addresses. It is
currently implemented as a sequence of calls to usbio_block_write(), but
should be optimized (some work on fx2.v is needed) to send the data in a
series of large packets. Due to the nature of the USB interface, short
transactions are quite slow. Single word reads/writes reach only 60 KiB/s,
while on larger blocks (8kWords) the rate rises to 20 MiB/s. The rate maxes
out at 35 MiB/s for reads and 27 MiB/s for writes.

-- Tools --

A number of simple, but useful command-line tools are included with USBIO.
Here are short descriptions of these tools:

usbr addr
	Read a single address, print out the result.

usbw addr val
	Write to a single location

usbrblk addr length
	Read a block, dump on stdout. Transfer rate stats printed to stderr

usbwblk addr length
	Get data from stdin, write to the FPGA. Transfer rate stats printed
	to stderr

memtest addr length count [width]
	Perform memory testing. Writes a block of memory with random data,
	reads it back and verifies. Argument count specifies the number of
	test cycles to perform. Last optional argument can be used to
	specify the width of memory to test, useful with blocks narrower
	than 32 bits.

usbio_utils
	Collection of service utilities. Valid commands are:
	i2c_read addr length
		Read length bytes from I2C device at address addr. For
                EEPROM use address 0x51. Address within the EEPROM should be
		first written to the device using i2c_write_addr
	i2c_write addr data
		Write data to I2C device. Data argument is a string of hex
		nibbles. First 16 bits define access address for EEPROMs.
	if_reset
		Pulse reset line from FX2 to the FPGA.
	usberr_read
		Read FX2 USB error counter. Counter is reset on read. First
		read after firmware load always seems to return 2.
	8051reset value
		Control 8051 in FX2. Value of 1 halts the device, writing 0
		restarts execution.
	fw_load filename
		Loads firmware in filename (Intel HEX) to FX2
	ram_read addr length
		Reads 8051 memory. Processor should be placed in reset using
		8051reset command for this to return meaningful data.
	write_serial serial_number
		Writes board serial number string to EEPROM.

-- EEPROM control --

EEPROM is used for storing the VID/PID/DID values, presented by the board
upon power-up. It can also store the unique board serial number. Here is a
readout of the EEPROM on one of the boards:

0000 c0 fe ff 20 00 01 00 00 ff ff ff ff ff ff ff ff
0010 5a fe 0a 4c 4c 52 46 34 5f 30 30 32 36 ff ff ff

First location of the EEPROM must be set to 0xc0.  Locations 1 through 7
store the VID/PID/DID information (0xfffe:0x0020:0x0001).  These can be
written to the board as follows:

usbio_utils i2c_write 0x51 0000c0feff2000010000

Starting at location 16 the serial number string is stored. It starts with
the header (0x5a 0xfe), then there is character count, next - ASCII string
(LLRF4_0026). This is written to the EEPROM using

usbio_utils write_serial LLRF4_0026

When the firmware is loaded to the board, it reads out the serial number and
reports it as iSerial.

=== xc3sprog ===

xc3sprog is a free set of tools for programming Xilinx FPGAs and EEPROMS via
JTAG (http://sourceforge.net/projects/xc3sprog/). A custom version of
xc3sprog is included - modifications add a new cable driver (fx2) to the two
existing ones (pp, ftdi). To load the bit file, execute:

xc3sprog -c fx2 -v bitfile

Verbose option is not strictly necessary :)

Note: xc3sprog should really be installed in a system-wide location, since it
needs to access devlist.txt file. Default prefix is /usr/local, editable in
the Makefile.

=== EPICS softIOC ===

The IOC has the standard layout. Main directories of interest are
LLRFapp/src for subroutine source code in C and LLRFapp/Db for the
databases.

All databases are template based, with at least two macros: SYS and DEV. all
channel names start from $(SYS):$(DEV):, so to change the IOC name one just
needs to run the substitutions on the template files. The overall database
is distributed over several files for modularity and ease of editing. One of
the most important databases is the regs.db - it defines the uniform
structure for all individual register accesses.

-- regs.db --

Register database contains three records: two subroutines and ao (analog
out). Subroutine $(SYS):$(DEV):$(NAME):SUBWR is used for writing to the
register. Input A is linked to the data source, input B - register address.
Flag at input D controls readback behavior. If the flag is zero, the
subroutine reads the register after writing. Readback value is available at
link C, while the subroutine value is set to 0 on success, 1 if the readback
differs from the write value. Normally, data source for this subroutine
comes from $(SYS):$(DEV):$(NAME), however it can be routed in the template
to a calculate record to create bit masks, etc. $(SYS):$(DEV):$(NAME):POLL
record is used for reading the register. Analog out is used as a data source,
since for "longout" record type EPICS does not understand unsigned 32-bit
integers, only signed.

Let's take a look at the regs.template file that defines the actual register
instances:

file db/regs.db
{
pattern
{SYS  DEV  NAME     SRC      SCAN    FLAG OFF  DOL    DRVL  DRVH       }
{LLRF TEST CR0      CALC:CR0 Passive 0    0    0      0     4294967295 }
{LLRF TEST PHDCM0   PHDCM0   Passive 0    2    255    128   384        }
{LLRF TEST CR3      CALC:CR3 Passive 0    3    1      1     4294967295 }
}

There are three registers defined, at addresses 0, 2, and 3. Register
LLRF:TEST:PHDCM0 has limits from 128 to 384, default value 255. Two control
registers LLRF:TEST:CR0 and LLRF:TEST:CR3 are driven by calculation records
in calcs.db

=== EDM panels ===

A set of panels, controlling and reading back all devices in the demo
application is provided in src/dl. Properly compiled and configured EDM
installation is needed to run these panels. Color mapping file colors.list
is included as well.
