Red Pitaya FPGA Project 5 – High-Bandwidth Averager
Introduction
Up to now we discussed simple projects that demonstrated basic ideas behind FPGA programming. In this post we will go a step further and create a data acquisition system. In particular, we will implement a High-Bandwidth Averager, which will acquire a sequence of measurements at the highest sampling rate (125 Msamples/s) and on each trigger accumulate these measurements on the on-chip Block Random Access Memory (BRAM). Our approach can be extended and ultimately used for ultra-sensitive homodyne or heterodyne detection in radio systems, radars, nuclear magnetic resonance (NMR) or even superconducting quantum circuits.
FPGA acquisition system will require more comprehensive post-processing options such as data analysis, visualization and storage. For this reason we will take Pavel Demin‘s approach and create a client program, running on any computer in the network, and a server program, running on the Red Pitaya’s Linux. Client will set configuration, start measurements, and download data from the server via a TCP/IP protocol.
This guide is split into to two parts, in the first part we will describe FPGA logic behind the Averager and in the second part we will demonstrate how to setup server on Red Pitaya’s Linux and a python client on a local computer. The purpose of this project is to learn how to store and read-out acquired data from the FPGA programmable logic.
Building the Project
To start off download the project from github as described in previous posts. Then build custom IP cores by navigating to /redpitaya_guide base folder in Vivado’s tcl console and execute
source make_cores.tcl
This will create the following IP cores in the folder tmp/cores needed for Averager project: axis_red_pitaya_adc_v1_0, axis_red_pitaya_dac_v1_0, bram_switch, axi_bram_reader and axis_averager_v1_0. The first two are taken from Pavel Demin, as discussed in the previous post, the last three are custom made for accumulation and read-out of measured data.
When custom cores are successfully generated, build the project by appropriately modifying the make_project.tcl script and execute it in the Vivado’s tcl console as described in Project 1
source make_project.tcl
FPGA part
The general block design of the Averager is composed of five main parts: processing system, configuration, data acquisition, triggering and averaging as shown in the figure below. We have also added a signal generator part for testing the data acquisition system. The signal generator creates a sine tone on both Red Pitayas’ DAC output ports with frequency 125 MHz/32 = 3.90625 MHz. This design is based on block design from Project 4, where processing system, data acquisition and signal generator hierarchies are described in greater detail.
Block Design Overview
Averager Hierarchy
High-Bandwidth Averager collects a number of consecutive measurements from the ADC block, reads the same number of values stored in the BRAM memory and writes the sum of collected and read values back to the memory. This is repeated on every trigger signal. Each memory location, therefore, contains a sum of ADC values always measured at the specific time after the trigger.
Acquisition begins with a release of a reset signal. At this points the Averager sets BRAM memory locations to zero and awaits for a trigger. On each trigger the Averager accumulates NSAMPLES number of ADC values, adds them to the previous measurements and writes the sums back to the memory. After NAVERAGES number of triggers the Averager stops and asserts the finished signal.
The finished signal has two functions: first it lets the program running on Red Pitaya’s Linux know that acquisition is completed and results can be read. The second function is to re-route the BRAM PORTA line of the Block Memory Generator from Averager to AXI BRAM Reader block. This gives a program running on the processing system/Linux access for reading BRAM memory.
Averager Hierarchy
The key feature of our averaging algorithm is its capability to read and write data to the BRAM in a single clock cycle. Averager is ready for the next trigger on the second clock cycle after the previous measurement sequence is finished. This reduces the dead time during averaging and with that the overall duration of data acquisition. The High-Bandwidth Averager can measure up to approx. 65k, 32-bit wide ADC values set by the 2.1 Mb size of the on-chip BRAM memory. However, the FPGA implementation might suffer from timing issues when using all of the resources. If we need more measurement storage we can use Red Pitaya’s on-board 512 MB DDR memory. We will show how to do that in future projects. Reading and writing in a single clock cycle requires Averager to use both read (PORTB) and write (PORTA) ports of the True Dual-Port Block Memory Generator and a special BRAM Switch block to share the BRAM Generator’s reading port between the Averager and the AXI BRAM Reader. The later is connected to the Processing System via the AXI interface.
Implementation detailsTo view the Averager’s Verilog code right-click on the block and select Edit in IP Packager. The averaging is coded as a state machine with 5 states. The 0th state clears the registers, the 1st state prepares registers for writing zeros to the BRAM memory, the 2nd state performs the zero writing, the 3rd state prepares addresses for reading and writing and the 4th state performs reading and writing to the BRAM generator. Once the number of averages is met the system goes and stops in the 5th state.
Configuration GPIO Hierarchy
In the previous project we learned how to read and set registers on the FPGA logic. We will use the same approach here for setting configuration values such as number of samples, number of averages and trigger rate. We will use the second GPIO port as an input to indicate the completion of the averaging. 32 bits of the output port contain four 8-bit configuration values: log2NAVERAGES, log2NSAMPLES, trigger_select and control:
gpio_out1[31:0] = 31[{log2NAVERAGES} {log2NSAMPLES} {trigger_select} {control}]0
gpio_in2[31:0] = 31[{31-bit empty} {1-bit finished}]0
For now only the LSB of control value is used as a user_reset bit. When this bit becomes asserted averaging resets and when it is disserted averaging starts with the next trigger signal.
If you ever need more configuration output bits you can use Pavel Demin’s axi_configuration_v1_0 IP core with a custom number of bits in a single output port. axi_configuration_v1_0 can be found in redpitaya_guide/core folder and is automatically created when running make_cores.tcl script.
GPIO Hierarchy
Trigger Hierarchy
Trigger hierarchy creates a low frequency clock, which is used as a trigger for the Averager. The principle is very similar to the one in the LED blink project where we used an input clock to trigger a binary counter and select a specific bit of the counter’s 32-bit output. Here we implement this principle in a custom RTL module. The selected bit is specified by the trigger_select configuration value, which we extract from gpio1_io_o[15:8] port using Slice core.
Trigger Hierarchy
Trigger rate after the selector module is calculated as
trigger_rate = f0 / 2^(26 - trigger_select + 1),
where f0 = 125 MHz is the ADC clock frequency, +26 is an offset set in the selector module and +1 comes from the fact that binary counter is increased by one on every clock cycle, which makes the frequency of its first LSB equal to f0/2. This is a very simple trigger that might not yield the best performance, but is sufficient to demonstrate the basic idea behind the Averager. Since our trigger rate is commensurate with DAC signal frequency the Averager will always start a measurement at the same phase of the signal. This will result in full averaged signal even for millions of averages. If trigger rate would be incommensurate relative to f0, the average would converge to zero.
A more realistic design would include an external trigger. This could be brought into the programmable logic using exp_# port.
Software part
Server
To setup a server on Red Pitaya’ Linux, copy server.c file from the project folder 5_averager/server to the folder on the Linux, using for example WinSCP. With Putty compile and run the server by typing
gcc -o server server.c ./server
Server is listening on port 1001 for incoming connections. After a connection is established client can send parameters, nsamples, naverages and trigger_select, and at the end request a measurement start. Parameters are sent as 32-bit values, where 4 MSB represent parameter identification number and 28 LSB the parameter value.
After start_measurement is received server releases the user_restart bit and waits for the finished signal from FPGA. When the measurement is finished server reads the data from BRAM memory and sends it to the client.
Client
Client program is written in Python 2.7 and uses Qt Graphical User Interface (GUI). If you don’t already have python on your computer, I recommend installing Python(X,Y) for windows users. This collection includes all the required packages including Qt Designer and Spyder (IDE).
averager.py and averager.ui files can be found in the project directory 5_averager/client. I would suggest looking into the GUI file averager.ui by opening it with the Qt Editor and checking the element’s properties. Also reading python code averager.py using Spyder or any other text editor might be interesting.
My preferred way of running the client is by executing the following line in IPython (Qt) enhanced console
run averager.py
Averager’s GUI should be intuitive. Enter Red Pitaya’s IP address and press Start button. If you didn’t connect Red Pitaya’s OUT1 port to the IN1 port you will see a noisy signal. After connecting the two ports and pressing the start button a sinusoidal signal will appear in the graph section of the GUI as shown in the left screenshot below.
Python Client Program
You can play with different parameters. I recommend setting trigger rate to f0/1024, nsamples to 1024 and naverages to 262114. The measurement should take approximately 4 s. It will be hard to detect differences between highly averaged measurements in a time domain. To see how noise reduces when increasing naverages plot FFT in log scale by checking FFT and Log Scale boxes and choose Abs value.
Scale check box re-scales the 14-bit ADC values into an appropriate voltage range. The scaling values depend on your jumper setting on the Red Pitaya board and slowly change with time. Since my scaling parameters might differ from yours, you can calibrate your Red Pitaya with known voltage sources and modify the scaling coefficients hard-coded in the averager.py file.
Conclusions
In this guide we have demonstrated how to efficiently write to the BRAM memory and read-out values at the end of the measurement. One could use Averager as an oscilloscope with a single average. However, to exploit all its capabilities one might need to extend it with an external trigger option or add a module that would detect signal transitions and issue automatic triggers to the Averager. In any case, I hope this project gives you enough material to further pursue your own FPGA ideas.
| << Red Pitaya Project 4 – Frequency Counter |
References
- Reference on RTL module interface Xilinx User Guide – Designing IP Subsystems Using IP Integrator.
- Zynq7000 datasheet (Part number on Red Pitaya: XC7Z010).
26 Comments
Hi Anton, how are you?
I’m facing some problmes in my project, i’ll be glad if you can give me some help.
I’m trying to build a simple ADC project. I have a signal generator that will generate the input data.
In my Block Design, i only have the blocks: PS7, DataAcquisition, AXI4-Stream Averager and Block Memory Generator.
I’m trying to modify the AXI4-Stream Averager to just save the input data into memory (by the Block Memory Generator), but i have the following doubt:
Where can i set the address to the data to be saved? Looking the Verilog code of block AXI4-Stream Averager, there is no place for that. I just want to start the system and save some data. I’m not using any trigger, maybe in the future i use a simple trigger set by the user by the GPIO, i dont know yet. But for now i just want to get something working.
The “nsamples” i set for a fixed number (in Verilog code), like 1024 samples. And the “naverages” i dont use. The “PORTA” of the AXI4-Stream Averager block, is connected to the PORTA of the Block Memory Generator, the PORTB of Averager and Block Memory arent used.
Am i in the right direction by excluding bram_reader and bram_switch and only using the Block Memory Generator and a modified AXI4-Stream Averager?
I’m trying to build this very simple design for weeks, but i couldn’t get it working yet.
Thanks in advance.
Hi Bruno,
in the AXI4-Stream Averager IP the address for writing is specified in the register int_addrA_reg. Here we simply increment this register at each clock cycle.
I don’t fully understand why would you remove the bram_reader core. How do you plan to access your data? If your custom block is not reading stored data, you don’t need a switch, but you still need both ports of the Block Memory Generator.
Anton
Ok, just to clarify some points, i looked again at the Verilog code of AXI4-Stream Averager IP. Here are some more doubts i had.
1) We know that, PORTA of Block Memory Generator is used to write data, and PORTB is used to read data. Why do you use (as you said in the last reply) the register “int_addrB_reg” to set the writing address? Shouldnt it be the register “int_addrA_reg”?
2) In the state 0 of the state machine you’ve created, it clears the registers, so the registers “int_addrB_reg” and “int_addrA_reg” are set to zero. Does that mean that the first memory address to receive the data is going to be the 0x0000_0000? And in the state 3, where you increment this register, does that mean that second data will be saved at 0x0000_0001, and so on?
3)In the state 2, why do you set the register “int_addrA_next” to -2?
4) Why do you access the data at the address 0x4000_0000 in your C code (server.c)? Shouldnt it be accessed by the address 0x00000000,0x00000001,…?
Thanks for clarifying about the bram_reader core!! I realy need this IP to read the data, i’m using it in my design now.
Bruno
Hi,
1) sorry for the confusion, you are right, the address for writing should be indeed stored in the int_addrA_reg. I corrected my previous reply.
2) Yes, first data is stored at address 0, second at address 1, etc.
3) If you look at the simulation results (projects/5_averager/sim/Sim2_a.png) you can see that this results in the correct behavior, so that after next increment the address is 0.
4) Each AXI core connected to the processing system needs to have its own address offset. This is set in the Address Editor (See Project 3). The address offset for our bram reader is 0x4000 0000.
Good luck!
Hi, sorry for bother but im having some issues whit this project, i did everything accordingly to the post, but when i run the file on the red pitaya and i also run the Python code to make the plot, it does not show anything, so could it be that the file generated by XIlinix could be wrong? or am i doing something wrong?
Thanks
Greetings Anton,
I’m having troubles with this project, when I generate the bitstream with Vivado 2016.4 I get these critical warnings:
-[BD 41-1228] Width mismatch when connecting input pin ‘/Trigger/selector_0/div'(5) to net ‘xls_trigger_Dout'(8) – Only lower order bits will be connected, and other input bits of this pin will be left unconnected.
-[BD 41-1228] Width mismatch when connecting input pin ‘/Averager/pow2_1/log2N'(5) to net ‘xls_NAVERAGES_Dout'(8) – Only lower order bits will be connected, and other input bits of this pin will be left unconnected.
-[BD 41-1228] Width mismatch when connecting input pin ‘/Averager/axis_averager_0/nsamples'(16) to net ‘pow2_0_N'(32) – Only lower order bits will be connected, and other input bits of this pin will be left unconnected.
-[BD 41-1228] Width mismatch when connecting input pin ‘/Averager/pow2_0/log2N'(5) to net ‘xls_NSAMPLES_Dout'(8) – Only lower order bits will be connected, and other input bits of this pin will be left unconnected.
-[BD 41-1228] Width mismatch when connecting input pin ‘/xlconcat_0/In1′(7) to net ‘axis_averager_0_averages_out'(32) – Only lower order bits will be connected, and other input bits of this pin will be left unconnected.
I generated it anyways. However the Averager’s GUI does not plot anything at all. I can fix the trigger and GPIO mismatch easily but I need a little help with the others, I’m yet to modify anything in the project, any suggestions of what should I do?
Thanks in advanced.
Hi Raul,
In principle, these warnings are not project critical. You can solve the last three by concatenating the line with constant 0 or slicing your signals lines to end up with proper number of bits before connecting it to the next block.
The problem with not getting a signal to your client might be more involved. Here you can test separately communication between the server and the client and the FPGA program.
To test communication check if LEDs response to your click on the start measurement button in the client program. Also, I usually run the server in foreground where you can see in the output which commands have been received from the client. If this does not work, check your IP address, port number and firewall settings.
To test the FPGA program you can manually set the configuration using monitor program as described in project 3.
I haven’t tested this program with Vivado 2016.4. If there is a problem with the FPGA program, consider installing a newer version. In my case I used Vivado 2017.2.
Hi Anton
We tested the communication and it worked, we got this in the QT console:
run averager.py
connecting …
Connected
number of samples = 1024
number of averages = 262144
trigger = 17
got 1448
got 2648
and this in the red pitaya terminal:
1024 samples measured in 4.286392 s
however the GUI does not plot anything at all.
I’m yet to try using the monitor tool, I don’t know how to use it.
Any another suggestions?
Thanks a lot.
Hi,
It would seem that the server is running fine and that there is communication between the client and the server. I guess you don’t see any errors in the python console? Try selecting Abs/Re/Im value in the GUI and play with the scaling. It could be that your signal is only 0. Did you connect DAC OUT point to the ADC IN port?
To debug further, you can look into the python code and add a few print statements around the plotting routine.
Regarding the monitor, I wrote a few lines on how to use it in project 3. But I think, the server part works fine.
Hi Anton
I just noticed that there’s an error that appears in the QT console after closing the client program:
—————————————————————————
AttributeError Traceback (most recent call last)
/home/hawc/redpitaya_guide/projects/5_averager/client/averager.py in read_data(self)
150 self.offset = 0
151 self.haveData = True
–> 152 self.plot()
153 self.idle = True
154 self.socket.close()
/home/hawc/redpitaya_guide/projects/5_averager/client/averager.py in plot(self)
165 # reset toolbar
166 self.toolbar.home()
–> 167 self.toolbar._views.clear()
168 self.toolbar._positions.clear()
169 # reset plot
AttributeError: ‘NavigationToolbar2QT’ object has no attribute ‘_views’
—————————————————————————————-
I’m yet to learn python, could you please lead me on how t fix this?
Thanks a lot.
HI
For some reson the averager doest come up.
Hi Jose,
Could you be a bit more specific? At which step does not work?
hi
I was able to fix the earlier problem, now I have a diferent problem, a finish the tutorial and when I run the synthesis there are no errors , but when I run the implementation I get 21 errors.
ipx::edit_ip_in_project -upgrade true -name axis_averager_v1_0_project -directory C:/Users/jgiboyeaux.FCIRCE/Documents/1_led_blink/1_led_blink.tmp/axis_averager_v1_0_project c:/Users/jgiboyeaux.FCIRCE/Documents/1_led_blink/1_led_blink.ipdefs/cores_0/axis_averager_v1_0/component.xml
[BD 5-104] A block design must be open to run this command. Please create/open a block design.
[BD 41-1273] Error running can_apply_rule TCL procedure: ERROR: [Common 17-39] ‘get_bd_intf_pins’ failed due to earlier errors.
::bd::board_utils::is_intf_pin Line 3
those are some of the errors, and alot of them say the block desing most be open to run this comand even tho the block desing is alredy open.
thanks
Hi Anton
Sorry for keep asking the dum questions , but I have another one . I’m trying to set up the server but I dont know in which folder I have to copy the server.c file . The file is already on my red pitaya but when I go to putty and execute the command it says ¨no such file in directory¨
again thank
Hi Jose,
as long as the file (server.c) is on red pitaya’s Linux it doesn’t matter which folder you put it in. But, of course, you have to be in this folder to compile and execute it. In case the compilation is successful but the executable (server file) has no executable permissions you can add those with the command: chmod a+x server .
Hi Anton
I finish this project and it work great. Now I’m trying to change it a little to do my own project , for mine I want to get the data from the adc, save it , and then compare it to a different set of data, my goal is to measure 2 diferent voltage from an outside signal genarator, then comapare them , also if the voltage from A is greater than the voltage from B I want to turn LED3 on, vice versa if B is greater than A I want to turn LED4 on. so I have 2 questions, can I jsut do an output wire for int_addrA_reg, to get the data to another block to compare it to the other voltage? if I dont use naverages do I alter the code to much? I red on the description of the averger that:
“After NAVERAGES number of triggers the Averager stops and asserts the finished signal.The finished signal has two functions: first it lets the program running on Red Pitaya’s Linux know that acquisition is completed and results can be read. The second function is to re-route the BRAM PORTA line of the Block Memory Generator from Averager to AXI BRAM Reader block. This gives a program running on the processing system/Linux access for reading BRAM memory.”,
I still need the linux conection.
thanks
jose
Hi Jose,
I am not sure what is required for your project, but just comparing two values for example between IN1 and IN2 ports could be made very fast with a block similar to the one needed in frequency counter project. If you also need to read the values using your computer than I would save the values using averager with NAVERAGES = 1 and compare values at specific addresses. Perhaps the easiest way to add the comparison is to modify the averager core.
Did you managed to compile and run server.c?
Best, Anton
Yes I manage to compile server.c, thanks for all your help!!!! I just have one more question, I want to take the values that are added and divide them, to create a kind of filter to get less data, do you recommend that I do this in the fpga part or in the Linux code; my goal is to put this data that I get from the ADC into the SD card.
Thanks for such a great tutorial it really help !!!!
Yes I manage to compile server.c, thanks for all your help!!!! I just have one more question, I want to take the values that are added and divide them, to create a kind of filter to get less data, do you recommend that I do this in the fpga part or in the Linux code; my goal is to put this data that I get from the ADC into the SD card.
Thanks for such a great tutorial it really help !!!!
Hi Anton
I was able to finish all the other practice projects, but for this one I am getting an error. when I run the server it says listing to port 1001 , it connects to client, I put my info, hit start and nothing happens, then in putty says there is a bus error, How can I fix that?
thanks
Hi, is the putty unresponsive afterwards? If yes, then something went wrong in the fpga. Try recompiling the fpga project.
Hi,
I need to dump the file permanently ie, mcs (QSPI Bootup).
My board should bootup from SPI flash.
I tried dumping the mcs file on board from XSDK, but i am not sure about the flash memory type of SPI which is sitting on redpitaya.but no luck.Any suggestion?
Hi, I am not sure if I understand what you want to do correctly. If you need your bit file to be written to the FPGA after a restart, you can execute the command
at linux startup. See https://www.simplified.guide/linux/automatically-run-program-on-startup
You can also check if Pavel Demin uses another approach (http://pavel-demin.github.io/red-pitaya-notes/).
Hi Anton,
Can you give me an example how to adding a FIR filter into this project?
I will greatly appreciate your help.
Thanks
BR,
Amin
Hi Anton,
I wanted to understand if I can increase the number of samples captured from the program.
Also is continuous monitoring of data possible here (by reducing the sampling rates)
Kindly direct me , how is this possible.
Thanks a lot
Hi Anton, just wanted to say thank you very much for all your tutorials. I’ve followed all of them and they have worked well for me, while giving me some introduction to development on the Red Pitaya and the Zynq in general.