Re: Fwd: [Discuss-gnuradio] USRP2 eth_buffer
On Thu, Apr 23, 2009 at 9:40 PM, Eric Blossom e...@comsec.com wrote: Juha, thanks for the patch! This has been applied to the trunk at revision 11000. Johnathan ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org http://lists.gnu.org/mailman/listinfo/discuss-gnuradio
Re: [Discuss-gnuradio] USRP2 eth_buffer
ext file system is the go, with my high speed digitizer I stream 250 MB/s (thats bytes) to a six disk raid (0) array. The raid zero is the go if you can afford to loose data in the unlikely event of a disk failure. I'd guess that your high-speed digitizer has a buffer that is larger than 25 MB too. Do you know what the buffer size is for your sampler? I did a simple benchmark of filesystem bandwidth with xfs and ext2. xfs: j...@liang:/data0$ sudo time dd if=/dev/zero of=tmp.bin bs=1M count=1 1+0 records in 1+0 records out 1048576 bytes (10 GB) copied, 68.1847 s, 154 MB/s ext2: j...@liang:/data0$ sudo time dd if=/dev/zero of=tmp.bin bs=1M count=1 1+0 records in 1+0 records out 1048576 bytes (10 GB) copied, 68.1712 s, 154 MB/s Both give approximately the same bandwidth. I totally agree that ext2 might have less variability in i/o bandwidth, but at the same time, I don't really think there is that large difference between any decent modern filesystem in terms of long time average bandwidth of writing large files to disk. Large distributed filesystems are a different issue, and I'd guess that XFS and IBM's GPFS are good for those uses. I now took the time to reformat the disk to ext2 and tried to write 25 MHz to disk with the vanilla eth_buffer. It also gave an underrun after a few seconds. This might be because I am chopping the data into 100 MB files, but this is a necessity. I cannot have 24 hours of 25 MHz data in one large file. I have suggested a modification to the usrp2 API that would allow increasing the packet_ring buffer, why is that not a good idea? Isn't it a good idea to add a feature that allows people to reliably sample and store to disk at high bandwidth, even with more jittery filesystems? I think nobody is using a pre 2.6.5 kernel, so this there shouldn't really be any reason to restrict the size to the number of pointers that fit into a kernel slab size. I'll write a patch anyway and send it to the list. BR, juha ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org http://lists.gnu.org/mailman/listinfo/discuss-gnuradio
Fwd: [Discuss-gnuradio] USRP2 eth_buffer
I have attached a patch to allow users to define the ethernet packet ring size. I remove the SLAB_SIZE restriction. I think gnuradio needs a fairly new 2.6.5 kernel anyway. Why is this needed? I challenge anyone to sample at 25 MHz continuously for two hours without overruns or missing packets to a five disk raid array (be it ext2 or anything else) with the default 25 MB buffer. Still, the patch maintains the original 25e6 buffer size. A value of 250e6 to 500e6 allows fairly reliable sampling to disk at 25 MHz, so I recommend increasing the default buffer size to something higher than 25 MB. Otherwise new users will have problems with overruns. Even Firefox consumes hundreds of megabytes. juha -- Forwarded message -- From: Juha Vierinen jvier...@gmail.com Date: Thu, Apr 23, 2009 at 11:00 Subject: Re: [Discuss-gnuradio] USRP2 eth_buffer To: Bruce Stansby bruce.stan...@postgrad.curtin.edu.au Cc: Eric Blossom e...@comsec.com, Johnathan Corgan jcor...@corganenterprises.com, discuss-gnuradio@gnu.org discuss-gnuradio@gnu.org ext file system is the go, with my high speed digitizer I stream 250 MB/s (thats bytes) to a six disk raid (0) array. The raid zero is the go if you can afford to loose data in the unlikely event of a disk failure. I'd guess that your high-speed digitizer has a buffer that is larger than 25 MB too. Do you know what the buffer size is for your sampler? I did a simple benchmark of filesystem bandwidth with xfs and ext2. xfs: j...@liang:/data0$ sudo time dd if=/dev/zero of=tmp.bin bs=1M count=1 1+0 records in 1+0 records out 1048576 bytes (10 GB) copied, 68.1847 s, 154 MB/s ext2: j...@liang:/data0$ sudo time dd if=/dev/zero of=tmp.bin bs=1M count=1 1+0 records in 1+0 records out 1048576 bytes (10 GB) copied, 68.1712 s, 154 MB/s Both give approximately the same bandwidth. I totally agree that ext2 might have less variability in i/o bandwidth, but at the same time, I don't really think there is that large difference between any decent modern filesystem in terms of long time average bandwidth of writing large files to disk. Large distributed filesystems are a different issue, and I'd guess that XFS and IBM's GPFS are good for those uses. I now took the time to reformat the disk to ext2 and tried to write 25 MHz to disk with the vanilla eth_buffer. It also gave an underrun after a few seconds. This might be because I am chopping the data into 100 MB files, but this is a necessity. I cannot have 24 hours of 25 MHz data in one large file. I have suggested a modification to the usrp2 API that would allow increasing the packet_ring buffer, why is that not a good idea? Isn't it a good idea to add a feature that allows people to reliably sample and store to disk at high bandwidth, even with more jittery filesystems? I think nobody is using a pre 2.6.5 kernel, so this there shouldn't really be any reason to restrict the size to the number of pointers that fit into a kernel slab size. I'll write a patch anyway and send it to the list. BR, juha Index: usrp2/host/include/usrp2/usrp2.h === --- usrp2/host/include/usrp2/usrp2.h (revision 10899) +++ usrp2/host/include/usrp2/usrp2.h (working copy) @@ -86,7 +86,7 @@ * If \p addr is HH:HH, it's treated as if it were 00:50:c2:85:HH:HH * will autoselect a USRP2 if there is only a single one on the local ethernet. */ -static sptr make(const std::string ifc, const std::string addr=); +static sptr make(const std::string ifc, const std::string addr=, size_t rx_bufsize=0); /*! * Class destructor @@ -578,10 +578,10 @@ private: // Static function to retrieve or create usrp2 instance -static sptr find_existing_or_make_new(const std::string ifc, props *p); +static sptr find_existing_or_make_new(const std::string ifc, props *p, size_t rx_bufsize); // Only class members can instantiate this class -usrp2(const std::string ifc, props *p); +usrp2(const std::string ifc, props *p, size_t rx_bufsize); // All private state is held in opaque pointer std::auto_ptrimpl d_impl; Index: usrp2/host/lib/usrp2_impl.cc === --- usrp2/host/lib/usrp2_impl.cc (revision 10899) +++ usrp2/host/lib/usrp2_impl.cc (working copy) @@ -128,8 +128,8 @@ } - usrp2::impl::impl(const std::string ifc, props *p) -: d_eth_buf(new eth_buffer()), d_interface_name(ifc), d_pf(0), d_bg_thread(0), + usrp2::impl::impl(const std::string ifc, props *p, size_t rx_bufsize) +: d_eth_buf(new eth_buffer(rx_bufsize)), d_interface_name(ifc), d_pf(0), d_bg_thread(0), d_bg_running(false), d_rx_seqno(-1), d_tx_seqno(0), d_next_rid(0), d_num_rx_frames(0), d_num_rx_missing(0), d_num_rx_overruns(0), d_num_rx_bytes(0), d_num_enqueued(0), d_enqueued_mutex(), d_bg_pending_cond(d_enqueued_mutex
Re: Fwd: [Discuss-gnuradio] USRP2 eth_buffer
On Thu, Apr 23, 2009 at 01:15:56PM +0300, Juha Vierinen wrote: I have attached a patch to allow users to define the ethernet packet ring size. I remove the SLAB_SIZE restriction. I think gnuradio needs a fairly new 2.6.5 kernel anyway. Why is this needed? I challenge anyone to sample at 25 MHz continuously for two hours without overruns or missing packets to a five disk raid array (be it ext2 or anything else) with the default 25 MB buffer. Still, the patch maintains the original 25e6 buffer size. A value of 250e6 to 500e6 allows fairly reliable sampling to disk at 25 MHz, so I recommend increasing the default buffer size to something higher than 25 MB. Otherwise new users will have problems with overruns. Even Firefox consumes hundreds of megabytes. juha Juha, thanks for the patch! Eric ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org http://lists.gnu.org/mailman/listinfo/discuss-gnuradio
[Discuss-gnuradio] USRP2 eth_buffer
Hi, I have been trying to get 25 MHz to disk with USRP2. I am using the C++ interface and a five disk software raid 0 that can do about 150 MB/s. I can easily run at 25 MHz with a simple nop_handler that only checks for underruns and timestamps continuity, but when I write to disk, I can barely do 10 MHz for longer than 30 s without overruns. I have tried just about every filesystem with the same result every time. The reason seems to be lack of buffering. I have gone with the easiest path of increasing the eth_buffer from approximately 25 MB to 500 MB. I know people will flame me for using this much kernel memory, but it seems to work fairly reliably (I have been saving 25 MHz to a five disk software raid already for an hour without problems). I think there should be a user configurable option similar to fusb_blocksize and fusb_nblocks with usrp1, which defines the eth_buffer size. I am willing write a patch if somebody is interested, but I don't fully understand why there is a MAX_SLAB_SIZE in eth_buffer.cc: // Calculate number of blocks req.tp_block_nr = std::min((int)(MAX_SLAB_SIZE/sizeof(void*)), (int)(d_buflen/req.tp_block_size)); Why is there a (int)(MAX_SLAB_SIZE/sizeof(void*)) limit? juha ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org http://lists.gnu.org/mailman/listinfo/discuss-gnuradio
Re: [Discuss-gnuradio] USRP2 eth_buffer
On Wed, Apr 22, 2009 at 1:48 PM, Juha Vierinen jvier...@gmail.com wrote: I have been trying to get 25 MHz to disk with USRP2. I am using the C++ interface and a five disk software raid 0 that can do about 150 MB/s. I can easily run at 25 MHz with a simple nop_handler that only checks for underruns and timestamps continuity, but when I write to disk, I can barely do 10 MHz for longer than 30 s without overruns. I have tried just about every filesystem with the same result every time. Try setting your application to run using real-time scheduling priority. This is done in C++ via a call to: gr_enable_realtime_scheduling() or from Python: gr.enable_realtime_scheduling() Check the return value to ensure that it worked, it should == gruel::RT_OK from C++ or in python gr.RT_OK. You must have permission to do this, either by virtue of running as root, or by allowing your user/group to do so by adding a line to /etc/security/limits.conf: @usrp - rtprio 50 Then add your username to the 'usrp' group (which needs to be created if it doesn't already exist.) Why is there a (int)(MAX_SLAB_SIZE/sizeof(void*)) limit? We use the Linux kernel packet ring method of receiving packets from sockets. This is a speed optimized method that maps memory in such a way that the kernel sees it as kernel memory and the user process sees it at its own memory, so there is no copying from kernel to user space. It also lets us receive multiple packets with one system call. (At full rate, we process about 50 packets per system call.) The kernel maintains a ring of pointers to pending packets, and these ring descriptors must be stored in one kernel memory region. These memory regions are of MAX_SLAB_SIZE, and each descriptor is sizeof(void*). So the tp_block_nr variable calculates the number of possible packets by dividing the buffer length by the block size, and if that is more than can be stored in MAX_SLAB_SIZE, it reduces it to the limit that imposes. So you probably aren't using all 500 MB of that memory. You can uncomment the debug printf in that part of code to see the number of blocks actually allocated. What tends to happen if you aren't running your user process as RTPRIO is that the libusrp2 driver grabs the packets from the kernel okay, but your flowgraph doesn't read them from the driver fast enough, and you get backed up into an overflow. Johnathan ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org http://lists.gnu.org/mailman/listinfo/discuss-gnuradio
Re: [Discuss-gnuradio] USRP2 eth_buffer
Try setting your application to run using real-time scheduling priority. This is done in C++ via a call to: gr_enable_realtime_scheduling() I am using this. We use the Linux kernel packet ring method of receiving packets from sockets. This is a speed optimized method that maps memory in such a way that the kernel sees it as kernel memory and the user process sees it at its own memory, so there is no copying from kernel to user space. It also lets us receive multiple packets with one system call. (At full rate, we process about 50 packets per system call.) The kernel maintains a ring of pointers to pending packets, and these ring descriptors must be stored in one kernel memory region. These memory regions are of MAX_SLAB_SIZE, and each descriptor is sizeof(void*). So the tp_block_nr variable calculates the number of possible packets by dividing the buffer length by the block size, and if that is more than can be stored in MAX_SLAB_SIZE, it reduces it to the limit that imposes. Doesn't this apply only for pre 2.6.5 and 2.4.26 kernels? At least that is what the Documentation/networking/packet_mmap.txt says. BTW, shouldn't the MAX_SLAB_SIZE should be 131072 instead of 131702? So you probably aren't using all 500 MB of that memory. You can uncomment the debug printf in that part of code to see the number of blocks actually allocated. I think I'm using it all. I removed the MAX_SLAB_SIZE constraint and it still works in mmaped mode. The setsockopt still succeeds and the data looks ok. What tends to happen if you aren't running your user process as RTPRIO is that the libusrp2 driver grabs the packets from the kernel okay, but your flowgraph doesn't read them from the driver fast enough, and you get backed up into an overflow. This is exactly the problem. On average the disk bandwidth is more than enough, but there are fairly large hiccups that cause the buffer to overrun. I could try to write my own buffer, but that would be one extra memory copy, I'd prefer a large kernel-space buffer to a large user space buffer. juha ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org http://lists.gnu.org/mailman/listinfo/discuss-gnuradio
Re: [Discuss-gnuradio] USRP2 eth_buffer
On Wed, Apr 22, 2009 at 11:06:19PM +, Juha Vierinen wrote: Try setting your application to run using real-time scheduling priority. This is done in C++ via a call to: gr_enable_realtime_scheduling() I am using this. Juha, What kind of filesystem are you using? I've never been able to stream reliably to disk using the ext3 filesystem. I think it chokes when posting its journal. I have been successful when mounting an ext3 filesystem as an ext2 filesystem. I've got a RAID 10 system using 6 drives plus 1 hot spare. Eric ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org http://lists.gnu.org/mailman/listinfo/discuss-gnuradio
Re: [Discuss-gnuradio] USRP2 eth_buffer
Hi all ext file system is the go, with my high speed digitizer I stream 250 MB/s (thats bytes) to a six disk raid (0) array. The raid zero is the go if you can afford to loose data in the unlikely event of a disk failure. Bruce - Original Message - From: Eric Blossom e...@comsec.com Date: Thursday, April 23, 2009 8:04 am Subject: Re: [Discuss-gnuradio] USRP2 eth_buffer To: j...@sgo.fi Cc: discuss-gnuradio@gnu.org discuss-gnuradio@gnu.org, Johnathan Corgan jcor...@corganenterprises.com On Wed, Apr 22, 2009 at 11:06:19PM +, Juha Vierinen wrote: Try setting your application to run using real-time scheduling priority. This is done in C++ via a call to: gr_enable_realtime_scheduling() I am using this. Juha, What kind of filesystem are you using? I've never been able to stream reliably to disk using the ext3 filesystem. I think it chokes when posting its journal. I have been successful when mounting an ext3 filesystem as an ext2 filesystem. I've got a RAID 10 system using 6 drives plus 1 hot spare. Eric ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org http://lists.gnu.org/mailman/listinfo/discuss-gnuradio ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org http://lists.gnu.org/mailman/listinfo/discuss-gnuradio