While working to get coherent streams working, I ran into an issue using an x310 with two TwinRX daughterboards. The issue starts with a series of "ERROR_CODE_OVERFLOW (Out of sequence error)" errors. In an attempt to recover from that, the rx streamer is thrown out and recreated. The next stream errors change to "ERROR_CODE_TIMEOUT". Once in this state, all future streams end with this error. The x310 is connected over 10G ethernet.
I managed to reproduce this error with an example program based off of "rx_multi_samples.cpp". I had to make the following changes: 1. STREAM_MODE_START_CONTINUOUS is now used, ending the stream with STREAM_MODE_STOP_CONTINUOUS 2. The stream delay was set to .01 (mostly to speed up the rate the error would occur) 3. Multi stream commands (and stop commands) are issued in repetition (start, stop, start, stop, etc.) rather than just one long stream 4. Each stream uses a different sampling rate (alternates between 25Msps and 50Msps) 5. A small loop was added to the collect loop to slow down the thread enough to get overflow errors (but only sometimes, nothing crazy) 6. Once the out of sequence error is encountered 10 times in a row, the rx streamer is destroyed and re-created 7. Every stream command after step 5 ends in a timeout error It is also worth pointing out that this does not happen if the sample rate does not change. The out of sequence errors will still happen until the rx streamer is re-created, but the timeout errors do not occur after that... I attached the entire example program (with modifications) to this email. I started it with the args: rx_multi_samples --args addr=192.168.30.2 --subdev "A:0 A:1 B:0 B:1" --channels "0,1,2,3" --dilv --rate 50000000 --nsamps 8000000 Is there something wrong with how we are using the interface? Is there steps we can take to either avoid or recover from this state? I appreciate any help we can get... Will
// // Copyright 2011 Ettus Research LLC // // This program is free software: you can redistribute it and/or modify // it under the terms of the GNU General Public License as published by // the Free Software Foundation, either version 3 of the License, or // (at your option) any later version. // // This program is distributed in the hope that it will be useful, // but WITHOUT ANY WARRANTY; without even the implied warranty of // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the // GNU General Public License for more details. // // You should have received a copy of the GNU General Public License // along with this program. If not, see <http://www.gnu.org/licenses/>. // #include <uhd/utils/thread_priority.hpp> #include <uhd/utils/safe_main.hpp> #include <uhd/usrp/multi_usrp.hpp> #include <boost/program_options.hpp> #include <boost/format.hpp> #include <boost/thread.hpp> #include <boost/lexical_cast.hpp> #include <boost/algorithm/string.hpp> #include <iostream> #include <complex> #include <thread> #include <chrono> #include <unistd.h> namespace po = boost::program_options; const size_t MAX_OVERFLOW_ERRORS = 5; // the maximum number of overflows at zero samples received before the capture errors out int UHD_SAFE_MAIN(int argc, char *argv[]){ uhd::set_thread_priority_safe(); //variables to be set by po std::string args, sync, subdev, channel_list; double seconds_in_future; size_t total_num_samps; double rate; //setup the program options po::options_description desc("Allowed options"); desc.add_options() ("help", "help message") ("args", po::value<std::string>(&args)->default_value(""), "single uhd device address args") ("secs", po::value<double>(&seconds_in_future)->default_value(1.5), "number of seconds in the future to receive") ("nsamps", po::value<size_t>(&total_num_samps)->default_value(10000), "total number of samples to receive") ("rate", po::value<double>(&rate)->default_value(100e6/16), "rate of incoming samples") ("sync", po::value<std::string>(&sync)->default_value("now"), "synchronization method: now, pps, mimo") ("subdev", po::value<std::string>(&subdev), "subdev spec (homogeneous across motherboards)") ("dilv", "specify to disable inner-loop verbose") ("channels", po::value<std::string>(&channel_list)->default_value("0"), "which channel(s) to use (specify \"0\", \"1\", \"0,1\", etc)") ; po::variables_map vm; po::store(po::parse_command_line(argc, argv, desc), vm); po::notify(vm); //print the help message if (vm.count("help")){ std::cout << boost::format("UHD RX Multi Samples %s") % desc << std::endl; std::cout << " This is a demonstration of how to receive aligned data from multiple channels.\n" " This example can receive from multiple DSPs, multiple motherboards, or both.\n" " The MIMO cable or PPS can be used to synchronize the configuration. See --sync\n" "\n" " Specify --subdev to select multiple channels per motherboard.\n" " Ex: --subdev=\"0:A 0:B\" to get 2 channels on a Basic RX.\n" "\n" " Specify --args to select multiple motherboards in a configuration.\n" " Ex: --args=\"addr0=192.168.10.2, addr1=192.168.10.3\"\n" << std::endl; return ~0; } bool verbose = vm.count("dilv") == 0; //create a usrp device std::cout << std::endl; std::cout << boost::format("Creating the usrp device with: %s...") % args << std::endl; uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make(args); //always select the subdevice first, the channel mapping affects the other settings if (vm.count("subdev")) usrp->set_rx_subdev_spec(subdev); //sets across all mboards std::cout << boost::format("Using Device: %s") % usrp->get_pp_string() << std::endl; //set the rx sample rate (sets across all channels) std::cout << boost::format("Setting RX Rate: %f Msps...") % (rate/1e6) << std::endl; usrp->set_rx_rate(rate); std::cout << boost::format("Actual RX Rate: %f Msps...") % (usrp->get_rx_rate()/1e6) << std::endl << std::endl; std::cout << boost::format("Setting device timestamp to 0...") << std::endl; //Set the time based on GPS time { std::cout << "Setting to GPS time" << std::endl; uhd::time_spec_t last = usrp->get_time_last_pps(); uhd::time_spec_t next = usrp->get_time_last_pps(); while(next == last) { std::this_thread::sleep_for(std::chrono::milliseconds(50)); last = next; next = usrp->get_time_last_pps(); } // sleep 200 ms to allow new time data from the GPSDO to propagate through the system std::this_thread::sleep_for(std::chrono::milliseconds(200)); // set time usrp->set_time_next_pps(uhd::time_spec_t(time_t(usrp->get_mboard_sensor("gps_time").to_int()+1))); // wait for PPS again last = usrp->get_time_last_pps(); next = usrp->get_time_last_pps(); while(next == last) { std::this_thread::sleep_for(std::chrono::milliseconds(50)); last = next; next = usrp->get_time_last_pps(); } // sleep 200 ms to allow new time data from the GPSDO to propagate through the system std::this_thread::sleep_for(std::chrono::milliseconds(200)); usrp->set_clock_source("gpsdo"); } //detect which channels to use std::vector<std::string> channel_strings; std::vector<size_t> channel_nums; boost::split(channel_strings, channel_list, boost::is_any_of("\"',")); for(size_t ch = 0; ch < channel_strings.size(); ch++){ size_t chan = boost::lexical_cast<int>(channel_strings[ch]); if(chan >= usrp->get_rx_num_channels()){ throw std::runtime_error("Invalid channel(s) specified."); } else channel_nums.push_back(boost::lexical_cast<int>(channel_strings[ch])); } //create a receive streamer //linearly map channels (index0 = channel0, index1 = channel1, ...) uhd::stream_args_t stream_args("sc16", ""); //16 bit complex integers stream_args.channels = channel_nums; uhd::rx_streamer::sptr rx_stream = usrp->get_rx_stream(stream_args); //allocate buffers to receive with samples (one buffer per channel) const size_t samps_per_buff = rx_stream->get_max_num_samps(); std::vector<std::vector<std::complex<int16_t> > > buffs( usrp->get_rx_num_channels(), std::vector<std::complex<int16_t> >(samps_per_buff) ); //create a vector of pointers to point to each of the channel buffers std::vector<std::complex<int16_t> *> buff_ptrs; for (size_t i = 0; i < buffs.size(); i++) buff_ptrs.push_back(&buffs[i].front()); //will switch each time between 50Msps and 25Msps uint64_t seqErrs = 0; bool rateFlag = false; while(true) { try{ //set the sampling rate to a new value if(rateFlag) rate = 50000000.0; else rate = 25000000.0; rateFlag = !rateFlag; usrp->set_rx_rate(rate); //wait for LO lock for(size_t channel : channel_nums) { const size_t MAX_LOCK_WAIT = 500; // number of 10 millisecond sleep periods to wait for LO lock size_t count = 0; for(; count < MAX_LOCK_WAIT; count++) { if(usrp->get_rx_sensor("lo_locked", channel).to_bool()) { break; } usleep(10000); // 10 milliseconds } } //meta-data will be filled in by recv() uhd::rx_metadata_t md; //the first call to recv() will block this many seconds before receiving double timeout = 0.5; //timeout (delay before receive + padding) size_t num_acc_samps = 0; //number of accumulated samples //clear out any old packets left from previous captures while(true) { uhd::rx_metadata_t sampleMetadata; size_t samplesRecieved = rx_stream->recv(buff_ptrs, samps_per_buff, md, 0, true); if(samplesRecieved == 0) { break; } } //Stream continuously until told to stop... uhd::stream_cmd_t stream_cmd(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS); uhd::stream_cmd_t stop_cmd(uhd::stream_cmd_t::STREAM_MODE_STOP_CONTINUOUS); stream_cmd.stream_now = false; //delay the command by 10ms uhd::time_spec_t stream_delay = 0.01; //required to support coherent collects for multi-channel USRPs stream_cmd.time_spec = usrp->get_time_now(); stream_cmd.time_spec += stream_delay.get_real_secs(); rx_stream->issue_stream_cmd(stream_cmd); //tells all channels to stream size_t overflowErrors = 0; while(num_acc_samps < total_num_samps){ //receive a single packet (or slightly less depending on how much is needed) size_t samps_to_recv = std::min(total_num_samps-num_acc_samps, samps_per_buff); size_t num_rx_samps = rx_stream->recv( buff_ptrs, samps_per_buff, md, timeout ); //use a small timeout for subsequent packets timeout = 0.1; //handle the error code // ignore overflow if no samples have been received yet if(md.error_code == uhd::rx_metadata_t::ERROR_CODE_OVERFLOW && num_acc_samps == 0 && overflowErrors < MAX_OVERFLOW_ERRORS) { overflowErrors++; continue; } if (md.error_code == uhd::rx_metadata_t::ERROR_CODE_TIMEOUT) break; if (md.error_code != uhd::rx_metadata_t::ERROR_CODE_NONE){ std::cerr << "got error!" << md.strerror() << std::endl; break; } if(verbose) std::cout << boost::format( "Received packet: %u samples, %u full secs, %f frac secs" ) % num_rx_samps % md.time_spec.get_full_secs() % md.time_spec.get_frac_secs() << std::endl; num_acc_samps += num_rx_samps; //some garbage code to slow this thread down just enough to cause overflows for(auto& buf : buffs) { for(auto& num : buf) num.real(num.real() - 1); } } if (md.error_code == uhd::rx_metadata_t::ERROR_CODE_TIMEOUT && num_acc_samps < total_num_samps) std::cerr << "Receive timeout before all samples received..." << std::endl; rx_stream->issue_stream_cmd(stop_cmd); //tells all channels to stop //If there was an out of sequence error AND we have seen a few in a row, reset and re-create the rx streamer... if (md.error_code != uhd::rx_metadata_t::ERROR_CODE_NONE){ if(md.out_of_sequence) { seqErrs++; if(seqErrs > 10) { seqErrs = 0; std::cout << "Reseting RX stream!" << std::endl; rx_stream.reset(); rx_stream = usrp->get_rx_stream(stream_args); } } } } catch (...) { std::cout << "Caught exception" << std::endl; } } return EXIT_SUCCESS; }
_______________________________________________ USRP-users mailing list USRP-users@lists.ettus.com http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
