Closing: GNU Radio 3.9.3.0 Python basic block - issues with, forecast and produce
Hello Johannes and hello Marcus, thank you both for the suggestions and explanations, it really helps a lot and is much appreciated! With your explanations, I was able to produce my minimal working example, which I have linked below. The best place to check for specifics about Python blocks is here: https://github.com/gnuradio/gnuradio/blob/main/gnuradio-runtime/python/gnuradio/gr/gateway.py This is the interface that is implemented. The `forecast` method is implemented in L153ff. It is different from the C++ version. Thanks for this hint, I've taken a look at the implementation. Also thanks for the suggestion with `bpython3` it is a really nice tool. For now my block works without the `forecast` method so I won't bother implementing it for now. I think it makes more sense to implement `forecast` if I should decide to work with a C++ block. The mechanism is that the scheduler asks forecast what it needs to produce the largest amount of output that the output buffer can deal with, then it halves the amount until it hits the minimum number. If none of these requests yield a fulfillable input requirement, the block is permanently input-blocked. Also thanks for clearing this up and explaining it Marcus! First off, the `consume_each` call needs to go behind any read on the input buffer. You really tell the system at this point: I'm finally done with these items, do whatever. Since GR is a multi-threaded system, this may cause trouble because the samples you want to read are already overwritten. So let's comment on your `general_work`. > def general_work(self, input_items, output_items): > # Firstly check, if input_items has a sufficient amount of items > if len(input_items[0]) >= self.buffer_len: > # Then consume exactly self.buffer_len items > self.consume_each(self.buffer_len) `consume_each` should go after the last use of `input_items`. The comment, together with your explanation makes perfect sense, I have changed it accordingly. > # Now only output a fraction of the input items, say the first > self.out_items, on output port[0] > output_items[0] = input_items[0][:self.out_items] This is probably the line causing issues. Try `output_items[0][0:self.out_items] = input_items[0][0:self.out_items]` The difference is the left hand indexing. Now, you write items to positions in an array. Previously, you overwrote the array and replaced it with a new one. Thus, the output buffer was still full of zeros. Changing this line was the last missing piece, the minimal example now works! I have posted my working minimal example here: https://pastebin.com/qxgnPcpL And with that I would like to close this problem, thanks again for the feedback! Kind regards, Patric
Re: GNU Radio 3.9.3.0 Python basic block - issues with forecast and produce
Hi Johannes, hi Patric, On 22.01.22 12:25, Johannes Demel wrote: The `forecast` method expects estimates. The scheduler will call `general_work` anyways at some point, if the system is unable to fulfill your forecast requirement. No, that is not correct; at least in the 3.8-style scheduler you can deadlock the whole flowgraph by letting the forecast method tell the scheduling algorithm that it needs more input to produce the minimum possible number of output (==1 or output_multiple) than the input could ever offer. The mechanism is that the scheduler asks forecast what it needs to produce the largest amount of output that the output buffer can deal with, then it halves the amount until it hits the minimum number. If none of these requests yield a fulfillable input requirement, the block is permanently input-blocked. Best regards, Marcus
Re: GNU Radio 3.9.3.0 Python basic block - issues with forecast and produce
Hi Patric, first off, the `consume_each` call needs to go behind any read on the input buffer. You really tell the system at this point: I'm finally done with these items, do whatever. Since GR is a multi-threaded system, this may cause trouble because the samples you want to read are already overwritten. The best place to check for specifics about Python blocks is here: https://github.com/gnuradio/gnuradio/blob/main/gnuradio-runtime/python/gnuradio/gr/gateway.py This is the interface that is implemented. The `forecast` method is implemented in L153ff. It is different from the C++ version. I guess Python blocks (and embedded Python blocks even more) are considered simple blocks for beginners. Unfortunately, this is not how they're treated by developers. Also, they are leaky abstractions over their C++ originals. To rephrase the new issue: - You insert a known pattern and expect corresponding output. - Actual output is all zeros. (The screenshot is missing). So let's comment on your `general_work`. > def general_work(self, input_items, output_items): > # Firstly check, if input_items has a sufficient amount of items > if len(input_items[0]) >= self.buffer_len: > # Then consume exactly self.buffer_len items > self.consume_each(self.buffer_len) `consume_each` should go after the last use of `input_items`. > # Now only output a fraction of the input items, say the first > self.out_items, on output port[0] > output_items[0] = input_items[0][:self.out_items] This is probably the line causing issues. Try `output_items[0][0:self.out_items] = input_items[0][0:self.out_items]` The difference is the left hand indexing. Now, you write items to positions in an array. Previously, you overwrote the array and replaced it with a new one. Thus, the output buffer was still full of zeros. This behavior is very unpythonic. It makes perfect sense though from a C++ block perspective. You pass pointers or references to a data structure and write data to it in C++. You wouldn't normally do that in Python. Cheers Johannes On 23.01.22 19:27, Patric Müller wrote: Hello everyone and hello Johannes, first of all, a huge thanks for the quick and detailed response, it is really helpful. After implementing your suggested changes and additional tinkering, I am still facing two issues: The `forecast` method expects estimates. The scheduler will call `general_work` anyways at some point, if the system is unable to fulfill your forecast requirement. `set_output_multiple` is much more strict. In your case I'd start with: `ninput_items_required[0] = noutput_items` I've changed the `forecast`method accordingly, however I still get the same errors. If I use `ninput_items_required[0] = noutput_items` then I receive: `TypeError: 'int' object does not support item assignment` I've checked the type and value of `ninput_items_required` and sure enough, it is of `` and has a value of `1`. My next thought was that it may not be a , since I only have one input port, so I also tried: `ninput_items_required = noutput_items` This yields me with: `Unable to cast Python instance to C++ type (compile in debug mode for details)` For now and since it isn't strictly necessary, I've commented the `forecast` method. At the end of `general_work` report these values to the system: - consumed items via `consume_each` - produced items via `return integer_with_number_of_consumed_items`. I have changed my code accordingly. My `general_work` function now looks like this: def general_work(self, input_items, output_items): # Firstly check, if input_items has a sufficient amount of items if len(input_items[0]) >= self.buffer_len: # Then consume exactly self.buffer_len items self.consume_each(self.buffer_len) # Now only output a fraction of the input items, say the first self.out_items, on output port[0] output_items[0] = input_items[0][:self.out_items] else: # if we do not have enough input_items, set empty output output_items[0] = [] # finally, return len(output_items[0]), which is either equal to `out_items` or `0` return len(output_items[0]) It runs and I did some tests. I have created a signal of length 1024, which is composed of a random sequence with length 312 and a cosine signal with a length of 712 both muxed together. Next follows the python block and the signal is compared before and after by using time sinks. Screenshot of the flowgraph: I would expect that I should only observe the random sequence (first 312 samples) in the `processed signal` time sink. However the time sink only shows zeros, which is probably due to a coding error. In order to investigate, I took a look at the content of `output_items[0]` and also at the length. The length is 312, which seems fine, since I specified it with `self.out_items`. The contents also seem fine, it is definitely not z
Re: Re: GNU Radio 3.9.3.0 Python basic block - issues with forecast and produce
Hello everyone and hello Johannes, first of all, a huge thanks for the quick and detailed response, it is really helpful. After implementing your suggested changes and additional tinkering, I am still facing two issues: The `forecast` method expects estimates. The scheduler will call `general_work` anyways at some point, if the system is unable to fulfill your forecast requirement. `set_output_multiple` is much more strict. In your case I'd start with: `ninput_items_required[0] = noutput_items` I've changed the `forecast`method accordingly, however I still get the same errors. If I use `ninput_items_required[0] = noutput_items` then I receive: `TypeError: 'int' object does not support item assignment` I've checked the type and value of `ninput_items_required` and sure enough, it is of `` and has a value of `1`. My next thought was that it may not be a , since I only have one input port, so I also tried: `ninput_items_required = noutput_items` This yields me with: `Unable to cast Python instance to C++ type (compile in debug mode for details)` For now and since it isn't strictly necessary, I've commented the `forecast` method. At the end of `general_work` report these values to the system: - consumed items via `consume_each` - produced items via `return integer_with_number_of_consumed_items`. I have changed my code accordingly. My `general_work` function now looks like this: def general_work(self, input_items, output_items): # Firstly check, if input_items has a sufficient amount of items if len(input_items[0]) >= self.buffer_len: # Then consume exactly self.buffer_len items self.consume_each(self.buffer_len) # Now only output a fraction of the input items, say the first self.out_items, on output port[0] output_items[0] = input_items[0][:self.out_items] else: # if we do not have enough input_items, set empty output output_items[0] = [] # finally, return len(output_items[0]), which is either equal to `out_items` or `0` return len(output_items[0]) It runs and I did some tests. I have created a signal of length 1024, which is composed of a random sequence with length 312 and a cosine signal with a length of 712 both muxed together. Next follows the python block and the signal is compared before and after by using time sinks. Screenshot of the flowgraph: I would expect that I should only observe the random sequence (first 312 samples) in the `processed signal` time sink. However the time sink only shows zeros, which is probably due to a coding error. In order to investigate, I took a look at the content of `output_items[0]` and also at the length. The length is 312, which seems fine, since I specified it with `self.out_items`. The contents also seem fine, it is definitely not zero and matches the content of `input_items[0]`. Since everything "seems" fine, I am not sure on how I could debug this further. My complete code, including the Debug statements, can be found here: https://pastebin.com/HRMzp0zA Thanks for taking your time and looking into this! Kind regards, Patric
Re: GNU Radio 3.9.3.0 Python basic block - issues with forecast and produce
Hi Patric, since your goal is to produce a variable number of output items, your `general_work` approach seems to be the correct one. I mean "variable number of output items" in the sense that there is no fixed relation between the number of input items to the number of output items. Just for reference. `WORK_CALLED_PRODUCE` is one of those "magic values" that are defined somewhere in the system. I've never personally used it in any block. It is documented here: https://www.gnuradio.org/doc/doxygen/classgr_1_1block.html The function is found here: https://www.gnuradio.org/doc/doxygen/classgr_1_1block.html#aa5581727d057bdd8113f8b2a3fc5bd66 I use `bpython3` to explore Python methods etc. In your case, I expect that `self.WORK_CALLED_PRODUCE` should be available. BUT: Don't use `produce`! In your case use: `return self.buffer_len` Why? Because the all the buffer pointer updates will be handled after you return from `general_work`. The `produce` method exists for cases where you have multiple output buffers and want to produce different numbers of output items. e.g 5 samples on `output_items[0]` and 25 samples on `output_items[1]`. In your case, you `return 0` which tells the scheduler that you did not produce any items. More generally, try to minimize calls to `consume`/`produce`. Keep track of the number of samples your `general_work` consumes and how many samples you've written to your output buffer. At the end of `general_work` report these values to the system: - consumed items via `consume_each` - produced items via `return integer_with_number_of_consumed_items`. The `forecast` method expects estimates. The scheduler will call `general_work` anyways at some point, if the system is unable to fulfill your forecast requirement. `set_output_multiple` is much more strict. In your case I'd start with: `ninput_items_required[0] = noutput_items` which might be inefficient. But you can update it with better forecasts later. Talking about efficiency: I'd only worry about efficiency in C++ blocks. Python blocks incur quite a penalty. Cheers Johannes On 21.01.22 15:14, Patric Müller wrote: Hello everyone, I am currently trying to create a basic block similar to this issue: https://stackoverflow.com/questions/68289222/gnu-radio-buffer-size Some software and hardware info: GNU Radio version: 3.9.3.0 Python version: 3.9.7 OS: Ubuntu 20.04 I always want to work on the same number of items, say 1024. Now I would like to use up all of the items, but I want to produce something between say 500 - 1500 items. For now, I would like to create a simple block with one input and one output, which consumes a set number of items and outputs only a fraction of those incoming items in order to understand it better. Therefore I started with a general block and wanted to change ninput_items_required[0]. However, ninput_items_required is an integer. Trying to directly change this integer values results in: Unable to cast Python instance to C++ type (compile in debug mode for details) My second question is about the produce() function. If it is used, general_work() should return WORK_CALLED_PRODUCE but how exactly? After calling produce() and telling the scheduler how many items are produced, can I then change output_items[0]? My attempt at the forecast() function and general_work() is provided below. General suggestions/hints or a basic example would be helpful and immensely appreciated. If additional background is required, let me know and I will gladly provide more information. def forecast(self, noutput_items, ninput_items_required): # Estimate, that we will always need self.buffer_len items on input port [0] ninput_items_required[0] = self.buffer_len def general_work(self, input_items, output_items): # Firstly check, if input_items has a sufficient amount of items if len(input_items[0]) >= self.buffer_len: # Then consume exactly self.buffer_len items on input_port[0] self.consume(0, self.buffer_len) # Now I would like to only output a fraction of the input items # say the first self.out_items, on output port[0] self.produce(0, self.out_items) # Do I set my output_items[0] here? # Now I only need to return WORK_CALLED_PRODUCE, but how exactly? return 0 Kind regards, Patric Müller
GNU Radio 3.9.3.0 Python basic block - issues with forecast and produce
Hello everyone, I am currently trying to create a basic block similar to this issue: https://stackoverflow.com/questions/68289222/gnu-radio-buffer-size Some software and hardware info: GNU Radio version: 3.9.3.0 Python version: 3.9.7 OS: Ubuntu 20.04 I always want to work on the same number of items, say 1024. Now I would like to use up all of the items, but I want to produce something between say 500 - 1500 items. For now, I would like to create a simple block with one input and one output, which consumes a set number of items and outputs only a fraction of those incoming items in order to understand it better. Therefore I started with a general block and wanted to change ninput_items_required[0]. However, ninput_items_required is an integer. Trying to directly change this integer values results in: Unable to cast Python instance to C++ type (compile in debug mode for details) My second question is about the produce() function. If it is used, general_work() should return WORK_CALLED_PRODUCE but how exactly? After calling produce() and telling the scheduler how many items are produced, can I then change output_items[0]? My attempt at the forecast() function and general_work() is provided below. General suggestions/hints or a basic example would be helpful and immensely appreciated. If additional background is required, let me know and I will gladly provide more information. def forecast(self, noutput_items, ninput_items_required): # Estimate, that we will always need self.buffer_len items on input port [0] ninput_items_required[0] = self.buffer_len def general_work(self, input_items, output_items): # Firstly check, if input_items has a sufficient amount of items if len(input_items[0]) >= self.buffer_len: # Then consume exactly self.buffer_len items on input_port[0] self.consume(0, self.buffer_len) # Now I would like to only output a fraction of the input items # say the first self.out_items, on output port[0] self.produce(0, self.out_items) # Do I set my output_items[0] here? # Now I only need to return WORK_CALLED_PRODUCE, but how exactly? return 0 Kind regards, Patric Müller
