Re: [Discuss-gnuradio] ZMQ REQ / REP naming Swap?
Hi Michael. Code isn't published, because it belongs to a project for the company I work for. But it was all in python and just for the combinations I needed... so you are probably not missing anything. :) Sorry. 2016-07-29 9:08 GMT-03:00 Michael Dickens : > Hi Francisco - Yes, I think having a generic ZMQ block would be awesome. > Do you have an OOT module or block that does what you listed: a generic ZMQ > block? If so, how do I gain access to it? This would save me lots of time! > Thanks! - MLD > > On Thu, Jul 28, 2016, at 10:21 PM, Francisco Albani wrote: > > Michael, I found the need to do the same for some applications. I also > needed to choose between bind and connect (I made this a parameter of my > block). > Wouldn't be better to have just one sink and one source with selectable > options, like: > * socket_type: REP/REQ/SUB/PUB/PUSH/PULL > * method: bind / connect > * interface: stream / message > > > ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
Re: [Discuss-gnuradio] ZMQ REQ / REP naming Swap?
On Fri, 2016-07-29 at 11:49 -0400, Michael Dickens wrote: > I'm looking for an OOT to do the REP /REQ way that GR does -not- > provide, or will just create one myself. I don't anticipate adding it to > the core gr-zeromq component. I agree that it would add unnecessary > complexity to those already-existing blocks. - MLD I don't care if new blocks are added to gr-zmq to cover the other combinations. I just think its a bad idea to code a super-flexible gr-zeromq block with "multiple-personalities" into a single C++ block and have its functionality change between REQ/REP/PUB/SUB/PUSH/PULL based on an input parameter. BTW, I know of no OOT that has the functionality you need. Sorry. Regards, Andy ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
Re: [Discuss-gnuradio] ZMQ REQ / REP naming Swap?
Hi Andy - Thanks for your detailed thoughts on the subject matter. I think we're all in general agreement that there are 2 ways to do REP / REQ with respect to which is client and which is server. GR implements one of those ways, but could easily provide the other, too. It's just another way of looking at which is server and which is client. Neither is more or less correct; they are just different implementations. I'm looking for an OOT to do the REP /REQ way that GR does -not- provide, or will just create one myself. I don't anticipate adding it to the core gr-zeromq component. I agree that it would add unnecessary complexity to those already-existing blocks. - MLD > I think this would make the gr-zmq C++ block code unnecessarily complex > and harder to maintain. The block requirements for each combination of > parameters grow in different directions. > > Some things you'd have to deal with: > 1. REP blocks have a receive then send main loop; >REQ blocks have a send then receive main loop. > > 2. REQ/REP can never drop messages/samples and so the tag propagation > code can be simple; > PUB/SUB can drop messages/samples and tag propagation code needs to > account for that in tag offsets > > 3. GNURadio messages are usually small enough that they can be handled > in one call to work(). GNURadio sample stream payloads can have a great > many samples that need to be handled across several calls to work(). ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
Re: [Discuss-gnuradio] ZMQ REQ / REP naming Swap?
> From: > Francisco Albani > Subject: > Re: [Discuss-gnuradio] ZMQ REQ / > REP naming Swap? > Date: > Thu, 28 Jul 2016 23:21:25 -0300 > > __ > Michael, I found the need to do the same for some applications. I also > needed to choose between bind and connect (I made this a parameter of > my block). > > > Wouldn't be better to have just one sink and one source with > selectable options, like: > * socket_type: REP/REQ/SUB/PUB/PUSH/PULL > > * method: bind / connect > > * interface: stream / message > I think this would make the gr-zmq C++ block code unnecessarily complex and harder to maintain. The block requirements for each combination of parameters grow in different directions. Some things you'd have to deal with: 1. REP blocks have a receive then send main loop; REQ blocks have a send then receive main loop. 2. REQ/REP can never drop messages/samples and so the tag propagation code can be simple; PUB/SUB can drop messages/samples and tag propagation code needs to account for that in tag offsets 3. GNURadio messages are usually small enough that they can be handled in one call to work(). GNURadio sample stream payloads can have a great many samples that need to be handled across several calls to work(). It's just not worth it at the C++ level. Maybe at the GRC block XML file level, it might be worth it. My $0.02 Regards, Andy ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
Re: [Discuss-gnuradio] ZMQ REQ / REP naming Swap?
From: Michael Dickens Date: Thu, 28 Jul 2016 20:23:36 -0400 > On Thu, Jul 28, 2016, at 06:44 PM, Johnathan Corgan wrote: >> The ZMQ REQ/REP dataflow has the receiving end REQuest data from the sending end when needed, which REPlies with a packet. It's a form of flow control. >> >> From the GNU Radio perspective, streams flow into GNU Radio sinks to exit the flowgraph, and data is sourced into a flowgraph from a GNU Radio source block. >> >> Thus, the correct model is GR Stream | GR REP Sink | ZMQ REP | ZMQ Transport | ZMQ REQ | GR REQ source | GR stream [snip] >My argument here is that it's equally valid to do the ZMQ REP/REQ >transport either way, and if other projects are doing it the opposite >way from GNU Radio, why aren't we providing an option to support that >way? [snip] The ZMQ Guide is pretty clear that the core REQ/REP pattern is intended to have the following usage: REQuests are sent by Clients REPlies are sent by Servers Likewise in GNURadio it is clear that Sinks send data out of a GNURadio instance Sources send data into a GNURadio instance Server vs. client and Source vs. Sink are orthogonal. So any of the following is valid: REQ source REQ sink REP source REP sink REQ msg source REQ msg sink REP msg source REP msg sink But what you actually should use depends on the functional context. In the context of using REQ/REP, here are some use cases where I would likely use a GNURadio instance as a server or a client. This is for GNURadio instances communicating with programs which are not GNURadio instances: GNURadio as a client: - (C1) Accepting medium to high rate sample data from an external source for RX processing (REQ source) (C2) After Rx processing, providing medium to low rate PDUs/messages to an external source. (REQ msg sink) (C3) Providing low rate status messages (REQ msg sink) (C4) Accepting low rate sample data from an external source for further Tx processing (REQ source) GNURadio as a server: - (S1) Tx processing output providing medium to high sample rate data to an external source (REP sink) (S2) Accepting medium to low rate PDUs/messages for Tx processing (REP msg source) (S3) Accepting low rate command and control messages (REP msg source) (S4) After some Rx processing providing low rate sample data to an external source (REP sink) Of course other developers can have other design preferences than those above. In the above, I would actually never use REQ/REP for cases C1 and S1. REQ/REP is too high overhead and provides a reliability guarantee way beyond what is the norm for SDR. (libuhd/USRP source will drop data on the floor if your GNURadio instance can't keep up; REQ/REP will not.) REQ/REP is not the right pattern to use for high rate data. From the ZeroMQ Guide: "Request-reply, which connects a set of clients to a set of services. This is a remote procedure call and task distribution pattern." TL;DR: REQ is for clients, REP is for servers. ZeroMQ REP/REQ is orthogonal to GNURadio source/sink. Some (REQ/REP, stream/msg, source/sink) block functional combinations are missing from gr-zmq. The current gr-zmq REQ/REP source/sink block combinations are not wrong or incorrect, but they might not fit the usage pattern one may need or envision in a particular context. Don't use REQ/REP for medium to high sample rate sample streams. Regards, Andy ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
Re: [Discuss-gnuradio] ZMQ REQ / REP naming Swap?
Hi Francisco - Yes, I think having a generic ZMQ block would be awesome. Do you have an OOT module or block that does what you listed: a generic ZMQ block? If so, how do I gain access to it? This would save me lots of time! Thanks! - MLD On Thu, Jul 28, 2016, at 10:21 PM, Francisco Albani wrote: > Michael, I found the need to do the same for some applications. I also > needed to choose between bind and connect (I made this a parameter of > my block). > Wouldn't be better to have just one sink and one source with > selectable options, like: > * socket_type: REP/REQ/SUB/PUB/PUSH/PULL > * method: bind / connect > * interface: stream / message ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
Re: [Discuss-gnuradio] ZMQ REQ / REP naming Swap?
Michael, I found the need to do the same for some applications. I also needed to choose between bind and connect (I made this a parameter of my block). Wouldn't be better to have just one sink and one source with selectable options, like: * socket_type: REP/REQ/SUB/PUB/PUSH/PULL * method: bind / connect * interface: stream / message 2016-07-28 21:23 GMT-03:00 Michael Dickens : > On Thu, Jul 28, 2016, at 06:44 PM, Johnathan Corgan wrote: > > The ZMQ REQ/REP dataflow has the receiving end REQuest data from the > sending end when needed, which REPlies with a packet. It's a form of flow > control. > > From the GNU Radio perspective, streams flow into GNU Radio sinks to exit > the flowgraph, and data is sourced into a flowgraph from a GNU Radio source > block. > > Thus, the correct model is GR Stream | GR REP Sink | ZMQ REP | ZMQ > Transport | ZMQ REQ | GR REQ source | GR stream > > > The GR stream is no longer the "pull" model that it was back in the day. > If anything, it's more the "push" model: data is created at sources and > moved downstream, processed & moved again until it reaches sinks. > > It's just as valid to say that the sending end generates data, sends it > via ZMQ, & REQuests a reply; the receiving end gets the data via ZMQ then > REPlies; hence REQ/sink -> REP/source instead of the other way around. That's > the model the ZMQ folks use in their Figure 2 from > http://zguide.zeromq.org/page:all . > > I'm not saying it's the only way to do things; what I am saying is that in > a quick internet search this REQ/sink -> REP/source is how some other > projects do their transport. I can't review all projects, so I can't say > "many" or "all"; I did not find any projects that didn't do it this way, > except GNU Radio, though they do likely exist. > > My argument here is that it's equally valid to do the ZMQ REP/REQ > transport either way, and if other projects are doing it the opposite way > from GNU Radio, why aren't we providing an option to support that way? > > As I wrote: I'm just going to create my own OOT that swaps the REP/REQ > names, so that it works with my project. - MLD > > > ___ > Discuss-gnuradio mailing list > Discuss-gnuradio@gnu.org > https://lists.gnu.org/mailman/listinfo/discuss-gnuradio > > ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
Re: [Discuss-gnuradio] ZMQ REQ / REP naming Swap?
On Thu, Jul 28, 2016, at 06:44 PM, Johnathan Corgan wrote: > The ZMQ REQ/REP dataflow has the receiving end REQuest data from the > sending end when needed, which REPlies with a packet. It's a form of > flow control. > > From the GNU Radio perspective, streams flow into GNU Radio sinks to > exit the flowgraph, and data is sourced into a flowgraph from a GNU > Radio source block. > > Thus, the correct model is GR Stream | GR REP Sink | ZMQ REP | ZMQ > Transport | ZMQ REQ | GR REQ source | GR stream The GR stream is no longer the "pull" model that it was back in the day. If anything, it's more the "push" model: data is created at sources and moved downstream, processed & moved again until it reaches sinks. It's just as valid to say that the sending end generates data, sends it via ZMQ, & REQuests a reply; the receiving end gets the data via ZMQ then REPlies; hence REQ/sink -> REP/source instead of the other way around. That's the model the ZMQ folks use in their Figure 2 from http://zguide.zeromq.org/page:all . I'm not saying it's the only way to do things; what I am saying is that in a quick internet search this REQ/sink -> REP/source is how some other projects do their transport. I can't review all projects, so I can't say "many" or "all"; I did not find any projects that didn't do it this way, except GNU Radio, though they do likely exist. My argument here is that it's equally valid to do the ZMQ REP/REQ transport either way, and if other projects are doing it the opposite way from GNU Radio, why aren't we providing an option to support that way? As I wrote: I'm just going to create my own OOT that swaps the REP/REQ names, so that it works with my project. - MLD ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
Re: [Discuss-gnuradio] ZMQ REQ / REP naming Swap?
On Thu, Jul 28, 2016 at 12:20 PM, Michael Dickens wrote: > OK. I think the issue is that's not how ZMQ is designed to work with > respect to REQ/REP. I think it's designed to do "sink -> REQ -> ZMQ -> > REP -> source", so that the REQ is a sink & REP is a source. In a quick > internet search, this model is what I turn up. GR's model is the name > swap of this model; it would work internally to GR, but I won't be able > to connect GR to the outside world (or, at least my application) because > of this swapping. > The ZMQ REQ/REP dataflow has the receiving end REQuest data from the sending end when needed, which REPlies with a packet. It's a form of flow control. >From the GNU Radio perspective, streams flow into GNU Radio sinks to exit the flowgraph, and data is sourced into a flowgraph from a GNU Radio source block. Thus, the correct model is GR Stream | GR REP Sink | ZMQ REP | ZMQ Transport | ZMQ REQ | GR REQ source | GR stream -Johnathan ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
Re: [Discuss-gnuradio] ZMQ REQ / REP naming Swap?
On Thu, Jul 28, 2016, at 03:11 PM, Sylvain Munaut wrote: > > But this model doesn't work for the REP / REQ in Figure 2 > > "Request-Reply". I would expect the ZMQ REQ to be a GR sink but instead > > if is a source; likewise, I'd expect the ZMQ REP to be a GR source & it > > is instead a sink. > > > > The internal coding is consistent with the external naming; I just > > question whether the naming was swapped. Am I missing something? - MLD > > Basically it's coded so that you REQuest some samples from a server. > > So the REQ is the source (it sends a request for data and gets back > some samples that are fed to GR). > And REP obviously then has to be a sink since it waits for a REQuest > and when it gets one, sends some data out of GR in the REPlies. OK. I think the issue is that's not how ZMQ is designed to work with respect to REQ/REP. I think it's designed to do "sink -> REQ -> ZMQ -> REP -> source", so that the REQ is a sink & REP is a source. In a quick internet search, this model is what I turn up. GR's model is the name swap of this model; it would work internally to GR, but I won't be able to connect GR to the outside world (or, at least my application) because of this swapping. I'll write my own OOT that does ZMQ REP/REQ the way I need. Thanks! - MLD ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
Re: [Discuss-gnuradio] ZMQ REQ / REP naming Swap?
> But this model doesn't work for the REP / REQ in Figure 2 > "Request-Reply". I would expect the ZMQ REQ to be a GR sink but instead > if is a source; likewise, I'd expect the ZMQ REP to be a GR source & it > is instead a sink. > > The internal coding is consistent with the external naming; I just > question whether the naming was swapped. Am I missing something? - MLD Basically it's coded so that you REQuest some samples from a server. So the REQ is the source (it sends a request for data and gets back some samples that are fed to GR). And REP obviously then has to be a sink since it waits for a REQuest and when it gets one, sends some data out of GR in the REPlies. Cheers, Sylvain ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
