Re: put vs. send
On 3/6/13 8:27 PM, Rafael Schloming wrote:
On Wed, Mar 6, 2013 at 7:35 AM, Ted Ross wrote:
On 03/06/2013 10:09 AM, Rafael Schloming wrote:
On Wed, Mar 6, 2013 at 6:52 AM, Ted Ross wrote:
On 03/06/2013 08:30 AM, Rafael Schloming wrote:
On Wed, Mar 6, 2013 at 5:15 AM, Ted Ross wrote:
This is exactly right. The API behaves in a surprising way and causes
reasonable programmers to write programs that don't work. For the sake
of
adoption, we should fix this, not merely document it.
This seems like a bit of a leap to me. Have we actually seen anyone
misusing or abusing the API due to this? Mick didn't come across it
till I
pointed it out and even then he had to construct an experiment where
he's
basically observing the over-the-wire behaviour in order to detect it.
--Rafael
The following code doesn't work:
while (True) {
wait_for_and_get_next_event(&event);
pn_messenger_put(event);
}
If I add a send after every put, I'm going to limit my maximum message
rate. If I amortize my sends over every N puts, I may have
arbitrarily/infinitely high latency on messages if the source of events
goes quiet.
I guess I'm questioning the mission of the Messenger API. Which is the
more important design goal: general-purpose ease of use, or strict
single-threaded asynchrony?
I wouldn't say it's a goal to avoid background threads, more of a really
nice thing to avoid if we can, and quite possibly a necessary mode of
operation in certain environments.
I don't think your example code will work though even if there is a
background thread. What do you want to happen when things start backing
up?
Do you want messages to be dropped? Do you want put to start blocking? Do
you just want memory to grow indefinitely?
Good question. I would want to either block so I can stop consuming
events or get an indication that I would-block so I can take other actions.
I understand that this is what "send" is for, but it's not clear when, or
how often I should call it.
I think there are really two orthogonal issues being discussed:
1) how do you get messenger to perform the appropriate amount of
outstanding work once enough has built up
2) should messenger be obligated to eventually complete any outstanding
work even if you never call into it again
The second one basically being the question of whether the API semantics
mandate a background thread or not, and I think (1) needs a good solution
regardless of how you answer (2).
I think we can address (1) by adding an integer argument N to
pn_messenger_send with the following semantics:
- if N is -1, then pn_messenger_send behaves as it currently does, i.e.
block until everything is sent
- if N is 0, then pn_messenger_send will send whatever it can/should
without blocking
- if N is positive, then pn_messenger_send will block until N messages
are sent
I'd also suggest we modify pn_messenger_recv(0) to match this on the recv
side, so the overall semantics of pn_messenger_recv(N) would be:
- if N is -1, then pn_messenger_recv behaves as it currently does, i.e.
block until something is received
- if N is 0, then pn_messenger_recv will receive whatever it can/should
without blocking
- if N is positive, then pn_messenger_recv behaves as it currently does,
i.e. block until something is received but with a limit of N
+1 This looks much nicer.
I think the above change would introduce some symmetry between send and
recv and let you implement a fully pipelined asynchronous sender in a more
obvious way than setting timeouts to zero, e.g. Ted's example could read:
while (True) {
wait_for_and_get_next_event(&**event);
pn_messenger_put(m, event);
if (pn_messenger_outgoing()> threshold) {
pn_messenger_send(m, pn_messenger_outgoing() - threshold);
}
}
We can also add a pn_messenger_work(timeout) that would block until it's
done some kind of work (either sending or receiving). Calling it with a
zero argument would be equivalent to simultaneously calling
pn_messenger_send(0) and pn_messenger_recv(0). (Note that currently send
and recv don't actually factor in any biases towards sending and receiving,
they just do whatever outstanding work needs doing, however the API would
admit an implementation that did apply some bias in those cases.)
What about keeping the timeout in pn_messenger_timeout() and expressing the
above two calls as helpers of one call with two parameters:
pn_messenger_sync(recv_amount, send_amount)
Right now I don't have the time to think through what the X in below
would be,
but 0 looks like a good start:
pn_messenger_recv(n) { return pn_messenger_sync(n, X); }
pn_messenger_send(n) { return pn_messenger_sync(X, n); }
Bozzo
Re: put vs. send
- Original Message -
> From: "Rafael Schloming"
> To: proton@qpid.apache.org
> Sent: Wednesday, March 6, 2013 2:27:19 PM
> Subject: Re: put vs. send
>
> On Wed, Mar 6, 2013 at 7:35 AM, Ted Ross wrote:
>
> >
> > On 03/06/2013 10:09 AM, Rafael Schloming wrote:
> >
> >> On Wed, Mar 6, 2013 at 6:52 AM, Ted Ross wrote:
> >>
> >> On 03/06/2013 08:30 AM, Rafael Schloming wrote:
> >>>
> >>> On Wed, Mar 6, 2013 at 5:15 AM, Ted Ross
> >>> wrote:
> >>>>
> >>>> This is exactly right. The API behaves in a surprising way
> >>>> and causes
> >>>>
> >>>>> reasonable programmers to write programs that don't work. For
> >>>>> the sake
> >>>>> of
> >>>>> adoption, we should fix this, not merely document it.
> >>>>>
> >>>>> This seems like a bit of a leap to me. Have we actually seen
> >>>>> anyone
> >>>> misusing or abusing the API due to this? Mick didn't come across
> >>>> it
> >>>> till I
> >>>> pointed it out and even then he had to construct an experiment
> >>>> where
> >>>> he's
> >>>> basically observing the over-the-wire behaviour in order to
> >>>> detect it.
> >>>>
> >>>> --Rafael
> >>>>
> >>>>
> >>>> The following code doesn't work:
> >>>
> >>> while (True) {
> >>>wait_for_and_get_next_event(&event);
> >>>
> >>>pn_messenger_put(event);
> >>> }
> >>>
> >>> If I add a send after every put, I'm going to limit my maximum
> >>> message
> >>> rate. If I amortize my sends over every N puts, I may have
> >>> arbitrarily/infinitely high latency on messages if the source of
> >>> events
> >>> goes quiet.
> >>>
> >>> I guess I'm questioning the mission of the Messenger API. Which
> >>> is the
> >>> more important design goal: general-purpose ease of use, or
> >>> strict
> >>> single-threaded asynchrony?
> >>>
> >>
> >> I wouldn't say it's a goal to avoid background threads, more of a
> >> really
> >> nice thing to avoid if we can, and quite possibly a necessary mode
> >> of
> >> operation in certain environments.
> >>
> >> I don't think your example code will work though even if there is
> >> a
> >> background thread. What do you want to happen when things start
> >> backing
> >> up?
> >> Do you want messages to be dropped? Do you want put to start
> >> blocking? Do
> >> you just want memory to grow indefinitely?
> >>
> >
> > Good question. I would want to either block so I can stop
> > consuming
> > events or get an indication that I would-block so I can take other
> > actions.
> > I understand that this is what "send" is for, but it's not clear
> > when, or
> > how often I should call it.
> >
>
> I think there are really two orthogonal issues being discussed:
>
> 1) how do you get messenger to perform the appropriate amount of
> outstanding work once enough has built up
> 2) should messenger be obligated to eventually complete any
> outstanding
> work even if you never call into it again
>
> The second one basically being the question of whether the API
> semantics
> mandate a background thread or not, and I think (1) needs a good
> solution
> regardless of how you answer (2).
>
> I think we can address (1) by adding an integer argument N to
> pn_messenger_send with the following semantics:
>
> - if N is -1, then pn_messenger_send behaves as it currently does,
> i.e.
> block until everything is sent
> - if N is 0, then pn_messenger_send will send whatever it
> can/should
> without blocking
> - if N is positive, then pn_messenger_send will block until N
> messages
> are sent
>
> I'd also suggest we modify pn_messenger_recv(0) to match this on the
> recv
> side, so the overall semantics of pn_messenger_recv(N) would be:
>
> - if N is -1, then pn_messenger_recv behaves as it currently does,
> i.e.
> block until something is received
> - if N is 0, then pn_messenger_recv will receive whatever it
> can/should
> without blocking
> - if N is positive, then
Re: put vs. send -- new doc
On Thu, Mar 7, 2013 at 3:18 PM, Michael Goulish wrote: > > > > I think this is actually quite helpful for the ongoing API > > discussions. One > > of the tricky things about developing a simple API is that everyone > > has > > their own scenario that they want to be simple, and sometimes making > > one > > scenario simpler ends up making another one more difficult, so it's > > actually really useful to have them all laid out so concisely so that > > we > > can observe the overall effect of any changes. I'm thinking we should > > expand this with the API changes I proposed in the other thread and > > see how > > they work out. > > > > > > Sure -- like a doc for your recent proposals for changes. > That would be fun, and useful I agree. > But -- probably shouldn't check that into tree, right? > Since it's for hypothetical code. > Just post to list ? > Right, I wouldn't put it in the tree, just go for the list right now. --Rafael
Re: put vs. send -- new doc
- Original Message - > I like this. Good! I'm trying to get at the intention, as I understood it from online discussions, and make a doc that makes the intention easy to see. > I don't think this is the same thing as high level > conceptual > intro, but may well be more useful right now, and it will always be > very > useful as recipe book style documentation. No, this isn't the intro -- I'm re-working that now. I guess I should have said what category of docs this is part of. Except I don't actually know. I guess I have an idea of a set of docs at this level that discuss different topics. i.e. addressing, message management, etc. > > I think this is actually quite helpful for the ongoing API > discussions. One > of the tricky things about developing a simple API is that everyone > has > their own scenario that they want to be simple, and sometimes making > one > scenario simpler ends up making another one more difficult, so it's > actually really useful to have them all laid out so concisely so that > we > can observe the overall effect of any changes. I'm thinking we should > expand this with the API changes I proposed in the other thread and > see how > they work out. Sure -- like a doc for your recent proposals for changes. That would be fun, and useful I agree. But -- probably shouldn't check that into tree, right? Since it's for hypothetical code. Just post to list ? > > --Rafael > > On Wed, Mar 6, 2013 at 9:43 AM, Michael Goulish > wrote: > > > > > OK, I'm trying here to express the spirit of Messenger I/O , > > greatly based on the conversation of the last 24 hrs. > > > > This probably needs some elaboration yet, but I want to > > see if I'm at least generally on the right track. > > > > Oh, please, give me feedback. > > > > > > > > > > Sending and Receiving Messages > > === > > > > The Proton Messenger API provides a mixture of synchronous > > and asynchronous operations to give you flexibility in > > deciding when you application should block waiting for I/O, > > and when it should not. > > > > > > When sending messages, you can: > > > > * send a message immediately, > > * enqueue a message to be sent later, > > * block until all enqueued messages are sent, > > * send enqueued messages until a timeout occurs, or > > * send all messages that can be sent without blocking. > > > > When receiving messages, you can: > > > > * receive messages that can be received without blocking, > > * block until at least one message is received, > > * receive no more than a fixed number of messages. > > > > > > Examples > > -- > > > > 1. send a message immediately > > > >put ( messenger, msg ); > >send ( messenger ); > > > > > > > > 2. enqueue a message to be sent later > > > > put ( messenger, msg ); > > > > note: > > The message will be sent whenever it is not blocked and > > the Messenger code has other I/O work to be done. > > > > > > > > 3. block until all enqueued messages are sent > > > >set_timeout ( messenger, -1 ); > >send ( messenger ); > > > > note: > > A negative timeout means 'forever'. That is the initial > > default for a messenger. > > > > > > > > 4. send enqueued messages until a timeout occurs > > > >set_timeout ( messenger, 100 ); /* 100 msec */ > >send ( messenger ); > > > > > > > > 5. send all messages that can be sent without blocking > > > >set_timeout ( messenger, 0 ); > >send ( messenger ); > > > > > > > > 6. receive messages that can be received without blocking > > > >set_timeout ( messenger, 0 ); > >recv ( messenger, -1 ); > > > > > > > > 7. block until at least one message is received > > > >set_timeout ( messenger, -1 ); > >recv ( messenger, -1 ); > > > > note: -1 is initial messenger default. > >If you didn't change it, you don't need to set it. > > > > > > > > 8. receive no more than a fixed number of messages > > > >recv ( messenger, 10 ); > > > > > > > > >
Re: put vs. send -- new doc
I like this. I don't think this is the same thing as high level conceptual intro, but may well be more useful right now, and it will always be very useful as recipe book style documentation. I think this is actually quite helpful for the ongoing API discussions. One of the tricky things about developing a simple API is that everyone has their own scenario that they want to be simple, and sometimes making one scenario simpler ends up making another one more difficult, so it's actually really useful to have them all laid out so concisely so that we can observe the overall effect of any changes. I'm thinking we should expand this with the API changes I proposed in the other thread and see how they work out. --Rafael On Wed, Mar 6, 2013 at 9:43 AM, Michael Goulish wrote: > > OK, I'm trying here to express the spirit of Messenger I/O , > greatly based on the conversation of the last 24 hrs. > > This probably needs some elaboration yet, but I want to > see if I'm at least generally on the right track. > > Oh, please, give me feedback. > > > > > Sending and Receiving Messages > === > > The Proton Messenger API provides a mixture of synchronous > and asynchronous operations to give you flexibility in > deciding when you application should block waiting for I/O, > and when it should not. > > > When sending messages, you can: > > * send a message immediately, > * enqueue a message to be sent later, > * block until all enqueued messages are sent, > * send enqueued messages until a timeout occurs, or > * send all messages that can be sent without blocking. > > When receiving messages, you can: > > * receive messages that can be received without blocking, > * block until at least one message is received, > * receive no more than a fixed number of messages. > > > Examples > -- > > 1. send a message immediately > >put ( messenger, msg ); >send ( messenger ); > > > > 2. enqueue a message to be sent later > > put ( messenger, msg ); > > note: > The message will be sent whenever it is not blocked and > the Messenger code has other I/O work to be done. > > > > 3. block until all enqueued messages are sent > >set_timeout ( messenger, -1 ); >send ( messenger ); > > note: > A negative timeout means 'forever'. That is the initial > default for a messenger. > > > > 4. send enqueued messages until a timeout occurs > >set_timeout ( messenger, 100 ); /* 100 msec */ >send ( messenger ); > > > > 5. send all messages that can be sent without blocking > >set_timeout ( messenger, 0 ); >send ( messenger ); > > > > 6. receive messages that can be received without blocking > >set_timeout ( messenger, 0 ); >recv ( messenger, -1 ); > > > > 7. block until at least one message is received > >set_timeout ( messenger, -1 ); >recv ( messenger, -1 ); > > note: -1 is initial messenger default. >If you didn't change it, you don't need to set it. > > > > 8. receive no more than a fixed number of messages > >recv ( messenger, 10 ); > > > >
Re: put vs. send
On Wed, Mar 6, 2013 at 7:35 AM, Ted Ross wrote:
>
> On 03/06/2013 10:09 AM, Rafael Schloming wrote:
>
>> On Wed, Mar 6, 2013 at 6:52 AM, Ted Ross wrote:
>>
>> On 03/06/2013 08:30 AM, Rafael Schloming wrote:
>>>
>>> On Wed, Mar 6, 2013 at 5:15 AM, Ted Ross wrote:
This is exactly right. The API behaves in a surprising way and causes
> reasonable programmers to write programs that don't work. For the sake
> of
> adoption, we should fix this, not merely document it.
>
> This seems like a bit of a leap to me. Have we actually seen anyone
misusing or abusing the API due to this? Mick didn't come across it
till I
pointed it out and even then he had to construct an experiment where
he's
basically observing the over-the-wire behaviour in order to detect it.
--Rafael
The following code doesn't work:
>>>
>>> while (True) {
>>>wait_for_and_get_next_event(&event);
>>>
>>>pn_messenger_put(event);
>>> }
>>>
>>> If I add a send after every put, I'm going to limit my maximum message
>>> rate. If I amortize my sends over every N puts, I may have
>>> arbitrarily/infinitely high latency on messages if the source of events
>>> goes quiet.
>>>
>>> I guess I'm questioning the mission of the Messenger API. Which is the
>>> more important design goal: general-purpose ease of use, or strict
>>> single-threaded asynchrony?
>>>
>>
>> I wouldn't say it's a goal to avoid background threads, more of a really
>> nice thing to avoid if we can, and quite possibly a necessary mode of
>> operation in certain environments.
>>
>> I don't think your example code will work though even if there is a
>> background thread. What do you want to happen when things start backing
>> up?
>> Do you want messages to be dropped? Do you want put to start blocking? Do
>> you just want memory to grow indefinitely?
>>
>
> Good question. I would want to either block so I can stop consuming
> events or get an indication that I would-block so I can take other actions.
> I understand that this is what "send" is for, but it's not clear when, or
> how often I should call it.
>
I think there are really two orthogonal issues being discussed:
1) how do you get messenger to perform the appropriate amount of
outstanding work once enough has built up
2) should messenger be obligated to eventually complete any outstanding
work even if you never call into it again
The second one basically being the question of whether the API semantics
mandate a background thread or not, and I think (1) needs a good solution
regardless of how you answer (2).
I think we can address (1) by adding an integer argument N to
pn_messenger_send with the following semantics:
- if N is -1, then pn_messenger_send behaves as it currently does, i.e.
block until everything is sent
- if N is 0, then pn_messenger_send will send whatever it can/should
without blocking
- if N is positive, then pn_messenger_send will block until N messages
are sent
I'd also suggest we modify pn_messenger_recv(0) to match this on the recv
side, so the overall semantics of pn_messenger_recv(N) would be:
- if N is -1, then pn_messenger_recv behaves as it currently does, i.e.
block until something is received
- if N is 0, then pn_messenger_recv will receive whatever it can/should
without blocking
- if N is positive, then pn_messenger_recv behaves as it currently does,
i.e. block until something is received but with a limit of N
I think the above change would introduce some symmetry between send and
recv and let you implement a fully pipelined asynchronous sender in a more
obvious way than setting timeouts to zero, e.g. Ted's example could read:
while (True) {
wait_for_and_get_next_event(&**event);
pn_messenger_put(m, event);
if (pn_messenger_outgoing() > threshold) {
pn_messenger_send(m, pn_messenger_outgoing() - threshold);
}
}
We can also add a pn_messenger_work(timeout) that would block until it's
done some kind of work (either sending or receiving). Calling it with a
zero argument would be equivalent to simultaneously calling
pn_messenger_send(0) and pn_messenger_recv(0). (Note that currently send
and recv don't actually factor in any biases towards sending and receiving,
they just do whatever outstanding work needs doing, however the API would
admit an implementation that did apply some bias in those cases.)
This would go some ways towards addressing issue (2) because the app could
create its own background thread that does work behind the scenes, and in
bindings where threads are readily available, we could have the buttercream
frosting layer supply such a thread if we wished.
--Rafael
Re: put vs. send -- new doc
OK, I'm trying here to express the spirit of Messenger I/O , greatly based on the conversation of the last 24 hrs. This probably needs some elaboration yet, but I want to see if I'm at least generally on the right track. Oh, please, give me feedback. Sending and Receiving Messages === The Proton Messenger API provides a mixture of synchronous and asynchronous operations to give you flexibility in deciding when you application should block waiting for I/O, and when it should not. When sending messages, you can: * send a message immediately, * enqueue a message to be sent later, * block until all enqueued messages are sent, * send enqueued messages until a timeout occurs, or * send all messages that can be sent without blocking. When receiving messages, you can: * receive messages that can be received without blocking, * block until at least one message is received, * receive no more than a fixed number of messages. Examples -- 1. send a message immediately put ( messenger, msg ); send ( messenger ); 2. enqueue a message to be sent later put ( messenger, msg ); note: The message will be sent whenever it is not blocked and the Messenger code has other I/O work to be done. 3. block until all enqueued messages are sent set_timeout ( messenger, -1 ); send ( messenger ); note: A negative timeout means 'forever'. That is the initial default for a messenger. 4. send enqueued messages until a timeout occurs set_timeout ( messenger, 100 ); /* 100 msec */ send ( messenger ); 5. send all messages that can be sent without blocking set_timeout ( messenger, 0 ); send ( messenger ); 6. receive messages that can be received without blocking set_timeout ( messenger, 0 ); recv ( messenger, -1 ); 7. block until at least one message is received set_timeout ( messenger, -1 ); recv ( messenger, -1 ); note: -1 is initial messenger default. If you didn't change it, you don't need to set it. 8. receive no more than a fixed number of messages recv ( messenger, 10 );
Re: put vs. send
On Wed, Mar 6, 2013 at 8:38 AM, Ken Giusti wrote:
>
>
> - Original Message -
> > From: "Ted Ross"
> > To: proton@qpid.apache.org
> > Sent: Wednesday, March 6, 2013 10:35:47 AM
> > Subject: Re: put vs. send
> >
> >
> > On 03/06/2013 10:09 AM, Rafael Schloming wrote:
> > > On Wed, Mar 6, 2013 at 6:52 AM, Ted Ross wrote:
> > >
> > >> On 03/06/2013 08:30 AM, Rafael Schloming wrote:
> > >>
> > >>> On Wed, Mar 6, 2013 at 5:15 AM, Ted Ross
> > >>> wrote:
> > >>>
> > >>> This is exactly right. The API behaves in a surprising way and
> > >>> causes
> > >>>> reasonable programmers to write programs that don't work. For
> > >>>> the sake of
> > >>>> adoption, we should fix this, not merely document it.
> > >>>>
> > >>> This seems like a bit of a leap to me. Have we actually seen
> > >>> anyone
> > >>> misusing or abusing the API due to this? Mick didn't come across
> > >>> it till I
> > >>> pointed it out and even then he had to construct an experiment
> > >>> where he's
> > >>> basically observing the over-the-wire behaviour in order to
> > >>> detect it.
> > >>>
> > >>> --Rafael
> > >>>
> > >>>
> > >> The following code doesn't work:
> > >>
> > >> while (True) {
> > >>wait_for_and_get_next_event(&**event);
> > >>pn_messenger_put(event);
> > >> }
> > >>
> > >> If I add a send after every put, I'm going to limit my maximum
> > >> message
> > >> rate. If I amortize my sends over every N puts, I may have
> > >> arbitrarily/infinitely high latency on messages if the source of
> > >> events
> > >> goes quiet.
> > >>
> > >> I guess I'm questioning the mission of the Messenger API. Which
> > >> is the
> > >> more important design goal: general-purpose ease of use, or
> > >> strict
> > >> single-threaded asynchrony?
> > >
> > > I wouldn't say it's a goal to avoid background threads, more of a
> > > really
> > > nice thing to avoid if we can, and quite possibly a necessary mode
> > > of
> > > operation in certain environments.
> > >
> > > I don't think your example code will work though even if there is a
> > > background thread. What do you want to happen when things start
> > > backing up?
> > > Do you want messages to be dropped? Do you want put to start
> > > blocking? Do
> > > you just want memory to grow indefinitely?
> >
> > Good question. I would want to either block so I can stop consuming
> > events or get an indication that I would-block so I can take other
> > actions. I understand that this is what "send" is for, but it's not
> > clear when, or how often I should call it.
> >
>
> This begs a question that was asked before (by Mick, I believe) - what
> happens if a put() message can _never_ be sent? The destination has gone
> away and will never come back. AFAIK, every further call to send() will
> block due to that stuck message. How should that be dealt with? Use a
> TTL?
>
Send() won't block in this case, if the message is rejected or the
destination doesn't exist you can find out by checking the status via the
tracker you get when you call put.
--Rafael
Re: put vs. send
On Wed, Mar 6, 2013 at 9:01 AM, Michael Goulish wrote:
>
>
> - Original Message -
> >
> >
> > - Original Message -
> > > From: "Ted Ross"
> > > To: proton@qpid.apache.org
> > > Sent: Wednesday, March 6, 2013 10:35:47 AM
> > > Subject: Re: put vs. send
> > >
> > >
> > > On 03/06/2013 10:09 AM, Rafael Schloming wrote:
> > > > On Wed, Mar 6, 2013 at 6:52 AM, Ted Ross
> > > > wrote:
> > > >
> > > >> On 03/06/2013 08:30 AM, Rafael Schloming wrote:
> > > >>
> > > >>> On Wed, Mar 6, 2013 at 5:15 AM, Ted Ross
> > > >>> wrote:
> > > >>>
> > > >>> This is exactly right. The API behaves in a surprising way
> > > >>> and
> > > >>> causes
> > > >>>> reasonable programmers to write programs that don't work. For
> > > >>>> the sake of
> > > >>>> adoption, we should fix this, not merely document it.
> > > >>>>
> > > >>> This seems like a bit of a leap to me. Have we actually seen
> > > >>> anyone
> > > >>> misusing or abusing the API due to this? Mick didn't come
> > > >>> across
> > > >>> it till I
> > > >>> pointed it out and even then he had to construct an experiment
> > > >>> where he's
> > > >>> basically observing the over-the-wire behaviour in order to
> > > >>> detect it.
> > > >>>
> > > >>> --Rafael
> > > >>>
> > > >>>
> > > >> The following code doesn't work:
> > > >>
> > > >> while (True) {
> > > >>wait_for_and_get_next_event(&**event);
> > > >>pn_messenger_put(event);
> > > >> }
> > > >>
> > > >> If I add a send after every put, I'm going to limit my maximum
> > > >> message
> > > >> rate. If I amortize my sends over every N puts, I may have
> > > >> arbitrarily/infinitely high latency on messages if the source of
> > > >> events
> > > >> goes quiet.
> > > >>
> > > >> I guess I'm questioning the mission of the Messenger API. Which
> > > >> is the
> > > >> more important design goal: general-purpose ease of use, or
> > > >> strict
> > > >> single-threaded asynchrony?
> > > >
> > > > I wouldn't say it's a goal to avoid background threads, more of a
> > > > really
> > > > nice thing to avoid if we can, and quite possibly a necessary
> > > > mode
> > > > of
> > > > operation in certain environments.
> > > >
> > > > I don't think your example code will work though even if there is
> > > > a
> > > > background thread. What do you want to happen when things start
> > > > backing up?
> > > > Do you want messages to be dropped? Do you want put to start
> > > > blocking? Do
> > > > you just want memory to grow indefinitely?
> > >
> > > Good question. I would want to either block so I can stop
> > > consuming
> > > events or get an indication that I would-block so I can take other
> > > actions. I understand that this is what "send" is for, but it's
> > > not
> > > clear when, or how often I should call it.
> > >
> >
> > This begs a question that was asked before (by Mick, I believe) -
> > what happens if a put() message can _never_ be sent? The
> > destination has gone away and will never come back. AFAIK, every
> > further call to send() will block due to that stuck message. How
> > should that be dealt with? Use a TTL?
> >
>
>
>
>
> Well, I just tried it.
>
>
> Setup
> ---
>
> Set up a receiver to receive, get, and print out messages in a loop.
> ( receive blocking, i.e. timeout == default )
>
> Then start a sender (default timeout) that will:
>
> 1. put-and-send message 1.
> 2. put message 2.
> 3. countdown 12 seconds before it decides to send message 2.
> 4. send message 2.
>
> While the sender is getting ready to call send() on msg2, I kill the
> receiver.
>
>
>
> Result
> ---
>
> I see the receiver print out message 1. good.
> When the sender has put() msg2 but not yet sent(), I kill receiver.
> Sender calls send() on message2.
> send() returns immediately, return code is 0. (success)
>
>
>
> Analysis
> --
>
> Bummer. Dropped message.
>
If you want reliability you need to check the status of the tracker you get
when you call put(). You would also need to set a nonzero outgoing window
so that messenger actually retains that status.
--Rafael
Re: put vs. send
- Original Message -
>
>
> - Original Message -
> > From: "Ted Ross"
> > To: proton@qpid.apache.org
> > Sent: Wednesday, March 6, 2013 10:35:47 AM
> > Subject: Re: put vs. send
> >
> >
> > On 03/06/2013 10:09 AM, Rafael Schloming wrote:
> > > On Wed, Mar 6, 2013 at 6:52 AM, Ted Ross
> > > wrote:
> > >
> > >> On 03/06/2013 08:30 AM, Rafael Schloming wrote:
> > >>
> > >>> On Wed, Mar 6, 2013 at 5:15 AM, Ted Ross
> > >>> wrote:
> > >>>
> > >>> This is exactly right. The API behaves in a surprising way
> > >>> and
> > >>> causes
> > >>>> reasonable programmers to write programs that don't work. For
> > >>>> the sake of
> > >>>> adoption, we should fix this, not merely document it.
> > >>>>
> > >>> This seems like a bit of a leap to me. Have we actually seen
> > >>> anyone
> > >>> misusing or abusing the API due to this? Mick didn't come
> > >>> across
> > >>> it till I
> > >>> pointed it out and even then he had to construct an experiment
> > >>> where he's
> > >>> basically observing the over-the-wire behaviour in order to
> > >>> detect it.
> > >>>
> > >>> --Rafael
> > >>>
> > >>>
> > >> The following code doesn't work:
> > >>
> > >> while (True) {
> > >>wait_for_and_get_next_event(&**event);
> > >>pn_messenger_put(event);
> > >> }
> > >>
> > >> If I add a send after every put, I'm going to limit my maximum
> > >> message
> > >> rate. If I amortize my sends over every N puts, I may have
> > >> arbitrarily/infinitely high latency on messages if the source of
> > >> events
> > >> goes quiet.
> > >>
> > >> I guess I'm questioning the mission of the Messenger API. Which
> > >> is the
> > >> more important design goal: general-purpose ease of use, or
> > >> strict
> > >> single-threaded asynchrony?
> > >
> > > I wouldn't say it's a goal to avoid background threads, more of a
> > > really
> > > nice thing to avoid if we can, and quite possibly a necessary
> > > mode
> > > of
> > > operation in certain environments.
> > >
> > > I don't think your example code will work though even if there is
> > > a
> > > background thread. What do you want to happen when things start
> > > backing up?
> > > Do you want messages to be dropped? Do you want put to start
> > > blocking? Do
> > > you just want memory to grow indefinitely?
> >
> > Good question. I would want to either block so I can stop
> > consuming
> > events or get an indication that I would-block so I can take other
> > actions. I understand that this is what "send" is for, but it's
> > not
> > clear when, or how often I should call it.
> >
>
> This begs a question that was asked before (by Mick, I believe) -
> what happens if a put() message can _never_ be sent? The
> destination has gone away and will never come back. AFAIK, every
> further call to send() will block due to that stuck message. How
> should that be dealt with? Use a TTL?
>
Well, I just tried it.
Setup
---
Set up a receiver to receive, get, and print out messages in a loop.
( receive blocking, i.e. timeout == default )
Then start a sender (default timeout) that will:
1. put-and-send message 1.
2. put message 2.
3. countdown 12 seconds before it decides to send message 2.
4. send message 2.
While the sender is getting ready to call send() on msg2, I kill the receiver.
Result
---
I see the receiver print out message 1. good.
When the sender has put() msg2 but not yet sent(), I kill receiver.
Sender calls send() on message2.
send() returns immediately, return code is 0. (success)
Analysis
--
Bummer. Dropped message.
>
> > >
> > > --Rafael
> > >
> >
> >
>
> --
> -K
>
Re: put vs. send
On Wed, Mar 6, 2013 at 11:37 AM, Rajith Attapattu wrote:
> On Wed, Mar 6, 2013 at 10:09 AM, Rafael Schloming wrote:
>> On Wed, Mar 6, 2013 at 6:52 AM, Ted Ross wrote:
>>
>>>
>>> On 03/06/2013 08:30 AM, Rafael Schloming wrote:
>>>
On Wed, Mar 6, 2013 at 5:15 AM, Ted Ross wrote:
This is exactly right. The API behaves in a surprising way and causes
> reasonable programmers to write programs that don't work. For the sake of
> adoption, we should fix this, not merely document it.
>
This seems like a bit of a leap to me. Have we actually seen anyone
misusing or abusing the API due to this? Mick didn't come across it till I
pointed it out and even then he had to construct an experiment where he's
basically observing the over-the-wire behaviour in order to detect it.
--Rafael
>>> The following code doesn't work:
>>>
>>> while (True) {
>>> wait_for_and_get_next_event(&**event);
>>> pn_messenger_put(event);
>>> }
>>>
>>> If I add a send after every put, I'm going to limit my maximum message
>>> rate. If I amortize my sends over every N puts, I may have
>>> arbitrarily/infinitely high latency on messages if the source of events
>>> goes quiet.
Having a background thread in the Messenger will only push this
problem from your application to the Messenger implementation.
Furthermore you will be at the mercy of the particulars of the client
library implementation as to how this background thread will take care
of the "outstanding work".
We could provide all kinds of knobs to tweak and tune this behaviour,
but I'd be far more comfortable if I as the application developer can
be in control of when the "flush" happens.
Either way you will have arbitrarily/infinitely high latency due to
complications at the TCP stack or the OS level. But you can at least
help your case a bit by having the application issue the flush than
letting the messenger doing it, bcos the application is in a better
position to determine what are the optimal conditions for doing so and
those conditions could be other than time, msg or byte count.
> You can employ a timer along with your event count (or based on a byte
> count) to get around that problem.
> The timer will ensure you flush events when there isn't enough activity.
> Isn't that acceptable ?
>
>>> I guess I'm questioning the mission of the Messenger API. Which is the
>>> more important design goal: general-purpose ease of use, or strict
>>> single-threaded asynchrony?
>>
>>
>> I wouldn't say it's a goal to avoid background threads, more of a really
>> nice thing to avoid if we can, and quite possibly a necessary mode of
>> operation in certain environments.
>>
>> I don't think your example code will work though even if there is a
>> background thread.
>
> This is a key point I missed when I thought about the problem along
> the same lines as Ted.
> Having a background thread cannot guarantee that your messages will be
> written on to the wire as that thread can be blocked due to TCP
> buffers being full or the thread being suppressed in favour of another
> more higher priority thread (for longer than you desire) thus
> increasing your latency beyond acceptable limits.
> You will invariably have outliers in your latency graph.
>
> On the other hand the library code will be much more simpler without
> the background thread.
>
>> What do you want to happen when things start backing up?
>> Do you want messages to be dropped? Do you want put to start blocking? Do
>> you just want memory to grow indefinitely?
>>
>> --Rafael
Re: put vs. send
- Original Message -
> From: "Ted Ross"
> To: proton@qpid.apache.org
> Sent: Wednesday, March 6, 2013 10:35:47 AM
> Subject: Re: put vs. send
>
>
> On 03/06/2013 10:09 AM, Rafael Schloming wrote:
> > On Wed, Mar 6, 2013 at 6:52 AM, Ted Ross wrote:
> >
> >> On 03/06/2013 08:30 AM, Rafael Schloming wrote:
> >>
> >>> On Wed, Mar 6, 2013 at 5:15 AM, Ted Ross
> >>> wrote:
> >>>
> >>> This is exactly right. The API behaves in a surprising way and
> >>> causes
> >>>> reasonable programmers to write programs that don't work. For
> >>>> the sake of
> >>>> adoption, we should fix this, not merely document it.
> >>>>
> >>> This seems like a bit of a leap to me. Have we actually seen
> >>> anyone
> >>> misusing or abusing the API due to this? Mick didn't come across
> >>> it till I
> >>> pointed it out and even then he had to construct an experiment
> >>> where he's
> >>> basically observing the over-the-wire behaviour in order to
> >>> detect it.
> >>>
> >>> --Rafael
> >>>
> >>>
> >> The following code doesn't work:
> >>
> >> while (True) {
> >>wait_for_and_get_next_event(&**event);
> >>pn_messenger_put(event);
> >> }
> >>
> >> If I add a send after every put, I'm going to limit my maximum
> >> message
> >> rate. If I amortize my sends over every N puts, I may have
> >> arbitrarily/infinitely high latency on messages if the source of
> >> events
> >> goes quiet.
> >>
> >> I guess I'm questioning the mission of the Messenger API. Which
> >> is the
> >> more important design goal: general-purpose ease of use, or
> >> strict
> >> single-threaded asynchrony?
> >
> > I wouldn't say it's a goal to avoid background threads, more of a
> > really
> > nice thing to avoid if we can, and quite possibly a necessary mode
> > of
> > operation in certain environments.
> >
> > I don't think your example code will work though even if there is a
> > background thread. What do you want to happen when things start
> > backing up?
> > Do you want messages to be dropped? Do you want put to start
> > blocking? Do
> > you just want memory to grow indefinitely?
>
> Good question. I would want to either block so I can stop consuming
> events or get an indication that I would-block so I can take other
> actions. I understand that this is what "send" is for, but it's not
> clear when, or how often I should call it.
>
This begs a question that was asked before (by Mick, I believe) - what happens
if a put() message can _never_ be sent? The destination has gone away and will
never come back. AFAIK, every further call to send() will block due to that
stuck message. How should that be dealt with? Use a TTL?
> >
> > --Rafael
> >
>
>
--
-K
Re: put vs. send
On Wed, Mar 6, 2013 at 10:09 AM, Rafael Schloming wrote:
> On Wed, Mar 6, 2013 at 6:52 AM, Ted Ross wrote:
>
>>
>> On 03/06/2013 08:30 AM, Rafael Schloming wrote:
>>
>>> On Wed, Mar 6, 2013 at 5:15 AM, Ted Ross wrote:
>>>
>>> This is exactly right. The API behaves in a surprising way and causes
reasonable programmers to write programs that don't work. For the sake of
adoption, we should fix this, not merely document it.
>>>
>>> This seems like a bit of a leap to me. Have we actually seen anyone
>>> misusing or abusing the API due to this? Mick didn't come across it till I
>>> pointed it out and even then he had to construct an experiment where he's
>>> basically observing the over-the-wire behaviour in order to detect it.
>>>
>>> --Rafael
>>>
>>>
>> The following code doesn't work:
>>
>> while (True) {
>> wait_for_and_get_next_event(&**event);
>> pn_messenger_put(event);
>> }
>>
>> If I add a send after every put, I'm going to limit my maximum message
>> rate. If I amortize my sends over every N puts, I may have
>> arbitrarily/infinitely high latency on messages if the source of events
>> goes quiet.
You can employ a timer along with your event count (or based on a byte
count) to get around that problem.
The timer will ensure you flush events when there isn't enough activity.
Isn't that acceptable ?
>> I guess I'm questioning the mission of the Messenger API. Which is the
>> more important design goal: general-purpose ease of use, or strict
>> single-threaded asynchrony?
>
>
> I wouldn't say it's a goal to avoid background threads, more of a really
> nice thing to avoid if we can, and quite possibly a necessary mode of
> operation in certain environments.
>
> I don't think your example code will work though even if there is a
> background thread.
This is a key point I missed when I thought about the problem along
the same lines as Ted.
Having a background thread cannot guarantee that your messages will be
written on to the wire as that thread can be blocked due to TCP
buffers being full or the thread being suppressed in favour of another
more higher priority thread (for longer than you desire) thus
increasing your latency beyond acceptable limits.
You will invariably have outliers in your latency graph.
On the other hand the library code will be much more simpler without
the background thread.
> What do you want to happen when things start backing up?
> Do you want messages to be dropped? Do you want put to start blocking? Do
> you just want memory to grow indefinitely?
>
> --Rafael
Re: put vs. send
On 03/06/2013 10:09 AM, Rafael Schloming wrote:
On Wed, Mar 6, 2013 at 6:52 AM, Ted Ross wrote:
On 03/06/2013 08:30 AM, Rafael Schloming wrote:
On Wed, Mar 6, 2013 at 5:15 AM, Ted Ross wrote:
This is exactly right. The API behaves in a surprising way and causes
reasonable programmers to write programs that don't work. For the sake of
adoption, we should fix this, not merely document it.
This seems like a bit of a leap to me. Have we actually seen anyone
misusing or abusing the API due to this? Mick didn't come across it till I
pointed it out and even then he had to construct an experiment where he's
basically observing the over-the-wire behaviour in order to detect it.
--Rafael
The following code doesn't work:
while (True) {
wait_for_and_get_next_event(&**event);
pn_messenger_put(event);
}
If I add a send after every put, I'm going to limit my maximum message
rate. If I amortize my sends over every N puts, I may have
arbitrarily/infinitely high latency on messages if the source of events
goes quiet.
I guess I'm questioning the mission of the Messenger API. Which is the
more important design goal: general-purpose ease of use, or strict
single-threaded asynchrony?
I wouldn't say it's a goal to avoid background threads, more of a really
nice thing to avoid if we can, and quite possibly a necessary mode of
operation in certain environments.
I don't think your example code will work though even if there is a
background thread. What do you want to happen when things start backing up?
Do you want messages to be dropped? Do you want put to start blocking? Do
you just want memory to grow indefinitely?
Good question. I would want to either block so I can stop consuming
events or get an indication that I would-block so I can take other
actions. I understand that this is what "send" is for, but it's not
clear when, or how often I should call it.
--Rafael
Re: put vs. send
On Wed, Mar 6, 2013 at 6:52 AM, Ted Ross wrote:
>
> On 03/06/2013 08:30 AM, Rafael Schloming wrote:
>
>> On Wed, Mar 6, 2013 at 5:15 AM, Ted Ross wrote:
>>
>> This is exactly right. The API behaves in a surprising way and causes
>>> reasonable programmers to write programs that don't work. For the sake of
>>> adoption, we should fix this, not merely document it.
>>>
>>
>> This seems like a bit of a leap to me. Have we actually seen anyone
>> misusing or abusing the API due to this? Mick didn't come across it till I
>> pointed it out and even then he had to construct an experiment where he's
>> basically observing the over-the-wire behaviour in order to detect it.
>>
>> --Rafael
>>
>>
> The following code doesn't work:
>
> while (True) {
> wait_for_and_get_next_event(&**event);
> pn_messenger_put(event);
> }
>
> If I add a send after every put, I'm going to limit my maximum message
> rate. If I amortize my sends over every N puts, I may have
> arbitrarily/infinitely high latency on messages if the source of events
> goes quiet.
>
> I guess I'm questioning the mission of the Messenger API. Which is the
> more important design goal: general-purpose ease of use, or strict
> single-threaded asynchrony?
I wouldn't say it's a goal to avoid background threads, more of a really
nice thing to avoid if we can, and quite possibly a necessary mode of
operation in certain environments.
I don't think your example code will work though even if there is a
background thread. What do you want to happen when things start backing up?
Do you want messages to be dropped? Do you want put to start blocking? Do
you just want memory to grow indefinitely?
--Rafael
Re: put vs. send
On Wed, Mar 6, 2013 at 6:37 AM, Rob Godfrey wrote: > > > > Whether that's reported as an error is really a choice of the bindings. > In > > C it's all just return codes. We could add a separate non-blocking flag > > that causes the blocking operations to return distinct error codes, i.e. > > the equivalent of EWOULDBLOCK, but I don't think this makes a whole lot > of > > sense in C. I can buy that in the higher level bindings the extra flag > > would tell the API whether to signal timeout by returning false vs > throwing > > an exception. > > > > With the Java implementation (not just the binding) I would expect an > (expensive) exception to be thrown here. I don't think you should be > triggering an exception for a non-exceptional condition. > How do you decide whether it's an exceptional condition or not? It seems like it's really down to how the app is designed as to whether timing out is normal or exceptional. > > > I do agree that we'll want a work interface at some point, but I've been > > thinking that would not just do the work, but also tell you what work has > > been done, so you can, e.g., go check whatever tracker statuses may have > > been updated. > > > > Yeah - i think what you are currently suggesting is more of a "you can > get round the lack of an explicit API because this sort of does the > same thing if you squint at it". Calling a blocking method with a > zero timeout is a hack to cover the lack of a method for the desired > semantic. Moreover if this is a recommended use case for send then I > think you'd need to document it, which would really muddy the waters > as to what "send" is. I'm suggesting it as a way to avoid adding a do_work() call because I'm not actually clear on how you would use the latter without busy looping or what scenarios you would document its use for. I'm not saying there aren't any, but it's not obvious to me right now. If you imagine the split between the non blocking and blocking portions of the API, where all the blocking portions are of the form do_work_and_block_until_condition_X_is_met, we now have two conditions: - the outgoing queue is empty - the incoming queue is non empty What you're asking for is to add a third condition that is always true, and I can possibly buy that for logical completeness, but in terms of usefulness I actually think expanding the set of conditions is actually more interesting, e.g. adding something like "the outgoing queue is < N" perhaps via an optional parameter to send would seem to have a direct and obvious use for pipelined publishing in a way that wouldn't require busy looping. --Rafael
Re: put vs. send
On 03/06/2013 08:30 AM, Rafael Schloming wrote:
On Wed, Mar 6, 2013 at 5:15 AM, Ted Ross wrote:
This is exactly right. The API behaves in a surprising way and causes
reasonable programmers to write programs that don't work. For the sake of
adoption, we should fix this, not merely document it.
This seems like a bit of a leap to me. Have we actually seen anyone
misusing or abusing the API due to this? Mick didn't come across it till I
pointed it out and even then he had to construct an experiment where he's
basically observing the over-the-wire behaviour in order to detect it.
--Rafael
The following code doesn't work:
while (True) {
wait_for_and_get_next_event(&event);
pn_messenger_put(event);
}
If I add a send after every put, I'm going to limit my maximum message
rate. If I amortize my sends over every N puts, I may have
arbitrarily/infinitely high latency on messages if the source of events
goes quiet.
I guess I'm questioning the mission of the Messenger API. Which is the
more important design goal: general-purpose ease of use, or strict
single-threaded asynchrony?
-Ted
Re: put vs. send
> > Whether that's reported as an error is really a choice of the bindings. In > C it's all just return codes. We could add a separate non-blocking flag > that causes the blocking operations to return distinct error codes, i.e. > the equivalent of EWOULDBLOCK, but I don't think this makes a whole lot of > sense in C. I can buy that in the higher level bindings the extra flag > would tell the API whether to signal timeout by returning false vs throwing > an exception. > With the Java implementation (not just the binding) I would expect an (expensive) exception to be thrown here. I don't think you should be triggering an exception for a non-exceptional condition. > I do agree that we'll want a work interface at some point, but I've been > thinking that would not just do the work, but also tell you what work has > been done, so you can, e.g., go check whatever tracker statuses may have > been updated. > Yeah - i think what you are currently suggesting is more of a "you can get round the lack of an explicit API because this sort of does the same thing if you squint at it". Calling a blocking method with a zero timeout is a hack to cover the lack of a method for the desired semantic. Moreover if this is a recommended use case for send then I think you'd need to document it, which would really muddy the waters as to what "send" is. -- Rob
Re: put vs. send
On Wed, Mar 6, 2013 at 5:32 AM, Rob Godfrey wrote:
> On 6 March 2013 13:26, Rafael Schloming wrote:
> > On Wed, Mar 6, 2013 at 1:44 AM, Rob Godfrey
> wrote:
> >
> >> On 5 March 2013 21:10, Rafael Schloming wrote:
> >> > On Tue, Mar 5, 2013 at 11:24 AM, Ted Ross wrote:
> >>
> >> [.. snip ..]
> >>
> >> >
> >> > It isn't really possible to have "put" cause messages to be eventually
> >> sent
> >> > without a background thread, something we don't currently have.
> >>
> >> I think it's this that is what makes me find the API slightly odd.
> >> That put is an asynchronous operation is fine, but the fact that the
> >> only way to get work to occur is for a synchronous operation to be
> >> called seems a little screwy.
> >>
> >> If I understand correctly, right now an application programmer cannot
> >> actually write an "asynchronous publisher", every so often they would
> >> have to call some form of synchronous operation.
> >>
> >> At the very least it would seem to suggest there might be call for a
> >> "do some work but don't block" function in the API. This could either
> >> take an aggressive strategy of flushing everything that it can to the
> >> wire, or it could attempt to optimize into larger transmission units.
> >
> >
> > This is exactly what happens when you set the timeout to zero and call
> send
> > (or recv). Are you saying you want some other way of doing the same thing
> > or you want a background thread?
>
> Surely though setting timeout to 0 and calling send results in
> something that looks like an error ("this timed out"). On a Java
> implementation I would expect this to throw an exception. That's not
> really the semantic I'm expecting. The semantic is "do some work if
> you can without blocking".
>
Whether that's reported as an error is really a choice of the bindings. In
C it's all just return codes. We could add a separate non-blocking flag
that causes the blocking operations to return distinct error codes, i.e.
the equivalent of EWOULDBLOCK, but I don't think this makes a whole lot of
sense in C. I can buy that in the higher level bindings the extra flag
would tell the API whether to signal timeout by returning false vs throwing
an exception.
I do agree that we'll want a work interface at some point, but I've been
thinking that would not just do the work, but also tell you what work has
been done, so you can, e.g., go check whatever tracker statuses may have
been updated.
--Rafael
Re: put vs. send
On 6 March 2013 13:26, Rafael Schloming wrote:
> On Wed, Mar 6, 2013 at 1:44 AM, Rob Godfrey wrote:
>
>> On 5 March 2013 21:10, Rafael Schloming wrote:
>> > On Tue, Mar 5, 2013 at 11:24 AM, Ted Ross wrote:
>>
>> [.. snip ..]
>>
>> >
>> > It isn't really possible to have "put" cause messages to be eventually
>> sent
>> > without a background thread, something we don't currently have.
>>
>> I think it's this that is what makes me find the API slightly odd.
>> That put is an asynchronous operation is fine, but the fact that the
>> only way to get work to occur is for a synchronous operation to be
>> called seems a little screwy.
>>
>> If I understand correctly, right now an application programmer cannot
>> actually write an "asynchronous publisher", every so often they would
>> have to call some form of synchronous operation.
>>
>> At the very least it would seem to suggest there might be call for a
>> "do some work but don't block" function in the API. This could either
>> take an aggressive strategy of flushing everything that it can to the
>> wire, or it could attempt to optimize into larger transmission units.
>
>
> This is exactly what happens when you set the timeout to zero and call send
> (or recv). Are you saying you want some other way of doing the same thing
> or you want a background thread?
Surely though setting timeout to 0 and calling send results in
something that looks like an error ("this timed out"). On a Java
implementation I would expect this to throw an exception. That's not
really the semantic I'm expecting. The semantic is "do some work if
you can without blocking".
-- Rob
>
> --Rafael
Re: put vs. send
On Wed, Mar 6, 2013 at 5:15 AM, Ted Ross wrote: > This is exactly right. The API behaves in a surprising way and causes > reasonable programmers to write programs that don't work. For the sake of > adoption, we should fix this, not merely document it. This seems like a bit of a leap to me. Have we actually seen anyone misusing or abusing the API due to this? Mick didn't come across it till I pointed it out and even then he had to construct an experiment where he's basically observing the over-the-wire behaviour in order to detect it. --Rafael
Re: put vs. send
On Wed, Mar 6, 2013 at 1:44 AM, Rob Godfrey wrote: > On 5 March 2013 21:10, Rafael Schloming wrote: > > On Tue, Mar 5, 2013 at 11:24 AM, Ted Ross wrote: > > [.. snip ..] > > > > > It isn't really possible to have "put" cause messages to be eventually > sent > > without a background thread, something we don't currently have. > > I think it's this that is what makes me find the API slightly odd. > That put is an asynchronous operation is fine, but the fact that the > only way to get work to occur is for a synchronous operation to be > called seems a little screwy. > > If I understand correctly, right now an application programmer cannot > actually write an "asynchronous publisher", every so often they would > have to call some form of synchronous operation. > > At the very least it would seem to suggest there might be call for a > "do some work but don't block" function in the API. This could either > take an aggressive strategy of flushing everything that it can to the > wire, or it could attempt to optimize into larger transmission units. This is exactly what happens when you set the timeout to zero and call send (or recv). Are you saying you want some other way of doing the same thing or you want a background thread? --Rafael
Re: put vs. send
Hah! I think I get it! Your comments about asynchronicity were the key. Rewriting now. - Original Message - > On Tue, Mar 5, 2013 at 1:50 PM, Michael Goulish > wrote: > > > > > > > - Original Message - > > > On Tue, Mar 5, 2013 at 3:20 PM, Rafael Schloming > > > > > > wrote: > > > > On Tue, Mar 5, 2013 at 11:33 AM, Rajith Attapattu > > > > wrote: > > > > > > > >> On Tue, Mar 5, 2013 at 2:24 PM, Ted Ross > > > >> wrote: > > > >> > > > > >> > On 03/05/2013 02:14 PM, Rajith Attapattu wrote: > > > >> >> > > > >> >> > > > >> >> This is a good explanation that we need to put in the docs, > > > >> >> as > > > >> >> Application developers certainly need to know how it > > > >> >> behaves. > > > >> >> If one were to use the current C impl, it certainly gives > > > >> >> the > > > >> >> impression that put() is meant to write messages into your > > > >> >> internal > > > >> >> buffer and send() will actually write it to the wire. > > > >> >> Unfortunately some applications will depend on this > > > >> >> behaviour, > > > >> >> even > > > >> >> though it's not advisable > > > >> >> > > > >> >> If we are to change from say #2 to #1 or even #3 we need to > > > >> >> release > > > >> >> note it prominently. > > > >> >> > > > >> >> I think the best solution is to make this behaviour > > > >> >> configurable, and > > > >> >> advertise the default very prominently. > > > >> >> This way application developers will know exactly what they > > > >> >> are > > > >> >> getting instead of us making changes underneath. > > > >> >> > > > >> >> Rajith > > > >> >> > > > >> > > > > >> > Making this configurable multiplies the size of the test > > > >> > matrix. > > > >> > Can't > > > >> we > > > >> > make this simpler? > > > >> > > > >> I do understand your concern here, but the Java impl already > > > >> does > > > >> both > > > >> #1 and #2 and Rafi wants to do #3 in the future. > > > >> The old JMS client does something similar. > > > >> > > > >> I agree that if we just do option #2 (as you suggest below), > > > >> then > > > >> the > > > >> application can easily do #1 and #3 on top of that. > > > >> But I'm sure they will like if the library implements those > > > >> strategies > > > >> for them and they have the ability to pick a strategy. > > > >> > > > > > > > > I don't see why we'd make this configurable. All three options > > > > actually > > > > fit the same general semantics. Even if you're optimistically > > > > trying to > > > > transmit every single time put is called it's entirely possible > > > > for > > > > the > > > > socket to be blocked every single time you try. If this were to > > > > happen the > > > > implementation of #1 would appear to behave precisely the same > > > > as > > > > #2 > > > > behaves. > > > > > > > > In other words if you're coding correctly against the API you > > > > can't > > > > assume > > > > that put will or won't have transmitted anything regardless of > > > > which > > > > strategy is used internally. > > > > > > > > > > I agree with you. You make a very good point. > > > Perhaps we should explicitly make that clear in our docs to avoid > > > applications written against wrong assumptions. > > > > > > I can certainly do that, but it seems to me that semantics should > > be > > simple, obvious, and orthogonal. > > > > What seems non-simple and non-obvious to me so far is: > > > > put() might send, or not. It doesn't send now, but it > > might later. > > > > This behaviour is fundamental to an asynchronous API. You're not > actually > doing things, you're scheduling things to be done asynchronously. > This is > why put() returns a tracker so you can come back and check on the > status of > your asynchronous operation. > > recv() can cause messages to be sent. > > > > send() can cause messages to be received. > > > > I don't think that's a correct way of describing what is going on. > You > scheduled an asynchronous operation via put(). That means it can > occur at > any point later on. The fact that it happens to be trigger by the > recv() in > the example I gave is simply because recv() is blocking waiting for > the > reply and so it is inevitably going to end up blocking until the > request is > sent because the reply won't be triggered until after the request is > sent. > > As for send(), it's simply inaccurate to say that send causes > messages to > be received. Messages can be spontaneously sent by remote parties at > any > time (given sufficient credit has been previously granted). What > caused > them to be received is the other party actually sending them, and if > message data happens to arrived while we're inside a call to send(), > we > can't simply throw those messages away, so they go onto the incoming > queue > just as if they had arrived during a call to recv(). > > > > I would think that > > > > 1. every verb should only mean one thing > > > > 2. there should be a simple mental model, > > against which every verb perf
Re: put vs. send
On 03/06/2013 04:44 AM, Rob Godfrey wrote: On 5 March 2013 21:10, Rafael Schloming wrote: On Tue, Mar 5, 2013 at 11:24 AM, Ted Ross wrote: [.. snip ..] It isn't really possible to have "put" cause messages to be eventually sent without a background thread, something we don't currently have. I think it's this that is what makes me find the API slightly odd. That put is an asynchronous operation is fine, but the fact that the only way to get work to occur is for a synchronous operation to be called seems a little screwy. This is exactly right. The API behaves in a surprising way and causes reasonable programmers to write programs that don't work. For the sake of adoption, we should fix this, not merely document it. If I understand correctly, right now an application programmer cannot actually write an "asynchronous publisher", every so often they would have to call some form of synchronous operation. +1 At the very least it would seem to suggest there might be call for a "do some work but don't block" function in the API. This could either take an aggressive strategy of flushing everything that it can to the wire, or it could attempt to optimize into larger transmission units. -- Rob
Re: put vs. send
On 5 March 2013 21:10, Rafael Schloming wrote: > On Tue, Mar 5, 2013 at 11:24 AM, Ted Ross wrote: [.. snip ..] > > It isn't really possible to have "put" cause messages to be eventually sent > without a background thread, something we don't currently have. I think it's this that is what makes me find the API slightly odd. That put is an asynchronous operation is fine, but the fact that the only way to get work to occur is for a synchronous operation to be called seems a little screwy. If I understand correctly, right now an application programmer cannot actually write an "asynchronous publisher", every so often they would have to call some form of synchronous operation. At the very least it would seem to suggest there might be call for a "do some work but don't block" function in the API. This could either take an aggressive strategy of flushing everything that it can to the wire, or it could attempt to optimize into larger transmission units. -- Rob
Re: put vs. send
On 3/5/13 8:52 PM, Rafael Schloming wrote:
On Tue, Mar 5, 2013 at 11:10 AM, Ted Ross wrote:
On 03/05/2013 02:01 PM, Rafael Schloming wrote:
On Tue, Mar 5, 2013 at 10:42 AM, Michael Goulish
wrote:
So, am I understanding correctly? -- I should be able to get messages
from my sender to my receiver just by calling put() -- if the receiver
is ready to receive?
Not necessarily, the receiver being ready just means you are unblocked
on
AMQP level flow control. You could also potentially block on the socket
write (i.e. TCP level flow control). You need to be unblocked on both for
put to succeed.
Certainly there is no TCP flow control happening in Mick's scenario.
What I said was put is *allowed* to send optimistically, not that it is
required to. It actually did send optimistically in a previous version of
the code, however I commented that line out.
I would say the documented semantics of put and send should allow the
implementation the flexibility to do any of the following:
1) optimistically transmit whatever it can everytime so long as it
doesn't block
2) never bother transmitting anything until you force it to by calling
send
3) anything in between the first two, e.g. magically transmit once
you've
put enough messages to reach the optimal batch size
The reason for the behaviour you are observing is that we currently do
option 2 in the C impl, however we've done option 1 in the past (and I
think we do option 1 still in the Java impl), and we will probably do
option 3 in the future.
If this is the case, then Mick's original view is correct. The
application must assume that messages will not ever be sent unless "send"
is called. There is no flowing, pipelined, non-blocking producer.
It's not correct as documentation of the API semantics. It's also not
correct to say that messages will never be sent unless "send" is called,
e.g. the following code will work fine:
Client:
m.put(request);
m.recv(); // wait for reply
m.get(reply);
Server:
while True:
m.recv(); // wait for request
m.get(request)
m.put(reply);
Wow, by not ever calling pn_messenger_send(), but only pn_messenger_recv()
things are, unexpectedly, better! I'll explain below what 'better' means.
But first, this begs the question, what is the purpose of
pn_messenger_send()
and where (and why) it's appropriate/required to call it.
The results are for slightly modified 0.3 release.
Most notably, I have a local change that exposes pn_driver_wakeup()
through the messenger api.
Our API is threaded internally but all proton communication is done
via a dedicated thread that runs the following (stub) event loop:
(The same event loop is used by both, the client library, and the 'broker')
while(1) {
ret = pn_messenger_recv(m,100) // the 100 is hard to explain...
if (ret != PN_WAKED_UP) {// new error code for wakeup case
/*
* apparently there's no need to call send...
* pn_messenger_send(m);
*/
}
Command cmd = cmd_queue.get(); // cmd_queue.put() from another thread
// will call pn_driver_wakeup()
and will
// break out of pn_messenger_recv()
if (cmd)
handle(cmd); // ends up calling pn_messenger_put()
if (pn_messenger_incoming(m)) {
msg = pn_messenger_get(m);// handle just one message
// pn_messenger_recv() will not block
// until we're done
handle(msg); // can end up calling
pn_messenger_put()
}
}
So, before the change, a test client that produced messages needed to
throttle a bit, about 8ms between each 'command' that resulted in
one 'pn_messenger_put()'
If a lower delay (or no delay) was used, the client's messenger got confused
after some fairly small number of messages sent (order of magnitude 10)
and ended up sitting in pn_driver_wait while it had unsent messages to send.
With the one line change of commenting out the send() it can go full speed!
I know it's hard to comment on out-of-tree modified pseudo code, but
is such an event loop within the design goals of the messenger?
Longer term we'll most likely be switching from messenger to engine + driver
so we can go multithreaded with the event loop.
I also did the reverse experiment, just for fun, commented out the recv()
and left just send(). The effect is absolutely not the same. The
'broker' which expects
first to get a message goes into a wild spin inside the above loop,
consuming 100%cpu and never receives even one message (kinda expected).
The clients spin too a bit, until they receive first command (and do a
put())
after that they settle into pn_driver_wait() but never make any progress
since
broker is so busy not listening to them :)
Bozzo
As for there being "no flowing, pipelined, non-blocking producer," that
seems like an orthogonal issue,
Re: put vs. send
On Tue, Mar 5, 2013 at 1:50 PM, Michael Goulish wrote: > > > - Original Message - > > On Tue, Mar 5, 2013 at 3:20 PM, Rafael Schloming > > wrote: > > > On Tue, Mar 5, 2013 at 11:33 AM, Rajith Attapattu > > > wrote: > > > > > >> On Tue, Mar 5, 2013 at 2:24 PM, Ted Ross wrote: > > >> > > > >> > On 03/05/2013 02:14 PM, Rajith Attapattu wrote: > > >> >> > > >> >> > > >> >> This is a good explanation that we need to put in the docs, as > > >> >> Application developers certainly need to know how it behaves. > > >> >> If one were to use the current C impl, it certainly gives the > > >> >> impression that put() is meant to write messages into your > > >> >> internal > > >> >> buffer and send() will actually write it to the wire. > > >> >> Unfortunately some applications will depend on this behaviour, > > >> >> even > > >> >> though it's not advisable > > >> >> > > >> >> If we are to change from say #2 to #1 or even #3 we need to > > >> >> release > > >> >> note it prominently. > > >> >> > > >> >> I think the best solution is to make this behaviour > > >> >> configurable, and > > >> >> advertise the default very prominently. > > >> >> This way application developers will know exactly what they are > > >> >> getting instead of us making changes underneath. > > >> >> > > >> >> Rajith > > >> >> > > >> > > > >> > Making this configurable multiplies the size of the test matrix. > > >> > Can't > > >> we > > >> > make this simpler? > > >> > > >> I do understand your concern here, but the Java impl already does > > >> both > > >> #1 and #2 and Rafi wants to do #3 in the future. > > >> The old JMS client does something similar. > > >> > > >> I agree that if we just do option #2 (as you suggest below), then > > >> the > > >> application can easily do #1 and #3 on top of that. > > >> But I'm sure they will like if the library implements those > > >> strategies > > >> for them and they have the ability to pick a strategy. > > >> > > > > > > I don't see why we'd make this configurable. All three options > > > actually > > > fit the same general semantics. Even if you're optimistically > > > trying to > > > transmit every single time put is called it's entirely possible for > > > the > > > socket to be blocked every single time you try. If this were to > > > happen the > > > implementation of #1 would appear to behave precisely the same as > > > #2 > > > behaves. > > > > > > In other words if you're coding correctly against the API you can't > > > assume > > > that put will or won't have transmitted anything regardless of > > > which > > > strategy is used internally. > > > > > > > I agree with you. You make a very good point. > > Perhaps we should explicitly make that clear in our docs to avoid > > applications written against wrong assumptions. > > > I can certainly do that, but it seems to me that semantics should be > simple, obvious, and orthogonal. > > What seems non-simple and non-obvious to me so far is: > > put() might send, or not. It doesn't send now, but it > might later. > This behaviour is fundamental to an asynchronous API. You're not actually doing things, you're scheduling things to be done asynchronously. This is why put() returns a tracker so you can come back and check on the status of your asynchronous operation. recv() can cause messages to be sent. > > send() can cause messages to be received. > I don't think that's a correct way of describing what is going on. You scheduled an asynchronous operation via put(). That means it can occur at any point later on. The fact that it happens to be trigger by the recv() in the example I gave is simply because recv() is blocking waiting for the reply and so it is inevitably going to end up blocking until the request is sent because the reply won't be triggered until after the request is sent. As for send(), it's simply inaccurate to say that send causes messages to be received. Messages can be spontaneously sent by remote parties at any time (given sufficient credit has been previously granted). What caused them to be received is the other party actually sending them, and if message data happens to arrived while we're inside a call to send(), we can't simply throw those messages away, so they go onto the incoming queue just as if they had arrived during a call to recv(). > I would think that > > 1. every verb should only mean one thing > > 2. there should be a simple mental model, > against which every verb performs a > predictable action. > > so for example: > > put( messenger, message ); // enqueue for sending > send ( messenger, BLOCK ); // block till all sent. > send ( messenger, DONT_BLOCK ); // send what you can. > credit ( messenger, 10 ); // limit incoming queue size > recv ( messenger, BLOCK ); // block till I get a message > recv ( messenger, DONT_BLOCK ); // if no messages incoming, return. > I'm not sure how your example is meaningfully different from the current
Re: put vs. send
On Tue, Mar 5, 2013 at 12:39 PM, Michael Goulish wrote: > > > - Original Message - > > On Tue, Mar 5, 2013 at 11:10 AM, Ted Ross wrote: > > > > > > > > On 03/05/2013 02:01 PM, Rafael Schloming wrote: > > > > > >> On Tue, Mar 5, 2013 at 10:42 AM, Michael Goulish > > >> > >> >wrote: > > >> > > >> > > >>> So, am I understanding correctly? -- I should be able to get > > >>> messages > > >>> from my sender to my receiver just by calling put() -- if the > > >>> receiver > > >>> is ready to receive? > > >>> > > >>> Not necessarily, the receiver being ready just means you are > > >>> unblocked > > >> on > > >> AMQP level flow control. You could also potentially block on the > > >> socket > > >> write (i.e. TCP level flow control). You need to be unblocked on > > >> both for > > >> put to succeed. > > >> > > > > > > Certainly there is no TCP flow control happening in Mick's > > > scenario. > > > > > > > > > What I said was put is *allowed* to send optimistically, not that > > > it is > > >> required to. It actually did send optimistically in a previous > > >> version of > > >> the code, however I commented that line out. > > >> > > >> I would say the documented semantics of put and send should allow > > >> the > > >> implementation the flexibility to do any of the following: > > >> > > >>1) optimistically transmit whatever it can everytime so long as > > >>it > > >> doesn't block > > >>2) never bother transmitting anything until you force it to by > > >>calling > > >> send > > >>3) anything in between the first two, e.g. magically transmit > > >>once > > >> you've > > >> put enough messages to reach the optimal batch size > > >> > > >> The reason for the behaviour you are observing is that we > > >> currently do > > >> option 2 in the C impl, however we've done option 1 in the past > > >> (and I > > >> think we do option 1 still in the Java impl), and we will probably > > >> do > > >> option 3 in the future. > > >> > > > > > > If this is the case, then Mick's original view is correct. The > > > application must assume that messages will not ever be sent unless > > > "send" > > > is called. There is no flowing, pipelined, non-blocking producer. > > > > > > > It's not correct as documentation of the API semantics. It's also not > > correct to say that messages will never be sent unless "send" is > > called, > > e.g. the following code will work fine: > > > > Client: > > m.put(request); > > m.recv(); // wait for reply > > m.get(reply); > > > > That recv() just caused an enqueued message to be sent? > > I just tried it in a C program and it worked, more or less. > At least the receiver got his 5 messages without send() > ever having been called. > > Does a call to recv() also cause all unblocked messages > to be sent? > When a messenger is blocked it will do whatever outstanding work it can. This won't necessarily cause all messages to be sent, but it may cause some. > Is that symmetric? Does a call to send() also cause > messages to be received ? > Yes it is symmetric, messages can arrive while you're blocking for send(). --Rafael
Re: put vs. send
- Original Message - > On Tue, Mar 5, 2013 at 3:20 PM, Rafael Schloming > wrote: > > On Tue, Mar 5, 2013 at 11:33 AM, Rajith Attapattu > > wrote: > > > >> On Tue, Mar 5, 2013 at 2:24 PM, Ted Ross wrote: > >> > > >> > On 03/05/2013 02:14 PM, Rajith Attapattu wrote: > >> >> > >> >> > >> >> This is a good explanation that we need to put in the docs, as > >> >> Application developers certainly need to know how it behaves. > >> >> If one were to use the current C impl, it certainly gives the > >> >> impression that put() is meant to write messages into your > >> >> internal > >> >> buffer and send() will actually write it to the wire. > >> >> Unfortunately some applications will depend on this behaviour, > >> >> even > >> >> though it's not advisable > >> >> > >> >> If we are to change from say #2 to #1 or even #3 we need to > >> >> release > >> >> note it prominently. > >> >> > >> >> I think the best solution is to make this behaviour > >> >> configurable, and > >> >> advertise the default very prominently. > >> >> This way application developers will know exactly what they are > >> >> getting instead of us making changes underneath. > >> >> > >> >> Rajith > >> >> > >> > > >> > Making this configurable multiplies the size of the test matrix. > >> > Can't > >> we > >> > make this simpler? > >> > >> I do understand your concern here, but the Java impl already does > >> both > >> #1 and #2 and Rafi wants to do #3 in the future. > >> The old JMS client does something similar. > >> > >> I agree that if we just do option #2 (as you suggest below), then > >> the > >> application can easily do #1 and #3 on top of that. > >> But I'm sure they will like if the library implements those > >> strategies > >> for them and they have the ability to pick a strategy. > >> > > > > I don't see why we'd make this configurable. All three options > > actually > > fit the same general semantics. Even if you're optimistically > > trying to > > transmit every single time put is called it's entirely possible for > > the > > socket to be blocked every single time you try. If this were to > > happen the > > implementation of #1 would appear to behave precisely the same as > > #2 > > behaves. > > > > In other words if you're coding correctly against the API you can't > > assume > > that put will or won't have transmitted anything regardless of > > which > > strategy is used internally. > > > > I agree with you. You make a very good point. > Perhaps we should explicitly make that clear in our docs to avoid > applications written against wrong assumptions. I can certainly do that, but it seems to me that semantics should be simple, obvious, and orthogonal. What seems non-simple and non-obvious to me so far is: put() might send, or not. It doesn't send now, but it might later. recv() can cause messages to be sent. send() can cause messages to be received. I would think that 1. every verb should only mean one thing 2. there should be a simple mental model, against which every verb performs a predictable action. so for example: put( messenger, message ); // enqueue for sending send ( messenger, BLOCK ); // block till all sent. send ( messenger, DONT_BLOCK ); // send what you can. credit ( messenger, 10 ); // limit incoming queue size recv ( messenger, BLOCK ); // block till I get a message recv ( messenger, DONT_BLOCK ); // if no messages incoming, return. > > Rajith > > > --Rafael >
Re: put vs. send
On Tue, Mar 5, 2013 at 3:20 PM, Rafael Schloming wrote: > On Tue, Mar 5, 2013 at 11:33 AM, Rajith Attapattu wrote: > >> On Tue, Mar 5, 2013 at 2:24 PM, Ted Ross wrote: >> > >> > On 03/05/2013 02:14 PM, Rajith Attapattu wrote: >> >> >> >> >> >> This is a good explanation that we need to put in the docs, as >> >> Application developers certainly need to know how it behaves. >> >> If one were to use the current C impl, it certainly gives the >> >> impression that put() is meant to write messages into your internal >> >> buffer and send() will actually write it to the wire. >> >> Unfortunately some applications will depend on this behaviour, even >> >> though it's not advisable >> >> >> >> If we are to change from say #2 to #1 or even #3 we need to release >> >> note it prominently. >> >> >> >> I think the best solution is to make this behaviour configurable, and >> >> advertise the default very prominently. >> >> This way application developers will know exactly what they are >> >> getting instead of us making changes underneath. >> >> >> >> Rajith >> >> >> > >> > Making this configurable multiplies the size of the test matrix. Can't >> we >> > make this simpler? >> >> I do understand your concern here, but the Java impl already does both >> #1 and #2 and Rafi wants to do #3 in the future. >> The old JMS client does something similar. >> >> I agree that if we just do option #2 (as you suggest below), then the >> application can easily do #1 and #3 on top of that. >> But I'm sure they will like if the library implements those strategies >> for them and they have the ability to pick a strategy. >> > > I don't see why we'd make this configurable. All three options actually > fit the same general semantics. Even if you're optimistically trying to > transmit every single time put is called it's entirely possible for the > socket to be blocked every single time you try. If this were to happen the > implementation of #1 would appear to behave precisely the same as #2 > behaves. > > In other words if you're coding correctly against the API you can't assume > that put will or won't have transmitted anything regardless of which > strategy is used internally. > I agree with you. You make a very good point. Perhaps we should explicitly make that clear in our docs to avoid applications written against wrong assumptions. Rajith > --Rafael
Re: put vs. send
- Original Message - > On Tue, Mar 5, 2013 at 11:10 AM, Ted Ross wrote: > > > > > On 03/05/2013 02:01 PM, Rafael Schloming wrote: > > > >> On Tue, Mar 5, 2013 at 10:42 AM, Michael Goulish > >> >> >wrote: > >> > >> > >>> So, am I understanding correctly? -- I should be able to get > >>> messages > >>> from my sender to my receiver just by calling put() -- if the > >>> receiver > >>> is ready to receive? > >>> > >>> Not necessarily, the receiver being ready just means you are > >>> unblocked > >> on > >> AMQP level flow control. You could also potentially block on the > >> socket > >> write (i.e. TCP level flow control). You need to be unblocked on > >> both for > >> put to succeed. > >> > > > > Certainly there is no TCP flow control happening in Mick's > > scenario. > > > > > > What I said was put is *allowed* to send optimistically, not that > > it is > >> required to. It actually did send optimistically in a previous > >> version of > >> the code, however I commented that line out. > >> > >> I would say the documented semantics of put and send should allow > >> the > >> implementation the flexibility to do any of the following: > >> > >>1) optimistically transmit whatever it can everytime so long as > >>it > >> doesn't block > >>2) never bother transmitting anything until you force it to by > >>calling > >> send > >>3) anything in between the first two, e.g. magically transmit > >>once > >> you've > >> put enough messages to reach the optimal batch size > >> > >> The reason for the behaviour you are observing is that we > >> currently do > >> option 2 in the C impl, however we've done option 1 in the past > >> (and I > >> think we do option 1 still in the Java impl), and we will probably > >> do > >> option 3 in the future. > >> > > > > If this is the case, then Mick's original view is correct. The > > application must assume that messages will not ever be sent unless > > "send" > > is called. There is no flowing, pipelined, non-blocking producer. > > > > It's not correct as documentation of the API semantics. It's also not > correct to say that messages will never be sent unless "send" is > called, > e.g. the following code will work fine: > > Client: > m.put(request); > m.recv(); // wait for reply > m.get(reply); That recv() just caused an enqueued message to be sent? I just tried it in a C program and it worked, more or less. At least the receiver got his 5 messages without send() ever having been called. Does a call to recv() also cause all unblocked messages to be sent? Is that symmetric? Does a call to send() also cause messages to be received ? > > Server: > while True: > m.recv(); // wait for request > m.get(request) > m.put(reply); > > As for there being "no flowing, pipelined, non-blocking producer," > that > seems like an orthogonal issue, and depending on what you mean I > wouldn't > say that's necessarily true either. You can certainly set the > messenger's > timeout to zero and then call put followed by send to get the exact > same > semantics you would get if put were to optimistically send every > time. > > --Rafael >
Re: put vs. send
On Tue, Mar 5, 2013 at 11:33 AM, Rajith Attapattu wrote: > On Tue, Mar 5, 2013 at 2:24 PM, Ted Ross wrote: > > > > On 03/05/2013 02:14 PM, Rajith Attapattu wrote: > >> > >> > >> This is a good explanation that we need to put in the docs, as > >> Application developers certainly need to know how it behaves. > >> If one were to use the current C impl, it certainly gives the > >> impression that put() is meant to write messages into your internal > >> buffer and send() will actually write it to the wire. > >> Unfortunately some applications will depend on this behaviour, even > >> though it's not advisable > >> > >> If we are to change from say #2 to #1 or even #3 we need to release > >> note it prominently. > >> > >> I think the best solution is to make this behaviour configurable, and > >> advertise the default very prominently. > >> This way application developers will know exactly what they are > >> getting instead of us making changes underneath. > >> > >> Rajith > >> > > > > Making this configurable multiplies the size of the test matrix. Can't > we > > make this simpler? > > I do understand your concern here, but the Java impl already does both > #1 and #2 and Rafi wants to do #3 in the future. > The old JMS client does something similar. > > I agree that if we just do option #2 (as you suggest below), then the > application can easily do #1 and #3 on top of that. > But I'm sure they will like if the library implements those strategies > for them and they have the ability to pick a strategy. > I don't see why we'd make this configurable. All three options actually fit the same general semantics. Even if you're optimistically trying to transmit every single time put is called it's entirely possible for the socket to be blocked every single time you try. If this were to happen the implementation of #1 would appear to behave precisely the same as #2 behaves. In other words if you're coding correctly against the API you can't assume that put will or won't have transmitted anything regardless of which strategy is used internally. --Rafael
Re: put vs. send
On Tue, Mar 5, 2013 at 11:24 AM, Ted Ross wrote: > > On 03/05/2013 02:14 PM, Rajith Attapattu wrote: > >> >> This is a good explanation that we need to put in the docs, as >> Application developers certainly need to know how it behaves. >> If one were to use the current C impl, it certainly gives the >> impression that put() is meant to write messages into your internal >> buffer and send() will actually write it to the wire. >> Unfortunately some applications will depend on this behaviour, even >> though it's not advisable >> >> If we are to change from say #2 to #1 or even #3 we need to release >> note it prominently. >> >> I think the best solution is to make this behaviour configurable, and >> advertise the default very prominently. >> This way application developers will know exactly what they are >> getting instead of us making changes underneath. >> >> Rajith >> >> > Making this configurable multiplies the size of the test matrix. Can't we > make this simpler? > > To me, this sounds like an I/O facility in which your output lines may > never get sent if you don't call fflush(). This will be a surprise to most > programmers, who rarely use fflush(). I think most programmers would be > happier if "put" caused the messages to be eventually sent and "send" was > used only for blocking until messages were flushed out. It isn't really possible to have "put" cause messages to be eventually sent without a background thread, something we don't currently have. I also don't think it's really analogous to the fflush() situation. If you're pumping lines into a file and you forget to call fflush(), eventually the buffer will run out of room and your I/O will get flushed to disk anyways in order to clear the buffer. By comparison, if you're pumping messages at a high rate down a slow TCP connection, even if they will all get written eventually by some background thread, you can quite easily fill up all your available memory. Given this, unlike in the file I/O situation, I think you fundamentally need some strategy for pausing and waiting for the wire to catch up. --Rafael
Re: put vs. send
On Tue, Mar 5, 2013 at 11:10 AM, Ted Ross wrote: > > On 03/05/2013 02:01 PM, Rafael Schloming wrote: > >> On Tue, Mar 5, 2013 at 10:42 AM, Michael Goulish > >wrote: >> >> >>> So, am I understanding correctly? -- I should be able to get messages >>> from my sender to my receiver just by calling put() -- if the receiver >>> is ready to receive? >>> >>> Not necessarily, the receiver being ready just means you are unblocked >> on >> AMQP level flow control. You could also potentially block on the socket >> write (i.e. TCP level flow control). You need to be unblocked on both for >> put to succeed. >> > > Certainly there is no TCP flow control happening in Mick's scenario. > > > What I said was put is *allowed* to send optimistically, not that it is >> required to. It actually did send optimistically in a previous version of >> the code, however I commented that line out. >> >> I would say the documented semantics of put and send should allow the >> implementation the flexibility to do any of the following: >> >>1) optimistically transmit whatever it can everytime so long as it >> doesn't block >>2) never bother transmitting anything until you force it to by calling >> send >>3) anything in between the first two, e.g. magically transmit once >> you've >> put enough messages to reach the optimal batch size >> >> The reason for the behaviour you are observing is that we currently do >> option 2 in the C impl, however we've done option 1 in the past (and I >> think we do option 1 still in the Java impl), and we will probably do >> option 3 in the future. >> > > If this is the case, then Mick's original view is correct. The > application must assume that messages will not ever be sent unless "send" > is called. There is no flowing, pipelined, non-blocking producer. > It's not correct as documentation of the API semantics. It's also not correct to say that messages will never be sent unless "send" is called, e.g. the following code will work fine: Client: m.put(request); m.recv(); // wait for reply m.get(reply); Server: while True: m.recv(); // wait for request m.get(request) m.put(reply); As for there being "no flowing, pipelined, non-blocking producer," that seems like an orthogonal issue, and depending on what you mean I wouldn't say that's necessarily true either. You can certainly set the messenger's timeout to zero and then call put followed by send to get the exact same semantics you would get if put were to optimistically send every time. --Rafael
Re: put vs. send
On Tue, Mar 5, 2013 at 2:24 PM, Ted Ross wrote: > > On 03/05/2013 02:14 PM, Rajith Attapattu wrote: >> >> >> This is a good explanation that we need to put in the docs, as >> Application developers certainly need to know how it behaves. >> If one were to use the current C impl, it certainly gives the >> impression that put() is meant to write messages into your internal >> buffer and send() will actually write it to the wire. >> Unfortunately some applications will depend on this behaviour, even >> though it's not advisable >> >> If we are to change from say #2 to #1 or even #3 we need to release >> note it prominently. >> >> I think the best solution is to make this behaviour configurable, and >> advertise the default very prominently. >> This way application developers will know exactly what they are >> getting instead of us making changes underneath. >> >> Rajith >> > > Making this configurable multiplies the size of the test matrix. Can't we > make this simpler? I do understand your concern here, but the Java impl already does both #1 and #2 and Rafi wants to do #3 in the future. The old JMS client does something similar. I agree that if we just do option #2 (as you suggest below), then the application can easily do #1 and #3 on top of that. But I'm sure they will like if the library implements those strategies for them and they have the ability to pick a strategy. > To me, this sounds like an I/O facility in which your output lines may never > get sent if you don't call fflush(). This will be a surprise to most > programmers, who rarely use fflush(). I think most programmers would be > happier if "put" caused the messages to be eventually sent and "send" was > used only for blocking until messages were flushed out. > > -Ted >
Re: put vs. send
On 03/05/2013 02:14 PM, Rajith Attapattu wrote: This is a good explanation that we need to put in the docs, as Application developers certainly need to know how it behaves. If one were to use the current C impl, it certainly gives the impression that put() is meant to write messages into your internal buffer and send() will actually write it to the wire. Unfortunately some applications will depend on this behaviour, even though it's not advisable If we are to change from say #2 to #1 or even #3 we need to release note it prominently. I think the best solution is to make this behaviour configurable, and advertise the default very prominently. This way application developers will know exactly what they are getting instead of us making changes underneath. Rajith Making this configurable multiplies the size of the test matrix. Can't we make this simpler? To me, this sounds like an I/O facility in which your output lines may never get sent if you don't call fflush(). This will be a surprise to most programmers, who rarely use fflush(). I think most programmers would be happier if "put" caused the messages to be eventually sent and "send" was used only for blocking until messages were flushed out. -Ted
Re: put vs. send
On Tue, Mar 5, 2013 at 2:01 PM, Rafael Schloming wrote: > On Tue, Mar 5, 2013 at 10:42 AM, Michael Goulish wrote: > >> >> quoth Rafi: >> >> > The semantics of pn_messenger_put allow it to send if it can do so >> without >> > blocking. >> >> >> So, am I understanding correctly? -- I should be able to get messages >> from my sender to my receiver just by calling put() -- if the receiver >> is ready to receive? >> > > Not necessarily, the receiver being ready just means you are unblocked on > AMQP level flow control. You could also potentially block on the socket > write (i.e. TCP level flow control). You need to be unblocked on both for > put to succeed. > > >> >> The only transmission difference between put() and send() is that send() >> will actually block until they're all sent (or timeout hits). put() should >> get rid of all the messages that aren't blocked, and leave all that are. >> >> . . . >> >> Because what I'm seeing is -- with my receiver hanging in recv(), >> I put 5 messages. Sender sits there for a while. No messages arrive at >> receiver. Then sender calls send() -- and all 5 messages arrive at >> the receiver. >> >> This is true whether on the receiver side, I use >> >>pn_messenger_recv ( messenger, 100 ); >>pn_messenger_recv ( messenger, 5 ); >>pn_messenger_recv ( messenger, 1 ); >>or >>pn_messenger_recv ( messenger, -1 ); >> >> >> That's why it seemed two-stage to me. put() seems to gets them staged, >> send() seems to shove them out the door. >> >> No? >> Or is this a bug? >> > > What I said was put is *allowed* to send optimistically, not that it is > required to. It actually did send optimistically in a previous version of > the code, however I commented that line out. > > I would say the documented semantics of put and send should allow the > implementation the flexibility to do any of the following: > > 1) optimistically transmit whatever it can everytime so long as it > doesn't block > 2) never bother transmitting anything until you force it to by calling > send > 3) anything in between the first two, e.g. magically transmit once you've > put enough messages to reach the optimal batch size > > The reason for the behaviour you are observing is that we currently do > option 2 in the C impl, however we've done option 1 in the past (and I > think we do option 1 still in the Java impl), and we will probably do > option 3 in the future. This is a good explanation that we need to put in the docs, as Application developers certainly need to know how it behaves. If one were to use the current C impl, it certainly gives the impression that put() is meant to write messages into your internal buffer and send() will actually write it to the wire. Unfortunately some applications will depend on this behaviour, even though it's not advisable If we are to change from say #2 to #1 or even #3 we need to release note it prominently. I think the best solution is to make this behaviour configurable, and advertise the default very prominently. This way application developers will know exactly what they are getting instead of us making changes underneath. Rajith > --Rafael
Re: put vs. send
On 03/05/2013 02:01 PM, Rafael Schloming wrote: On Tue, Mar 5, 2013 at 10:42 AM, Michael Goulish wrote: So, am I understanding correctly? -- I should be able to get messages from my sender to my receiver just by calling put() -- if the receiver is ready to receive? Not necessarily, the receiver being ready just means you are unblocked on AMQP level flow control. You could also potentially block on the socket write (i.e. TCP level flow control). You need to be unblocked on both for put to succeed. Certainly there is no TCP flow control happening in Mick's scenario. What I said was put is *allowed* to send optimistically, not that it is required to. It actually did send optimistically in a previous version of the code, however I commented that line out. I would say the documented semantics of put and send should allow the implementation the flexibility to do any of the following: 1) optimistically transmit whatever it can everytime so long as it doesn't block 2) never bother transmitting anything until you force it to by calling send 3) anything in between the first two, e.g. magically transmit once you've put enough messages to reach the optimal batch size The reason for the behaviour you are observing is that we currently do option 2 in the C impl, however we've done option 1 in the past (and I think we do option 1 still in the Java impl), and we will probably do option 3 in the future. If this is the case, then Mick's original view is correct. The application must assume that messages will not ever be sent unless "send" is called. There is no flowing, pipelined, non-blocking producer. --Rafael
Re: put vs. send
On Tue, Mar 5, 2013 at 10:42 AM, Michael Goulish wrote: > > quoth Rafi: > > > The semantics of pn_messenger_put allow it to send if it can do so > without > > blocking. > > > So, am I understanding correctly? -- I should be able to get messages > from my sender to my receiver just by calling put() -- if the receiver > is ready to receive? > Not necessarily, the receiver being ready just means you are unblocked on AMQP level flow control. You could also potentially block on the socket write (i.e. TCP level flow control). You need to be unblocked on both for put to succeed. > > The only transmission difference between put() and send() is that send() > will actually block until they're all sent (or timeout hits). put() should > get rid of all the messages that aren't blocked, and leave all that are. > > . . . > > Because what I'm seeing is -- with my receiver hanging in recv(), > I put 5 messages. Sender sits there for a while. No messages arrive at > receiver. Then sender calls send() -- and all 5 messages arrive at > the receiver. > > This is true whether on the receiver side, I use > >pn_messenger_recv ( messenger, 100 ); >pn_messenger_recv ( messenger, 5 ); >pn_messenger_recv ( messenger, 1 ); >or >pn_messenger_recv ( messenger, -1 ); > > > That's why it seemed two-stage to me. put() seems to gets them staged, > send() seems to shove them out the door. > > No? > Or is this a bug? > What I said was put is *allowed* to send optimistically, not that it is required to. It actually did send optimistically in a previous version of the code, however I commented that line out. I would say the documented semantics of put and send should allow the implementation the flexibility to do any of the following: 1) optimistically transmit whatever it can everytime so long as it doesn't block 2) never bother transmitting anything until you force it to by calling send 3) anything in between the first two, e.g. magically transmit once you've put enough messages to reach the optimal batch size The reason for the behaviour you are observing is that we currently do option 2 in the C impl, however we've done option 1 in the past (and I think we do option 1 still in the Java impl), and we will probably do option 3 in the future. --Rafael
