Re: Odg: gateway sender queue
+1 to Dan's suggestion. In addition we should think about the following scenario: In certain cases, cleanQueue would cause stop to block if Receivers are not up or causing exception while applying the events. In that case events will not be deletd from queue and stop will block. Currently, it is controlled by a timeout, however we can give an option to the user it he/she still wants to force stop the queue in case queue is not drained in the a given time. On Fri, Nov 15, 2019 at 6:18 AM Michael Oleske wrote: > > > > vsd stat in the gfs file > > > > Just here to say consider using a meter instead of stat as documented in > https://cwiki.apache.org/confluence/display/GEODE/How+to+add+a+Meter if > more than a log message is warrented. > > -michael > > On Thu, Nov 14, 2019 at 11:29 AM Nabarun Nag wrote: > > > +1 to Dan's suggestion > > > > What about a log and a vsd stat in the gfs file which tells us if any > > cleanQueue commands were executed. > > > > > > Regards > > Nabarun Nag > > > > On Thu, Nov 14, 2019 at 10:27 AM Udo Kohlmeyer wrote: > > > > > In addition... making is default has bigger consequences that we have > > > not thought about.. > > > > > > e.g if you purge an existing queue on start up.. is this the start up > of > > > the server node / GS Queue? Given that we have shared-nothing > > > architecture, purging *should* only be local and not cluster-wide... > I'd > > > be interested and see a proposal on this feature. > > > > > > --Udo > > > > > > On 11/14/19 10:24 AM, Jason Huynh wrote: > > > > +1 to Dan's suggestion > > > > > > > > On Thu, Nov 14, 2019 at 9:52 AM Dan Smith wrote: > > > > > > > >> I'm ok with adding a --cleanQueue option. > > > >> > > > >> However, I don't think it should default to be true, since that > would > > be > > > >> changing the behavior of the existing command. It should default to > > > false. > > > >> > > > >> -Dan > > > >> > > > >> On Thu, Nov 14, 2019 at 9:28 AM Xiaojian Zhou > > wrote: > > > >> > > > >>> The --cleanQueue option is a similar idea as Barry's "DeadLetter" > > > spike. > > > >> I > > > >>> remembered that we decided not to do it. > > > >>> > > > >>> > > > >>> On Wed, Nov 13, 2019 at 11:41 PM Mario Ivanac > > > > > > >>> wrote: > > > >>> > > > >>>> Hi, > > > >>>> > > > >>>> just to remind you on last question: > > > >>>> > > > >>>> what is your opinion on adding additional option in gfsh command > > > >> "start > > > >>>> gateway sender" > > > >>>> to control clearing of existing queues --cleanQueues. > > > >>>> > > > >>>> This option will indicate, when gateway sender is started, should > we > > > >>>> discard/clean existing queue, or should we use existing queue. > > > >>>> By default it will be to discard/clean existing queue. > > > >>>> > > > >>>> Best Regards, > > > >>>> Mario > > > >>>> ____ > > > >>>> Šalje: Mario Ivanac > > > >>>> Poslano: 8. studenog 2019. 13:00 > > > >>>> Prima: dev@geode.apache.org > > > >>>> Predmet: Odg: gateway sender queue > > > >>>> > > > >>>> Hi all, > > > >>>> > > > >>>> one more clarification regarding 3rd question: > > > >>>> > > > >>>> "* Could we add extra option in gfsh command "start gateway > > sender" > > > >>>> that allows to control queues reset (for instance > > > --cleanQueues)" > > > >>>> > > > >>>> This option will indicate, when gateway sender is started, should > we > > > >>>> discard/clean existing queue, or should we use existing queue. > > > >>>> By default it will be to discard/clean existing queue. > > > >>>> > > > >>>> Best Regards, > > > >>>> Mario > > > >>>> __
Re: Odg: gateway sender queue
> > vsd stat in the gfs file > Just here to say consider using a meter instead of stat as documented in https://cwiki.apache.org/confluence/display/GEODE/How+to+add+a+Meter if more than a log message is warrented. -michael On Thu, Nov 14, 2019 at 11:29 AM Nabarun Nag wrote: > +1 to Dan's suggestion > > What about a log and a vsd stat in the gfs file which tells us if any > cleanQueue commands were executed. > > > Regards > Nabarun Nag > > On Thu, Nov 14, 2019 at 10:27 AM Udo Kohlmeyer wrote: > > > In addition... making is default has bigger consequences that we have > > not thought about.. > > > > e.g if you purge an existing queue on start up.. is this the start up of > > the server node / GS Queue? Given that we have shared-nothing > > architecture, purging *should* only be local and not cluster-wide... I'd > > be interested and see a proposal on this feature. > > > > --Udo > > > > On 11/14/19 10:24 AM, Jason Huynh wrote: > > > +1 to Dan's suggestion > > > > > > On Thu, Nov 14, 2019 at 9:52 AM Dan Smith wrote: > > > > > >> I'm ok with adding a --cleanQueue option. > > >> > > >> However, I don't think it should default to be true, since that would > be > > >> changing the behavior of the existing command. It should default to > > false. > > >> > > >> -Dan > > >> > > >> On Thu, Nov 14, 2019 at 9:28 AM Xiaojian Zhou > wrote: > > >> > > >>> The --cleanQueue option is a similar idea as Barry's "DeadLetter" > > spike. > > >> I > > >>> remembered that we decided not to do it. > > >>> > > >>> > > >>> On Wed, Nov 13, 2019 at 11:41 PM Mario Ivanac > > > >>> wrote: > > >>> > > >>>> Hi, > > >>>> > > >>>> just to remind you on last question: > > >>>> > > >>>> what is your opinion on adding additional option in gfsh command > > >> "start > > >>>> gateway sender" > > >>>> to control clearing of existing queues --cleanQueues. > > >>>> > > >>>> This option will indicate, when gateway sender is started, should we > > >>>> discard/clean existing queue, or should we use existing queue. > > >>>> By default it will be to discard/clean existing queue. > > >>>> > > >>>> Best Regards, > > >>>> Mario > > >>>> > > >>>> Šalje: Mario Ivanac > > >>>> Poslano: 8. studenog 2019. 13:00 > > >>>> Prima: dev@geode.apache.org > > >>>> Predmet: Odg: gateway sender queue > > >>>> > > >>>> Hi all, > > >>>> > > >>>> one more clarification regarding 3rd question: > > >>>> > > >>>> "* Could we add extra option in gfsh command "start gateway > sender" > > >>>> that allows to control queues reset (for instance > > --cleanQueues)" > > >>>> > > >>>> This option will indicate, when gateway sender is started, should we > > >>>> discard/clean existing queue, or should we use existing queue. > > >>>> By default it will be to discard/clean existing queue. > > >>>> > > >>>> Best Regards, > > >>>> Mario > > >>>> > > >>>> Šalje: Mario Ivanac > > >>>> Poslano: 7. studenog 2019. 9:01 > > >>>> Prima: Dan Smith ; dev@geode.apache.org < > > >>>> dev@geode.apache.org> > > >>>> Predmet: Odg: gateway sender queue > > >>>> > > >>>> Hi, > > >>>> > > >>>> thanks for answers. > > >>>> > > >>>> Some more details regarding 1st question. > > >>>> > > >>>> Is this behavior same (for serial and parallel gateway sender) in > case > > >>>> queue is persistent? > > >>>> Meaning, should queue (persistent) be purged if we restart gateway > > >>> sender? > > >>>> > > >>>> Thanks, > > >>>> Mario > > >>>> > > >>>> > > >>>> Šalje: Dan Smith > > >>>> Poslano: 5. studenog 2019. 18:52 > > >>>> Prima: dev@geode.apache.org > > >>>> Predmet: Re: gateway sender queue > > >>>> > > >>>> Some replies, inline: > > >>>> > > >>>>* During testing we have observed, different behavior in > parallel > > >> and > > >>>>> serial gateway senders. In case we manually stop, than start > gateway > > >>>>> senders, for parallel gateway senders, queue is purged, but for > > >> serial > > >>>>> gateway senders this is not the case. Is this normal behavior or > bug? > > >>>>> > > >>>> Hmm, I also think stop is supposed to clear the queue. I think if > you > > >> are > > >>>> seeing that it doesn't clear the queue, that might be a bug. > > >>>> > > >>>> > > >>>> > > >>>>>* What happens with the queues when whole cluster is stopped > and > > >>>> later > > >>>>> started (In our tests with persistent queues, the events are kept)? > > >>>>> > > >>>> Persistent queues will keep all of the events when you restart. > > >>>> > > >>>> > > >>>>>* Could we add extra option in gfsh command "start gateway > > >> sender" > > >>>>> that allows to control queues reset (for instance --cleanQueues)? > > >>>>> > > >>>> If stop does clear the queue, would this be needed? It might still > be > > >>>> reasonable - I've heard folks request a way to clear running queues > as > > >>>> well. > > >>>> > > >>>> -Dan > > >>>> > > >
Re: Odg: gateway sender queue
+1 to Dan's suggestion What about a log and a vsd stat in the gfs file which tells us if any cleanQueue commands were executed. Regards Nabarun Nag On Thu, Nov 14, 2019 at 10:27 AM Udo Kohlmeyer wrote: > In addition... making is default has bigger consequences that we have > not thought about.. > > e.g if you purge an existing queue on start up.. is this the start up of > the server node / GS Queue? Given that we have shared-nothing > architecture, purging *should* only be local and not cluster-wide... I'd > be interested and see a proposal on this feature. > > --Udo > > On 11/14/19 10:24 AM, Jason Huynh wrote: > > +1 to Dan's suggestion > > > > On Thu, Nov 14, 2019 at 9:52 AM Dan Smith wrote: > > > >> I'm ok with adding a --cleanQueue option. > >> > >> However, I don't think it should default to be true, since that would be > >> changing the behavior of the existing command. It should default to > false. > >> > >> -Dan > >> > >> On Thu, Nov 14, 2019 at 9:28 AM Xiaojian Zhou wrote: > >> > >>> The --cleanQueue option is a similar idea as Barry's "DeadLetter" > spike. > >> I > >>> remembered that we decided not to do it. > >>> > >>> > >>> On Wed, Nov 13, 2019 at 11:41 PM Mario Ivanac > >>> wrote: > >>> > >>>> Hi, > >>>> > >>>> just to remind you on last question: > >>>> > >>>> what is your opinion on adding additional option in gfsh command > >> "start > >>>> gateway sender" > >>>> to control clearing of existing queues --cleanQueues. > >>>> > >>>> This option will indicate, when gateway sender is started, should we > >>>> discard/clean existing queue, or should we use existing queue. > >>>> By default it will be to discard/clean existing queue. > >>>> > >>>> Best Regards, > >>>> Mario > >>>> > >>>> Šalje: Mario Ivanac > >>>> Poslano: 8. studenog 2019. 13:00 > >>>> Prima: dev@geode.apache.org > >>>> Predmet: Odg: gateway sender queue > >>>> > >>>> Hi all, > >>>> > >>>> one more clarification regarding 3rd question: > >>>> > >>>> "* Could we add extra option in gfsh command "start gateway sender" > >>>> that allows to control queues reset (for instance > --cleanQueues)" > >>>> > >>>> This option will indicate, when gateway sender is started, should we > >>>> discard/clean existing queue, or should we use existing queue. > >>>> By default it will be to discard/clean existing queue. > >>>> > >>>> Best Regards, > >>>> Mario > >>>> > >>>> Šalje: Mario Ivanac > >>>> Poslano: 7. studenog 2019. 9:01 > >>>> Prima: Dan Smith ; dev@geode.apache.org < > >>>> dev@geode.apache.org> > >>>> Predmet: Odg: gateway sender queue > >>>> > >>>> Hi, > >>>> > >>>> thanks for answers. > >>>> > >>>> Some more details regarding 1st question. > >>>> > >>>> Is this behavior same (for serial and parallel gateway sender) in case > >>>> queue is persistent? > >>>> Meaning, should queue (persistent) be purged if we restart gateway > >>> sender? > >>>> > >>>> Thanks, > >>>> Mario > >>>> > >>>> > >>>> Šalje: Dan Smith > >>>> Poslano: 5. studenog 2019. 18:52 > >>>> Prima: dev@geode.apache.org > >>>> Predmet: Re: gateway sender queue > >>>> > >>>> Some replies, inline: > >>>> > >>>>* During testing we have observed, different behavior in parallel > >> and > >>>>> serial gateway senders. In case we manually stop, than start gateway > >>>>> senders, for parallel gateway senders, queue is purged, but for > >> serial > >>>>> gateway senders this is not the case. Is this normal behavior or bug? > >>>>> > >>>> Hmm, I also think stop is supposed to clear the queue. I think if you > >> are > >>>> seeing that it doesn't clear the queue, that might be a bug. > >>>> > >>>> > >>>> > >>>>>* What happens with the queues when whole cluster is stopped and > >>>> later > >>>>> started (In our tests with persistent queues, the events are kept)? > >>>>> > >>>> Persistent queues will keep all of the events when you restart. > >>>> > >>>> > >>>>>* Could we add extra option in gfsh command "start gateway > >> sender" > >>>>> that allows to control queues reset (for instance --cleanQueues)? > >>>>> > >>>> If stop does clear the queue, would this be needed? It might still be > >>>> reasonable - I've heard folks request a way to clear running queues as > >>>> well. > >>>> > >>>> -Dan > >>>> >
Re: Odg: gateway sender queue
In addition... making is default has bigger consequences that we have not thought about.. e.g if you purge an existing queue on start up.. is this the start up of the server node / GS Queue? Given that we have shared-nothing architecture, purging *should* only be local and not cluster-wide... I'd be interested and see a proposal on this feature. --Udo On 11/14/19 10:24 AM, Jason Huynh wrote: +1 to Dan's suggestion On Thu, Nov 14, 2019 at 9:52 AM Dan Smith wrote: I'm ok with adding a --cleanQueue option. However, I don't think it should default to be true, since that would be changing the behavior of the existing command. It should default to false. -Dan On Thu, Nov 14, 2019 at 9:28 AM Xiaojian Zhou wrote: The --cleanQueue option is a similar idea as Barry's "DeadLetter" spike. I remembered that we decided not to do it. On Wed, Nov 13, 2019 at 11:41 PM Mario Ivanac wrote: Hi, just to remind you on last question: what is your opinion on adding additional option in gfsh command "start gateway sender" to control clearing of existing queues --cleanQueues. This option will indicate, when gateway sender is started, should we discard/clean existing queue, or should we use existing queue. By default it will be to discard/clean existing queue. Best Regards, Mario Šalje: Mario Ivanac Poslano: 8. studenog 2019. 13:00 Prima: dev@geode.apache.org Predmet: Odg: gateway sender queue Hi all, one more clarification regarding 3rd question: "* Could we add extra option in gfsh command "start gateway sender" that allows to control queues reset (for instance --cleanQueues)" This option will indicate, when gateway sender is started, should we discard/clean existing queue, or should we use existing queue. By default it will be to discard/clean existing queue. Best Regards, Mario Šalje: Mario Ivanac Poslano: 7. studenog 2019. 9:01 Prima: Dan Smith ; dev@geode.apache.org < dev@geode.apache.org> Predmet: Odg: gateway sender queue Hi, thanks for answers. Some more details regarding 1st question. Is this behavior same (for serial and parallel gateway sender) in case queue is persistent? Meaning, should queue (persistent) be purged if we restart gateway sender? Thanks, Mario Šalje: Dan Smith Poslano: 5. studenog 2019. 18:52 Prima: dev@geode.apache.org Predmet: Re: gateway sender queue Some replies, inline: * During testing we have observed, different behavior in parallel and serial gateway senders. In case we manually stop, than start gateway senders, for parallel gateway senders, queue is purged, but for serial gateway senders this is not the case. Is this normal behavior or bug? Hmm, I also think stop is supposed to clear the queue. I think if you are seeing that it doesn't clear the queue, that might be a bug. * What happens with the queues when whole cluster is stopped and later started (In our tests with persistent queues, the events are kept)? Persistent queues will keep all of the events when you restart. * Could we add extra option in gfsh command "start gateway sender" that allows to control queues reset (for instance --cleanQueues)? If stop does clear the queue, would this be needed? It might still be reasonable - I've heard folks request a way to clear running queues as well. -Dan
Re: Odg: gateway sender queue
+1, I love that idea... Problem is that we don't have sufficient auditing around managment/Operational tasks. What happens if this task is run on an active cluster? Without proper auditing and possibly some notion on WHAT data was purged, this could open the product up for many head-aches.. i.e "Entry xxx,yyy,zzz was not replicated to ClusterA..." Without knowing THAT a cleanQueue was run and WHAT entries were purged, we could not dispute that that data was lost due to purge or possibly a system issue... I think as part of this effort we need to think about auditability. --Udo On 11/14/19 9:51 AM, Dan Smith wrote: I'm ok with adding a --cleanQueue option. However, I don't think it should default to be true, since that would be changing the behavior of the existing command. It should default to false. -Dan On Thu, Nov 14, 2019 at 9:28 AM Xiaojian Zhou wrote: The --cleanQueue option is a similar idea as Barry's "DeadLetter" spike. I remembered that we decided not to do it. On Wed, Nov 13, 2019 at 11:41 PM Mario Ivanac wrote: Hi, just to remind you on last question: what is your opinion on adding additional option in gfsh command "start gateway sender" to control clearing of existing queues --cleanQueues. This option will indicate, when gateway sender is started, should we discard/clean existing queue, or should we use existing queue. By default it will be to discard/clean existing queue. Best Regards, Mario Šalje: Mario Ivanac Poslano: 8. studenog 2019. 13:00 Prima: dev@geode.apache.org Predmet: Odg: gateway sender queue Hi all, one more clarification regarding 3rd question: "* Could we add extra option in gfsh command "start gateway sender" that allows to control queues reset (for instance --cleanQueues)" This option will indicate, when gateway sender is started, should we discard/clean existing queue, or should we use existing queue. By default it will be to discard/clean existing queue. Best Regards, Mario Šalje: Mario Ivanac Poslano: 7. studenog 2019. 9:01 Prima: Dan Smith ; dev@geode.apache.org < dev@geode.apache.org> Predmet: Odg: gateway sender queue Hi, thanks for answers. Some more details regarding 1st question. Is this behavior same (for serial and parallel gateway sender) in case queue is persistent? Meaning, should queue (persistent) be purged if we restart gateway sender? Thanks, Mario Šalje: Dan Smith Poslano: 5. studenog 2019. 18:52 Prima: dev@geode.apache.org Predmet: Re: gateway sender queue Some replies, inline: * During testing we have observed, different behavior in parallel and serial gateway senders. In case we manually stop, than start gateway senders, for parallel gateway senders, queue is purged, but for serial gateway senders this is not the case. Is this normal behavior or bug? Hmm, I also think stop is supposed to clear the queue. I think if you are seeing that it doesn't clear the queue, that might be a bug. * What happens with the queues when whole cluster is stopped and later started (In our tests with persistent queues, the events are kept)? Persistent queues will keep all of the events when you restart. * Could we add extra option in gfsh command "start gateway sender" that allows to control queues reset (for instance --cleanQueues)? If stop does clear the queue, would this be needed? It might still be reasonable - I've heard folks request a way to clear running queues as well. -Dan
Re: Odg: gateway sender queue
+1 to Dan's suggestion On Thu, Nov 14, 2019 at 9:52 AM Dan Smith wrote: > I'm ok with adding a --cleanQueue option. > > However, I don't think it should default to be true, since that would be > changing the behavior of the existing command. It should default to false. > > -Dan > > On Thu, Nov 14, 2019 at 9:28 AM Xiaojian Zhou wrote: > > > The --cleanQueue option is a similar idea as Barry's "DeadLetter" spike. > I > > remembered that we decided not to do it. > > > > > > On Wed, Nov 13, 2019 at 11:41 PM Mario Ivanac > > wrote: > > > > > Hi, > > > > > > just to remind you on last question: > > > > > > what is your opinion on adding additional option in gfsh command > "start > > > gateway sender" > > > to control clearing of existing queues --cleanQueues. > > > > > > This option will indicate, when gateway sender is started, should we > > > discard/clean existing queue, or should we use existing queue. > > > By default it will be to discard/clean existing queue. > > > > > > Best Regards, > > > Mario > > > > > > Šalje: Mario Ivanac > > > Poslano: 8. studenog 2019. 13:00 > > > Prima: dev@geode.apache.org > > > Predmet: Odg: gateway sender queue > > > > > > Hi all, > > > > > > one more clarification regarding 3rd question: > > > > > > "* Could we add extra option in gfsh command "start gateway sender" > > > that allows to control queues reset (for instance --cleanQueues)" > > > > > > This option will indicate, when gateway sender is started, should we > > > discard/clean existing queue, or should we use existing queue. > > > By default it will be to discard/clean existing queue. > > > > > > Best Regards, > > > Mario > > > > > > Šalje: Mario Ivanac > > > Poslano: 7. studenog 2019. 9:01 > > > Prima: Dan Smith ; dev@geode.apache.org < > > > dev@geode.apache.org> > > > Predmet: Odg: gateway sender queue > > > > > > Hi, > > > > > > thanks for answers. > > > > > > Some more details regarding 1st question. > > > > > > Is this behavior same (for serial and parallel gateway sender) in case > > > queue is persistent? > > > Meaning, should queue (persistent) be purged if we restart gateway > > sender? > > > > > > > > > Thanks, > > > Mario > > > > > > > > > Šalje: Dan Smith > > > Poslano: 5. studenog 2019. 18:52 > > > Prima: dev@geode.apache.org > > > Predmet: Re: gateway sender queue > > > > > > Some replies, inline: > > > > > > * During testing we have observed, different behavior in parallel > and > > > > serial gateway senders. In case we manually stop, than start gateway > > > > senders, for parallel gateway senders, queue is purged, but for > serial > > > > gateway senders this is not the case. Is this normal behavior or bug? > > > > > > > > > > Hmm, I also think stop is supposed to clear the queue. I think if you > are > > > seeing that it doesn't clear the queue, that might be a bug. > > > > > > > > > > > > > * What happens with the queues when whole cluster is stopped and > > > later > > > > started (In our tests with persistent queues, the events are kept)? > > > > > > > > > > Persistent queues will keep all of the events when you restart. > > > > > > > > > > * Could we add extra option in gfsh command "start gateway > sender" > > > > that allows to control queues reset (for instance --cleanQueues)? > > > > > > > > > > If stop does clear the queue, would this be needed? It might still be > > > reasonable - I've heard folks request a way to clear running queues as > > > well. > > > > > > -Dan > > > > > >
Re: Odg: gateway sender queue
I'm ok with adding a --cleanQueue option. However, I don't think it should default to be true, since that would be changing the behavior of the existing command. It should default to false. -Dan On Thu, Nov 14, 2019 at 9:28 AM Xiaojian Zhou wrote: > The --cleanQueue option is a similar idea as Barry's "DeadLetter" spike. I > remembered that we decided not to do it. > > > On Wed, Nov 13, 2019 at 11:41 PM Mario Ivanac > wrote: > > > Hi, > > > > just to remind you on last question: > > > > what is your opinion on adding additional option in gfsh command "start > > gateway sender" > > to control clearing of existing queues --cleanQueues. > > > > This option will indicate, when gateway sender is started, should we > > discard/clean existing queue, or should we use existing queue. > > By default it will be to discard/clean existing queue. > > > > Best Regards, > > Mario > > ________ > > Šalje: Mario Ivanac > > Poslano: 8. studenog 2019. 13:00 > > Prima: dev@geode.apache.org > > Predmet: Odg: gateway sender queue > > > > Hi all, > > > > one more clarification regarding 3rd question: > > > > "* Could we add extra option in gfsh command "start gateway sender" > > that allows to control queues reset (for instance --cleanQueues)" > > > > This option will indicate, when gateway sender is started, should we > > discard/clean existing queue, or should we use existing queue. > > By default it will be to discard/clean existing queue. > > > > Best Regards, > > Mario > > > > Šalje: Mario Ivanac > > Poslano: 7. studenog 2019. 9:01 > > Prima: Dan Smith ; dev@geode.apache.org < > > dev@geode.apache.org> > > Predmet: Odg: gateway sender queue > > > > Hi, > > > > thanks for answers. > > > > Some more details regarding 1st question. > > > > Is this behavior same (for serial and parallel gateway sender) in case > > queue is persistent? > > Meaning, should queue (persistent) be purged if we restart gateway > sender? > > > > > > Thanks, > > Mario > > > > > > Šalje: Dan Smith > > Poslano: 5. studenog 2019. 18:52 > > Prima: dev@geode.apache.org > > Predmet: Re: gateway sender queue > > > > Some replies, inline: > > > > * During testing we have observed, different behavior in parallel and > > > serial gateway senders. In case we manually stop, than start gateway > > > senders, for parallel gateway senders, queue is purged, but for serial > > > gateway senders this is not the case. Is this normal behavior or bug? > > > > > > > Hmm, I also think stop is supposed to clear the queue. I think if you are > > seeing that it doesn't clear the queue, that might be a bug. > > > > > > > > > * What happens with the queues when whole cluster is stopped and > > later > > > started (In our tests with persistent queues, the events are kept)? > > > > > > > Persistent queues will keep all of the events when you restart. > > > > > > > * Could we add extra option in gfsh command "start gateway sender" > > > that allows to control queues reset (for instance --cleanQueues)? > > > > > > > If stop does clear the queue, would this be needed? It might still be > > reasonable - I've heard folks request a way to clear running queues as > > well. > > > > -Dan > > >
Re: Odg: gateway sender queue
The --cleanQueue option is a similar idea as Barry's "DeadLetter" spike. I remembered that we decided not to do it. On Wed, Nov 13, 2019 at 11:41 PM Mario Ivanac wrote: > Hi, > > just to remind you on last question: > > what is your opinion on adding additional option in gfsh command "start > gateway sender" > to control clearing of existing queues --cleanQueues. > > This option will indicate, when gateway sender is started, should we > discard/clean existing queue, or should we use existing queue. > By default it will be to discard/clean existing queue. > > Best Regards, > Mario > > Šalje: Mario Ivanac > Poslano: 8. studenog 2019. 13:00 > Prima: dev@geode.apache.org > Predmet: Odg: gateway sender queue > > Hi all, > > one more clarification regarding 3rd question: > > "* Could we add extra option in gfsh command "start gateway sender" > that allows to control queues reset (for instance --cleanQueues)" > > This option will indicate, when gateway sender is started, should we > discard/clean existing queue, or should we use existing queue. > By default it will be to discard/clean existing queue. > > Best Regards, > Mario > ____ > Šalje: Mario Ivanac > Poslano: 7. studenog 2019. 9:01 > Prima: Dan Smith ; dev@geode.apache.org < > dev@geode.apache.org> > Predmet: Odg: gateway sender queue > > Hi, > > thanks for answers. > > Some more details regarding 1st question. > > Is this behavior same (for serial and parallel gateway sender) in case > queue is persistent? > Meaning, should queue (persistent) be purged if we restart gateway sender? > > > Thanks, > Mario > > > Šalje: Dan Smith > Poslano: 5. studenog 2019. 18:52 > Prima: dev@geode.apache.org > Predmet: Re: gateway sender queue > > Some replies, inline: > > * During testing we have observed, different behavior in parallel and > > serial gateway senders. In case we manually stop, than start gateway > > senders, for parallel gateway senders, queue is purged, but for serial > > gateway senders this is not the case. Is this normal behavior or bug? > > > > Hmm, I also think stop is supposed to clear the queue. I think if you are > seeing that it doesn't clear the queue, that might be a bug. > > > > > * What happens with the queues when whole cluster is stopped and > later > > started (In our tests with persistent queues, the events are kept)? > > > > Persistent queues will keep all of the events when you restart. > > > > * Could we add extra option in gfsh command "start gateway sender" > > that allows to control queues reset (for instance --cleanQueues)? > > > > If stop does clear the queue, would this be needed? It might still be > reasonable - I've heard folks request a way to clear running queues as > well. > > -Dan >
Odg: gateway sender queue
Hi, just to remind you on last question: what is your opinion on adding additional option in gfsh command "start gateway sender" to control clearing of existing queues --cleanQueues. This option will indicate, when gateway sender is started, should we discard/clean existing queue, or should we use existing queue. By default it will be to discard/clean existing queue. Best Regards, Mario Šalje: Mario Ivanac Poslano: 8. studenog 2019. 13:00 Prima: dev@geode.apache.org Predmet: Odg: gateway sender queue Hi all, one more clarification regarding 3rd question: "* Could we add extra option in gfsh command "start gateway sender" that allows to control queues reset (for instance --cleanQueues)" This option will indicate, when gateway sender is started, should we discard/clean existing queue, or should we use existing queue. By default it will be to discard/clean existing queue. Best Regards, Mario Šalje: Mario Ivanac Poslano: 7. studenog 2019. 9:01 Prima: Dan Smith ; dev@geode.apache.org Predmet: Odg: gateway sender queue Hi, thanks for answers. Some more details regarding 1st question. Is this behavior same (for serial and parallel gateway sender) in case queue is persistent? Meaning, should queue (persistent) be purged if we restart gateway sender? Thanks, Mario Šalje: Dan Smith Poslano: 5. studenog 2019. 18:52 Prima: dev@geode.apache.org Predmet: Re: gateway sender queue Some replies, inline: * During testing we have observed, different behavior in parallel and > serial gateway senders. In case we manually stop, than start gateway > senders, for parallel gateway senders, queue is purged, but for serial > gateway senders this is not the case. Is this normal behavior or bug? > Hmm, I also think stop is supposed to clear the queue. I think if you are seeing that it doesn't clear the queue, that might be a bug. > * What happens with the queues when whole cluster is stopped and later > started (In our tests with persistent queues, the events are kept)? > Persistent queues will keep all of the events when you restart. > * Could we add extra option in gfsh command "start gateway sender" > that allows to control queues reset (for instance --cleanQueues)? > If stop does clear the queue, would this be needed? It might still be reasonable - I've heard folks request a way to clear running queues as well. -Dan
Odg: gateway sender queue
Hi all, one more clarification regarding 3rd question: "* Could we add extra option in gfsh command "start gateway sender" that allows to control queues reset (for instance --cleanQueues)" This option will indicate, when gateway sender is started, should we discard/clean existing queue, or should we use existing queue. By default it will be to discard/clean existing queue. Best Regards, Mario Šalje: Mario Ivanac Poslano: 7. studenog 2019. 9:01 Prima: Dan Smith ; dev@geode.apache.org Predmet: Odg: gateway sender queue Hi, thanks for answers. Some more details regarding 1st question. Is this behavior same (for serial and parallel gateway sender) in case queue is persistent? Meaning, should queue (persistent) be purged if we restart gateway sender? Thanks, Mario Šalje: Dan Smith Poslano: 5. studenog 2019. 18:52 Prima: dev@geode.apache.org Predmet: Re: gateway sender queue Some replies, inline: * During testing we have observed, different behavior in parallel and > serial gateway senders. In case we manually stop, than start gateway > senders, for parallel gateway senders, queue is purged, but for serial > gateway senders this is not the case. Is this normal behavior or bug? > Hmm, I also think stop is supposed to clear the queue. I think if you are seeing that it doesn't clear the queue, that might be a bug. > * What happens with the queues when whole cluster is stopped and later > started (In our tests with persistent queues, the events are kept)? > Persistent queues will keep all of the events when you restart. > * Could we add extra option in gfsh command "start gateway sender" > that allows to control queues reset (for instance --cleanQueues)? > If stop does clear the queue, would this be needed? It might still be reasonable - I've heard folks request a way to clear running queues as well. -Dan
Odg: gateway sender queue
Hi, thanks for answers. Some more details regarding 1st question. Is this behavior same (for serial and parallel gateway sender) in case queue is persistent? Meaning, should queue (persistent) be purged if we restart gateway sender? Thanks, Mario Šalje: Dan Smith Poslano: 5. studenog 2019. 18:52 Prima: dev@geode.apache.org Predmet: Re: gateway sender queue Some replies, inline: * During testing we have observed, different behavior in parallel and > serial gateway senders. In case we manually stop, than start gateway > senders, for parallel gateway senders, queue is purged, but for serial > gateway senders this is not the case. Is this normal behavior or bug? > Hmm, I also think stop is supposed to clear the queue. I think if you are seeing that it doesn't clear the queue, that might be a bug. > * What happens with the queues when whole cluster is stopped and later > started (In our tests with persistent queues, the events are kept)? > Persistent queues will keep all of the events when you restart. > * Could we add extra option in gfsh command "start gateway sender" > that allows to control queues reset (for instance --cleanQueues)? > If stop does clear the queue, would this be needed? It might still be reasonable - I've heard folks request a way to clear running queues as well. -Dan
Re: gateway sender queue
Some replies, inline: * During testing we have observed, different behavior in parallel and > serial gateway senders. In case we manually stop, than start gateway > senders, for parallel gateway senders, queue is purged, but for serial > gateway senders this is not the case. Is this normal behavior or bug? > Hmm, I also think stop is supposed to clear the queue. I think if you are seeing that it doesn't clear the queue, that might be a bug. > * What happens with the queues when whole cluster is stopped and later > started (In our tests with persistent queues, the events are kept)? > Persistent queues will keep all of the events when you restart. > * Could we add extra option in gfsh command "start gateway sender" > that allows to control queues reset (for instance --cleanQueues)? > If stop does clear the queue, would this be needed? It might still be reasonable - I've heard folks request a way to clear running queues as well. -Dan
gateway sender queue
Hi geode dev, we have some questions regarding gateway senders and gateway sender queue. * During testing we have observed, different behavior in parallel and serial gateway senders. In case we manually stop, than start gateway senders, for parallel gateway senders, queue is purged, but for serial gateway senders this is not the case. Is this normal behavior or bug? * What happens with the queues when whole cluster is stopped and later started (In our tests with persistent queues, the events are kept)? * Could we add extra option in gfsh command "start gateway sender" that allows to control queues reset (for instance --cleanQueues)? Thanks, Mario
Re: [discussion]clear method implementation for Parallel Gateway Sender Queue
Dinesh,
I'm not sure if its better to send these comments on the dev list or on the
PR.
In any event, I'm not sure that ParallelGatewaySenderQueue should be
implementing PartitionedRegion clear.
I think PartitionedRegion clear should be implemented first, then
GatewaySender clearQueue should be built on top of that. It would use
PartitionedRegion clear and override BucketRegion clear as necessary in
AbstractBucketRegionQueue.
Your implementation of ParallelGatewaySenderQueue clearPartitionedRegion
(or some form of it) would become PartitionedRegion clear, although someone
who knows storage would have to review that to verify.
PartitionedRegion clear would delegate to BucketRegion clear which would be
like your ParallelGatewaySenderQueue clearBucketRegion. There might be a
better way to clear the BucketRegion than iterating the keys and destroying
each, but I'm not sure. Again someone who knows storage would have to
review that to verify.
Then, AbstractGatewaySender clearQueue could be implemented on top of that
something like:
- get RegionQueues
- for each RegionQueue, get its region or regions (depending on whether the
sender is serial or parallel)
- invoke clear on each region (or possibly localClear)
I think the implementation of PartitionedRegion clear and the testing of it
is trickier than it sounds. Concurrent updates, HA, rebalancing, etc. all
have to be taken into account.
Thanks,
Barry Oglesby
On Thu, Dec 7, 2017 at 2:21 AM, Dinesh Akhand wrote:
> Created the pull request for the same.
> https://github.com/apache/geode/pull/1139
>
>
> Thanks,
> Dinesh akhand
>
> -Original Message-
> From: Dan Smith [mailto:dsm...@pivotal.io]
> Sent: Tuesday, July 18, 2017 3:59 AM
> To: dev@geode.apache.org
> Subject: Re: need information about
> SerialGatewaySenderQueue/ParallelGatewaySenderQueue
> Clear
>
> Hi Dinesh,
>
> I think we probably just never got around to adding a clear. I think you
> could probably clear your queues just stop stopping and starting the
> gateway sender, which might be the easiest thing to do here.
>
> Regarding your code, for your parallel queue are you doing that inside of
> a function? The code you have will try to clear things on a single node.
> The queue also maintains some other metadata in memory. I'm not quite sure
> what the effect on the queue will be if you delete the region entries
> without changing that other metadata. I guess you could test it and find
> out.
> You'll probably want to see what the effect is while the queue is actually
> dispatching entries as well, because it's possible you could catch the
> system in a state where it is trying to read entries from the region as you
> are deleting them. Or maybe pause the queue first in your clear method?
>
> -Dan
>
> On Fri, Jul 14, 2017 at 2:23 AM, Dinesh Akhand
> wrote:
>
> > Hi Team,
> >
> >
> >
> > Please reply . why we don't have implementation of clear method in
> > ParallelGatewaySenderQueue/ SerialGatewaySenderQueue in geode.
> Requirement:
> > we want to clear the queue data.
> >
> >
> >
> > I have implement below method in our code.
> >
> > --
> >
> > Class ParallelGatewaySenderQueue.java
> >
> >
> >
> > //clear the partition region
> >
> > private void clearPartitionedRegion(PartitionedRegion
> > partitionedRegion)
> >
> > {
> >
> > LocalDataSet lds = (LocalDataSet)
> > PartitionRegionHelper.getLocalPrimaryData(partitionedRegion);
> >
> > Setset = lds.getBucketSet(); // this
> > returns bucket ids in the function context
> >
> >
> >
> > for (Integer bucketId : set) {
> >
> > Bucket bucket =
> partitionedRegion.
> > getRegionAdvisor().getBucket(bucketId);
> >
> > if (bucket instanceof
> > ProxyBucketRegion == false) {
> >
> > if (bucket
> > instanceof BucketRegion) {
> >
> >
> > BucketRegion bucketRegion = (BucketRegion) bucket;
> >
> >
> > Set keySet = bucketRegion.keySet();
> >
> >
> > for (Iterator iterator = keySet.iterator(); iterator.hasNext();)
> > {
> >
> >
> > Object key = iterator.next();
> >
> >
> > bucketRegion.remove(key);
> >
> >
> > }
> >
> > }
> >
> > }
> >
> > }
> >
> > }
> >
> > -
> >
> > Class : SerialGatewaySenderQueue.java
> >
> > @Override
> >
> > public void clearQueue() {
> >
> >
> >
> > this.sender.getLifeCycleLock().readLock().lock();
> >
> > Set keys = this.region.keys();
> >
> > for (Long key : keys) {
> >
> > this.region.remove(key);
> >
> > }
> >
> > this.sender.getLifeCycleLock().readLock().unlock();
> >
> >
> >
> > }
>
[discussion]clear method implementation for Parallel Gateway Sender Queue
Created the pull request for the same.
https://github.com/apache/geode/pull/1139
Thanks,
Dinesh akhand
-Original Message-
From: Dan Smith [mailto:dsm...@pivotal.io]
Sent: Tuesday, July 18, 2017 3:59 AM
To: dev@geode.apache.org
Subject: Re: need information about
SerialGatewaySenderQueue/ParallelGatewaySenderQueue Clear
Hi Dinesh,
I think we probably just never got around to adding a clear. I think you could
probably clear your queues just stop stopping and starting the gateway sender,
which might be the easiest thing to do here.
Regarding your code, for your parallel queue are you doing that inside of a
function? The code you have will try to clear things on a single node. The
queue also maintains some other metadata in memory. I'm not quite sure what the
effect on the queue will be if you delete the region entries without changing
that other metadata. I guess you could test it and find out.
You'll probably want to see what the effect is while the queue is actually
dispatching entries as well, because it's possible you could catch the system
in a state where it is trying to read entries from the region as you are
deleting them. Or maybe pause the queue first in your clear method?
-Dan
On Fri, Jul 14, 2017 at 2:23 AM, Dinesh Akhand wrote:
> Hi Team,
>
>
>
> Please reply . why we don't have implementation of clear method in
> ParallelGatewaySenderQueue/ SerialGatewaySenderQueue in geode. Requirement:
> we want to clear the queue data.
>
>
>
> I have implement below method in our code.
>
> --
>
> Class ParallelGatewaySenderQueue.java
>
>
>
> //clear the partition region
>
> private void clearPartitionedRegion(PartitionedRegion
> partitionedRegion)
>
> {
>
> LocalDataSet lds = (LocalDataSet)
> PartitionRegionHelper.getLocalPrimaryData(partitionedRegion);
>
> Setset = lds.getBucketSet(); // this
> returns bucket ids in the function context
>
>
>
> for (Integer bucketId : set) {
>
> Bucket bucket = partitionedRegion.
> getRegionAdvisor().getBucket(bucketId);
>
> if (bucket instanceof
> ProxyBucketRegion == false) {
>
> if (bucket
> instanceof BucketRegion) {
>
>
> BucketRegion bucketRegion = (BucketRegion) bucket;
>
>
> Set keySet = bucketRegion.keySet();
>
>
> for (Iterator iterator = keySet.iterator(); iterator.hasNext();)
> {
>
>
> Object key = iterator.next();
>
>
> bucketRegion.remove(key);
>
>
> }
>
> }
>
> }
>
> }
>
> }
>
> -
>
> Class : SerialGatewaySenderQueue.java
>
> @Override
>
> public void clearQueue() {
>
>
>
> this.sender.getLifeCycleLock().readLock().lock();
>
> Set keys = this.region.keys();
>
> for (Long key : keys) {
>
> this.region.remove(key);
>
> }
>
> this.sender.getLifeCycleLock().readLock().unlock();
>
>
>
> }
>
> -
>
>
>
> Any comment in above code will welcome.
>
>
>
>
>
> Thanks,
>
> Dinesh Akhand
>
>
>
> -Original Message-
> From: Dinesh Akhand
> Sent: Monday, May 15, 2017 2:39 PM
> To: dev@geode.apache.org
> Subject: need information about RegionQueue
>
>
>
>
>
> Hi Team,
>
>
>
> Why we do't have support to clear complete queue. Is there any
> limitation for it?.
>
>
>
> public void clear(PartitionedRegion pr, int bucketId) {
>
>throw new RuntimeException("This method(clear)is not supported
> by ParallelGatewaySenderQueue");
>
> }
>
>
>
> Class : ParallelGatewaySenderQueue
>
> Class : SerialGatewaySenderQueue
>
>
>
> Thanks,
>
> Dinesh Akhand
>
>
>
> This message and the information contained herein is proprietary and
> confidential and subject to the Amdocs policy statement,
>
>
>
> you may review at https://www.amdocs.com/about/email-disclaimer <
> https://www.amdocs.com/about/email-disclaimer>
> This message and the information contained herein is proprietary and
> confidential and subject to the Amdocs policy statement,
>
> you may review at https://www.amdocs.com/about/email-disclaimer <
> https://www.amdocs.com/about/email-disclaimer>
>
This message and the information contained herein is proprietary and
confidential and subject to the Amdocs policy statement,
you may review at https://www.amdocs.com/about/email-disclaimer
