Re: Help with affinityRun on collocated ignite queue
Hello! I don't understand your question. A collocated queue is located on a single node, but you still need a method to run code on that specific node, don't you? The benefit is presumably shorter round-trip times. Regards, -- Ilya Kasnacheev вт, 4 мая 2021 г. в 18:50, ps594 : > I am trying to understand the use case of affinityRun / affinityCall > < > https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteQueue.html#affinityCall-org.apache.ignite.lang.IgniteCallable-> > > methods of IgniteQueue interface. Even after delving into documentation I > could not understand a good use case why would we want to run jobs on the > collocated queue using affinityRun method of queue interface, since > collocated queues are on the same node why can't I simply write my own > lambda function. Specifically what benefits does the affinityRun offers for > ignite queue (given affinityRun method is not supported for non-collocated > queues)? > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Help with affinityRun on collocated ignite queue
I am trying to understand the use case of affinityRun / affinityCall <https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteQueue.html#affinityCall-org.apache.ignite.lang.IgniteCallable-> methods of IgniteQueue interface. Even after delving into documentation I could not understand a good use case why would we want to run jobs on the collocated queue using affinityRun method of queue interface, since collocated queues are on the same node why can't I simply write my own lambda function. Specifically what benefits does the affinityRun offers for ignite queue (given affinityRun method is not supported for non-collocated queues)? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite Queue (Documentation or Code defect)?
Hi guys, That has been done. https://issues.apache.org/jira/browse/IGNITE-13396 -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite Queue
I see it's already being picked up in another thread: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Queue-Documentation-or-Code-defect-td33703.html I can't delete this thread. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite Queue
Yes the common pattern is not documented. So was expecting if no exception is thrown that I'm getting a queue and can put something on it. But then if I don't check for null I'll get a NullpointerException. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite Queue (Documentation or Code defect)?
Could you please file a ticket in Ignite JIRA for 2.10 release? That's definitely an overlook in the current implementation. - Denis On Tue, Aug 18, 2020 at 9:57 AM Humphrey wrote: > Hope someone pics this up. That the code is not behaving as the > documentation > is saying. > (Not throwing exception when queue can not be fetched). > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Re: Ignite Queue (Documentation or Code defect)?
It’s a good catch, but I think this is one of the developer mailing list. I see the same behaviour. Arguably it’s consistent with the Ignite#cache() which returns null if the cache doesn’t exist. Regards, Stephen > On 18 Aug 2020, at 15:29, Humphrey wrote: > > Hope someone pics this up. That the code is not behaving as the documentation > is saying. > (Not throwing exception when queue can not be fetched). > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite Queue (Documentation or Code defect)?
Hope someone pics this up. That the code is not behaving as the documentation is saying. (Not throwing exception when queue can not be fetched). -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite Queue
Hello! I think that "could not be fetched or created" hints that some error should happen to get an exception. Returning null if an object does not exist is a common pattern in Ignite API so I suspect this is what is the norm here. Maybe I'm missing something? Regards, -- Ilya Kasnacheev вт, 18 авг. 2020 г. в 00:22, Humphrey : > According to the documentation (java doc) of ignite.queue(): > Will get a named queue from cache and create one if it has not been created > yet and cfg is not null. If queue is present already, queue properties will > not be changed. Use collocation for CacheMode.PARTITIONED caches if you > have > lots of relatively small queues as it will make fetching, querying, and > iteration a lot faster. If you have few very large queues, then you should > consider turning off collocation as they simply may not fit in a single > node's memory. > > Params: > name – Name of queue. > cap – Capacity of queue, 0 for unbounded queue. Ignored if cfg is null. > cfg – Queue configuration if new queue should be created. > Returns: > Queue with given properties. > Throws: > org.apache.ignite.IgniteException – If queue could not be fetched or > created. > > But when getting an non existing queue without QueueConfiguration the queue > is null and no exception is thrown. > > > > Documentation should say that it returns a Queue or Null, but it doesn't. > It > says that it will throw an exception if the queue could not be fetched or > created, but it doesn't. It just returns null. > > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Re: Ignite Queue (Documentation or Code defect)?
Great I see something went missing:
Ignite ignite = Ignition.start();
IgniteQueue queue = ignite.queue("Queue", 0, null);
ignite.close();
In the queue created above I expect it to throw an exception if the queue
can not be fetched, instead it is giving me a queue that is "null".
Check the JavaDoc:
Will get a named queue from cache and create one if it has not been created
yet and cfg is not null. If queue is present already, queue properties will
not be changed. Use collocation for CacheMode.PARTITIONED caches if you have
lots of relatively small queues as it will make fetching, querying, and
iteration a lot faster. If you have few very large queues, then you should
consider turning off collocation as they simply may not fit in a single
node's memory.
Params:
name – Name of queue.
cap – Capacity of queue, 0 for unbounded queue. Ignored if cfg is null.
cfg – Queue configuration if new queue should be created.
Returns:
Queue with given properties.
Throws:
org.apache.ignite.IgniteException – If queue could not be fetched or
created.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Ignite Queue
According to the documentation (java doc) of ignite.queue(): Will get a named queue from cache and create one if it has not been created yet and cfg is not null. If queue is present already, queue properties will not be changed. Use collocation for CacheMode.PARTITIONED caches if you have lots of relatively small queues as it will make fetching, querying, and iteration a lot faster. If you have few very large queues, then you should consider turning off collocation as they simply may not fit in a single node's memory. Params: name – Name of queue. cap – Capacity of queue, 0 for unbounded queue. Ignored if cfg is null. cfg – Queue configuration if new queue should be created. Returns: Queue with given properties. Throws: org.apache.ignite.IgniteException – If queue could not be fetched or created. But when getting an non existing queue without QueueConfiguration the queue is null and no exception is thrown. Documentation should say that it returns a Queue or Null, but it doesn't. It says that it will throw an exception if the queue could not be fetched or created, but it doesn't. It just returns null. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Ignite Queue (Documentation or Code defect)?
When creating an ignite queue without an CollectionConfiguration (null) I'm not getting the expected Queue or Exception as documented in the JavaDoc. But i'm getting a cache that is null. The documentation: -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: ignite queue poller
Here how it will look in the code: https://github.com/ezhuravl/ignite-code-examples/tree/master/src/main/java/examples/service/scheduler пн, 2 мар. 2020 г. в 15:23, Evgenii Zhuravlev : > Hi, > > There is no pooler out of the box, but you can simply implement one as a > Service. Even if one of the nodes with this service will be stopped, > Cluster will manage to restart the service on another node. > org.apache.ignite.services.Service.execute() should be implemented as > queue pooler. > > https://apacheignite.readme.io/docs/service-grid > > Best Regards, > Evgenii > > вс, 1 мар. 2020 г. в 10:05, Narsi Reddy Nallamilli < > narsi.nallami...@gmail.com>: > >> Hi, >> >> Is there a queue poller built-in ignite which constantly and indefinitely >> polls the queue and can perform some task? >> >> I know only way is to write igniterunnable compute where i should run >> loop forever and perform the task on ignite queue element? But this is bad >> as there are cases when the compute dies and no one can resume it or track >> it. >> >> Is there a way a where we add some ignite runnable queue elements and the >> elements pop up automatically and gets executed without any trigger? >> >
Re: ignite queue poller
Hi, There is no pooler out of the box, but you can simply implement one as a Service. Even if one of the nodes with this service will be stopped, Cluster will manage to restart the service on another node. org.apache.ignite.services.Service.execute() should be implemented as queue pooler. https://apacheignite.readme.io/docs/service-grid Best Regards, Evgenii вс, 1 мар. 2020 г. в 10:05, Narsi Reddy Nallamilli < narsi.nallami...@gmail.com>: > Hi, > > Is there a queue poller built-in ignite which constantly and indefinitely > polls the queue and can perform some task? > > I know only way is to write igniterunnable compute where i should run loop > forever and perform the task on ignite queue element? But this is bad as > there are cases when the compute dies and no one can resume it or track it. > > Is there a way a where we add some ignite runnable queue elements and the > elements pop up automatically and gets executed without any trigger? >
ignite queue poller
Hi, Is there a queue poller built-in ignite which constantly and indefinitely polls the queue and can perform some task? I know only way is to write igniterunnable compute where i should run loop forever and perform the task on ignite queue element? But this is bad as there are cases when the compute dies and no one can resume it or track it. Is there a way a where we add some ignite runnable queue elements and the elements pop up automatically and gets executed without any trigger?
Re: Speed Expectations for Ignite Queue
To answer my own question: the speed seems to be expected and lays in the same range like e.g. redis+redisson Peter Am 26.11.18 um 16:32 schrieb Peter: > > Hello, > > I'm currently trying Apache Ignite and love the concept. > > I have created a simple example of what I'm trying to achieve > (producer&consumer): > > https://github.com/karussell/igniteexample/blob/master/src/main/java/test/MyIgnite.java > > My assumption is that the default settings are tuned to get a setup > that is okayish-fast and also without persistence and so comparing the > speed to LinkedBlockingQueue shouldn't be that unfair. My expectation > here is that ignite is 10-50x slower due to serialization overhead > still happening for localhost. > > But it seems that ignite queue is 100-500 times slower. What am I > doing wrong here or did I simply have too optimistic expectations? > Maybe non-default settings like here > <http://apache-ignite-users.70518.x6.nabble.com/Impact-of-connectionsPerNode-and-pairedConnection-td18996.html> > would be better suited? See also MyIgniteSingle1.java where I also > warm up everything before measuring the speed. I also tried various > different JVM settings > <https://apacheignite.readme.io/docs/jvm-and-system-tuning> without > success. > > Kind Regards > Peter > > PS: I cross-posted this a few days ago on SO: > https://stackoverflow.com/q/53454232/194609 >
Re: Speed Expectations for Ignite Queue
Hello! I think that you had too optimistic expectations. LinkedBlockingQueue is a core Java data structure and it is very fast - it just writes a reference to data structure in best case scenario. Meanwhile, Ignite Queue will create several internal temporary objects (such as distributed locks/futures) while processing addition or removal as well as doing serialization. Regards, -- Ilya Kasnacheev пн, 26 нояб. 2018 г. в 18:32, Peter : > Hello, > > I'm currently trying Apache Ignite and love the concept. > > I have created a simple example of what I'm trying to achieve > (producer&consumer): > > > https://github.com/karussell/igniteexample/blob/master/src/main/java/test/MyIgnite.java > > My assumption is that the default settings are tuned to get a setup that > is okayish-fast and also without persistence and so comparing the speed to > LinkedBlockingQueue shouldn't be that unfair. My expectation here is that > ignite is 10-50x slower due to serialization overhead still happening for > localhost. > > But it seems that ignite queue is 100-500 times slower. What am I doing > wrong here or did I simply have too optimistic expectations? Maybe > non-default settings like here > <http://apache-ignite-users.70518.x6.nabble.com/Impact-of-connectionsPerNode-and-pairedConnection-td18996.html> > would be better suited? See also MyIgniteSingle1.java where I also warm up > everything before measuring the speed. I also tried various different JVM > settings <https://apacheignite.readme.io/docs/jvm-and-system-tuning> > without success. > > Kind Regards > Peter > > PS: I cross-posted this a few days ago on SO: > https://stackoverflow.com/q/53454232/194609 >
Speed Expectations for Ignite Queue
Hello, I'm currently trying Apache Ignite and love the concept. I have created a simple example of what I'm trying to achieve (producer&consumer): https://github.com/karussell/igniteexample/blob/master/src/main/java/test/MyIgnite.java My assumption is that the default settings are tuned to get a setup that is okayish-fast and also without persistence and so comparing the speed to LinkedBlockingQueue shouldn't be that unfair. My expectation here is that ignite is 10-50x slower due to serialization overhead still happening for localhost. But it seems that ignite queue is 100-500 times slower. What am I doing wrong here or did I simply have too optimistic expectations? Maybe non-default settings like here <http://apache-ignite-users.70518.x6.nabble.com/Impact-of-connectionsPerNode-and-pairedConnection-td18996.html> would be better suited? See also MyIgniteSingle1.java where I also warm up everything before measuring the speed. I also tried various different JVM settings <https://apacheignite.readme.io/docs/jvm-and-system-tuning> without success. Kind Regards Peter PS: I cross-posted this a few days ago on SO: https://stackoverflow.com/q/53454232/194609
RE: Initialize ignite queue when node is starting
Check out the Javadoc of the Ignite.queue() - https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/Ignite.html#queue-java.lang.String-int-org.apache.ignite.configuration.CollectionConfiguration-. You can pass null if you don’t want to start a new queue, just get an existing one by name if it exists. The cfg parameter is even annotated as @Nullable. Moreover, if the queue with the given name already exists, it will not be changed, the configuration will be ignored. Stan From: arunkjn Sent: 14 июня 2018 г. 17:48 To: user@ignite.apache.org Subject: RE: Initialize ignite queue when node is starting Thanks, I meant ignite queue. If I have initialized an ignite queue in one node, and I want to just obtain a reference to it on the second node; there is no API to do that currently without passing in the entire configuration. Is there something I am missing? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: Initialize ignite queue when node is starting
Thanks, I meant ignite queue. If I have initialized an ignite queue in one node, and I want to just obtain a reference to it on the second node; there is no API to do that currently without passing in the entire configuration. Is there something I am missing? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: Initialize ignite queue when node is starting
Hi, If you mean IgniteQueue then, unfortunately, there is no way to describe it in the XML. You have to call Ignite.queue() to create one. Thanks, Stan From: arunkjn Sent: 10 июня 2018 г. 10:14 To: user@ignite.apache.org Subject: Initialize ignite queue when node is starting Hi, I give my cache configuration in xml, which the node uses to initialize the cache when starting up. I want to similarly do this for initializing ignite queues. what would be a substitue of - for ignite queues -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Initialize ignite queue when node is starting
Hi, If you want to configure message queue limit, here is an example: ... Best Regards, Roman -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Initialize ignite queue when node is starting
Hi, I give my cache configuration in xml, which the node uses to initialize the cache when starting up. I want to similarly do this for initializing ignite queues. what would be a substitue of - for ignite queues -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Failed to get Item| Ignite queue
Ok. However, there was only one node and only node which holds that cache. A little strange behaviour. Memory footprint in heap was also half. On Sep 15, 2016 5:31 PM, "Vladislav Pyatkov" wrote: > Hi, > > It look like you are losing elements. This may be because node of grid has > failed. > You can try to increase count of backup copies, look at the setBackups() > in CollectionConfiguration[1]. > > [1]: https://apacheignite.readme.io/v1.7/docs/queue-and-set# > collection-configuration > > On Wed, Sep 14, 2016 at 10:55 AM, Anmol Rattan > wrote: > >> Hi, >> >> We are seeing frequent messages while fetching from our ignite queu. >> >> >> 016-09-14 08:47:32.254 BST [WARN] - thread="AsyncBatchBondPersistence2" >> - >> class="org.apache.ignite.internal.processors.datastructures.GridAtomicCacheQueueImpl" >> - Failed to get item, will retry poll >> [queue=PROCESSED_BONDS_QUEUE, idx=378465] >> >> >> Any suggestions. >> >> >> >> > > > -- > Vladislav Pyatkov >
Re: Failed to get Item| Ignite queue
Hi, It look like you are losing elements. This may be because node of grid has failed. You can try to increase count of backup copies, look at the setBackups() in CollectionConfiguration[1]. [1]: https://apacheignite.readme.io/v1.7/docs/queue-and-set#collection-configuration On Wed, Sep 14, 2016 at 10:55 AM, Anmol Rattan wrote: > Hi, > > We are seeing frequent messages while fetching from our ignite queu. > > > 016-09-14 08:47:32.254 BST [WARN] - thread="AsyncBatchBondPersistence2" - > class="org.apache.ignite.internal.processors.datastructures.GridAtomicCacheQueueImpl" > - Failed to get item, will retry poll > [queue=PROCESSED_BONDS_QUEUE, idx=378465] > > > Any suggestions. > > > > -- Vladislav Pyatkov
Failed to get Item| Ignite queue
Hi, We are seeing frequent messages while fetching from our ignite queu. 016-09-14 08:47:32.254 BST [WARN] - thread="AsyncBatchBondPersistence2" - class="org.apache.ignite.internal.processors.datastructures.GridAtomicCacheQueueImpl" - Failed to get item, will retry poll [queue=PROCESSED_BONDS_QUEUE, idx=378465] Any suggestions.
