Re: Semaphore blocking on tryAcquire() while holding a cache-lock

[Vladimir Ozerov]

Cross-posting to the dev-list.

Hi Mario,
Thank you for source code. I was able to reproduce the issue and deduced it
to simpler example when two nodes start in a single JVM and one of the
holds key lock. See https://issues.apache.org/jira/browse/IGNITE-2671

Igniters,
I think someone with experience with cache/preload should throw a glance at
this ticket. Looks pretty nasty to me. Essentially, new node cannot join
topology when a single cache key lock is held.

Vladimir.

On Tue, Feb 16, 2016 at 8:33 PM, nyname00  wrote:

> Yakov,
>
> please see the code below. Simply start one node (in my case I started it
> directly from eclipse using 1.8u45) and wait for a few cycles, then start
> up
> the second node. You should see one more "sem created" message and then ...
> silence. If you kill the first node, the code processes normal again - but
> not when you kill the second one.
>
> Also note that on the second node no "Topology snapshot" message appears
> which, I guess, is not a good sign.
>
> I'm still testing, but if you replace the "sched" map-lock with a
> semaphore,
> everything seems to work ... until you kill one of the nodes. Then an
> exception is thrown every time the worker tries to get a new semaphore.
>
> Thanks for investigating,
> Mario
>
>
> public static void main(String[] args) {
> Ignite ignite = Ignition.start(new IgniteConfiguration());
>
> Lock lock = ignite
> .getOrCreateCache(
> new CacheConfiguration Boolean>("sched").setAtomicityMode(TRANSACTIONAL))
> .lock("sched-lock");
>
> if (lock.tryLock()) {
> try {
> System.out.println("pre start sched");
>
> ScheduledExecutorService es =
> Executors.newSingleThreadScheduledExecutor();
> ScheduledFuture f = es.scheduleAtFixedRate(new
> Worker(ignite), 0, 2000,
> TimeUnit.MILLISECONDS);
>
> System.out.println("post start sched");
> try {
> f.get();
> } catch (Exception e) {
> e.printStackTrace();
> } finally {
> es.shutdownNow();
> }
> } finally {
> lock.unlock();
> }
> }
> }
>
> private static class Worker implements Runnable {
> private final Ignite ignite;
>
> public Worker(Ignite ignite) {
> this.ignite = ignite;
> }
>
> @Override
> public void run() {
> IgniteSemaphore sem = ignite.semaphore("sem-worker-1", 1,
> true, true);
>
> System.out.println("sem created");
> if (sem.tryAcquire()) {
> try {
> System.out.println("sem aquired");
> } finally {
> sem.release();
> System.out.println("sem released");
> }
> } else {
> System.out.println("sem not aquired");
> }
> }
> }
>
>
> yakov wrote
> > Hi Mario!
> >
> > Can you please share the code somehow? Reproducible example will be very
> > helpful.
> >
> > --Yakov
> >
> > 2016-02-16 16:57 GMT+03:00 nyname00 <
>
> > mario.stark@
>
> > >:
> >
> >> Hi Igniters,
> >>
> >> I'm getting some strange behavior when I try to call
> >> IgniteSemaphore#tryAquire() while holding a lock on a cache element -
> the
> >> method seems block forever.
> >>
> >> Here's the details:
> >>   - create a cache (default, transactional) and get a lock using
> >> Lock#tryLock()
> >>   - start a Executors.newSingleThreadScheduledExecutor() and schedule a
> >> worker to run every n seconds
> >>   - in the worker, get a semaphore and call IgniteSemaphore#tryAcquire()
> >>   - do some work and call IgniteSemaphore#release()
> >> (except for the cache/lock and semaphore no Ignite API calls are made)
> >>
> >> The whole thing works for a single node, but as soon as I start up an
> new
> >> node (with the same code) the IgniteSemaphore#tryAcquire() call starts
> >> blocking.
> >>
> >> Is this this a known limitation/issue or am I doing something wrong
> here?
> >>
> >> Thanks in advance,
> >> Mario
> >>
> >> ps: when I replace the cache-lock with an other semaphore, it works
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://apache-ignite-users.70518.x6.nabble.com/Semaphore-blocking-on-tryAcquire-while-holding-a-cache-lock-tp3031.html
> >> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
> >>
>
>
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Semaphore-blocking-on-tryAcquire-while-holding-a-cache-lock-tp3031p3034.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Huge heap size for ignite cache when storing 20 million objects of small size

[vidhu]

Yes we are not facing any issues related to gc and we can run SQL queries
as well. I tried performing gc through visualvm but size didn't reduced.

We need to load lot of data in our ignite cluster(approx 112 million
objects in cluster). If we go by current scenario we will need lot of ram
and ignite nodes to load this data.

Other option we are trying is to reduce memory size(heap size).

Since we have 112 million objects so we tried to intern fields of this
object.
We have thus implemented an intern cache. After implementing this intern
cache we are able to reduce object size. But the overall heap size is still
not reduced.

With rawWriter heap size reduced but SQL queries are not working.

Could you suggest a way to reduce heap size and SQL query still working.
Is the extra heap that ignite is taking is because of meta data needed by
it to run SQL queries?
On 17 Feb 2016 22:51, "vkulichenko [via Apache Ignite Users]" <
ml-node+s70518n3052...@n6.nabble.com> wrote:

> Hi,
>
> Do I understand correctly that everything works fine and you're not
> running out of memory or experiencing long GC issues? If so, most likely
> you don't have any memory issues, Java simply does not collect garbage
> because there is a lot of space available. You can try to perform GC from
> Visual VM and see how much memory will be available after that.
>
> -Val
>
> --
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-ignite-users.70518.x6.nabble.com/Huge-heap-size-for-ignite-cache-when-storing-20-million-objects-of-small-size-tp3049p3052.html
> To unsubscribe from Huge heap size for ignite cache when storing 20
> million objects of small size, click here
> 
> .
> NAML
> 
>




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Huge-heap-size-for-ignite-cache-when-storing-20-million-objects-of-small-size-tp3049p3053.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Huge heap size for ignite cache when storing 20 million objects of small size

[vkulichenko]

Hi,

Do I understand correctly that everything works fine and you're not running
out of memory or experiencing long GC issues? If so, most likely you don't
have any memory issues, Java simply does not collect garbage because there
is a lot of space available. You can try to perform GC from Visual VM and
see how much memory will be available after that.

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Huge-heap-size-for-ignite-cache-when-storing-20-million-objects-of-small-size-tp3049p3052.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Distributed queue problem with peerClassLoading enabled

[mp]

Denis,

Please see below for answers.

Cheers,
-Mateusz

On Tue, Feb 16, 2016 at 10:18 PM, Denis Magda  wrote:

> Hi Mateusz,
>
> I've revisited the whole discussion from the beginning and should say that
> the solution based on the distributed queue won't work for you even if all
> the issues listed below are fixed.
>
> Presently you're placing in the queue tasks coming form different nodes
> with different class versions. Even if the tasks are stored in the queue in
> a binary form they have to be deserialized to an original form before
> execution. This will lead to ClassNotFoundExceptions in your case.
>

No, I'm not filling the queue with different class versions. My test case
is quite simple. Please check the original description specified in
http://apache-ignite-users.70518.x6.nabble.com/Distributed-queue-problem-with-peerClassLoading-enabled-tp1762p1780.html

In 1.4.0 it works as follows:

1. A server node is started.
2. Client node (which acts as a server as well) is started, creates a
queue, fills it with 100 tasks, and both nodes are polling tasks from the
queue. When the queue is empty (all tasks called), the test ends. At this
point the client node leaves the cluster as well.
3. Then you run the same test again *without any code modifications*, ie,
exactly the same class on the same client node. I assume that even if the
server node already cached the class during the 1st test run, it should be
no problem for the 2nd test run. But unfortunately it fails with
"ignitetest.Task cannot be cast to ignitetest.Task", which I believe is due
to class loaders being different.

In 1.5.0 even the 1st run of the test fails due to a problem you reported
in https://issues.apache.org/jira/browse/IGNITE-2339

Actually, what I would like to achieve is the following (in 3 steps):

1. Have my original test pass, regardless of how many times I call the
test. This is a baseline.
2. Modify my test so that after I finish each test run, I can *modify* the
code of Task class, and my change is reflected in the subsequent test run,
ie, the server node will process the Task class with its new definition.
3. Multiple client nodes can have their own Task class definitions that run
on the cluster at the same time, assuming that they create *different*
queues for their tasks. Actually they will also broadcast their own
IgniteCallables to the cluster (note that in particular they can broadcast
the same IgniteCallable class, each in its own different version). But I'm
always assuming that a given queue is created/maintained/filled by one
client node therefore it will never mix objects with different versions of
the same class.



>
> My suggestion is the same is it was before. Avoid usage of the distributed
> queue but rather start sending ignite compute tasks for execution from the
> beginning.
>
> Will this work for you?
>

In my setup, it is simply more logical to model the computation as
IgniteCallables being broadcast to the cluster and then they pull tasks
from a distributed queue. Actually, the distributed queue (and other
distributed structures provided by Ignite) was one of main reasons that
attracted me to try it out. And I really like them and their simple usage :)
Without the use of distributed structures I think managing my computations
would probably be more cumbersome. But I will of course think about it, and
maybe I will come up with a suitable model.



>
> --
> Denis
>
>
>
> On 2/16/2016 11:33 AM, mp wrote:
>
> Hi Denis,
>
> Many thanks! I look forward to 1.6 then.
> Please also consider the following statement made by Dmitriy on Nov 03,
> 2015 (see his message in the thread):
>
> "With that in mind, we will be removing the requirement for caches to work
> only with SHARED and CONTINUOUS deployment modes, so you will be able to
> use PRIVATE or ISOLATED deployment modes to deploy your computations."
>
> As far as I understand, the above planned change is not covered by any
> Jira ticket.
>
> Cheers,
> -Mateusz
>
>
>
> On Fri, Feb 12, 2016 at 10:35 PM, Denis Magda  wrote:
>
>> Hi Mateusz,
>>
>> I assigned both tickets that you have problems with on myself. They will
>> be fixed as a part of the next release.
>> https://issues.apache.org/jira/browse/IGNITE-2339
>> https://issues.apache.org/jira/browse/IGNITE-1823
>>
>> There is one more issue that was reproduced locally and refers to
>> unexpected cache undeployment when the binary marshaller is used.
>> https://issues.apache.org/jira/browse/IGNITE-2647
>>
>> Thanks for your patience and still showing the interest in Ignite.
>>
>> Regards,
>> Denis
>>
>>
>> On 2/12/2016 4:41 PM, mp wrote:
>>
>> Hi Denis,
>>
>> But my test still fails in version 1.5 with default (ie, binary)
>> marshaller. See my message from January 7, and your reply in which you
>> mentioned a new Jira ticked for a bug concerning the new binary marshaller:
>> 
>> https://issues.apache.org/jira/browse/IGNITE-2339
>>
>> Basically, my test case (s

Re: Huge heap size for ignite cache when storing 20 million objects of small size

[Vladimir Ozerov]

Hi vidhu,

If your object have 101 int field, each instance serialized with
BinaryMarshaller (the default one) will be an array of 731 byte. This means
1.4GB of heap for 2M objects. How many nodes are there in your application?
Do you have backups? It possible pleas share your configuration and source
code.

Using Binarylizable interface and raw mode could shrink size of serialized
object to somewhat about 450 bytes per instance. It looks strange to me
that you cannot get objects serialized in this way form cache. Again, you
source code will help us to understand what the problem is.

Vladimir.

On Wed, Feb 17, 2016 at 8:18 AM, vidhu  wrote:

> Thanks for quick reply.
>
> I mistakenly write 20 million to earlier but objects I am putting in cache
> are 2 million which means .9 GB approx.
>
> Sorry for putting wrong number of objects earlier.
>
> I am starting 2 server nodes of 5 GB each and a client node of 5 gb to
> load data into cache.
>
> Apart from this cache I am having 2 more caches whose size combined is
> less than 400 MB.
> On 17 Feb 2016 10:44, "vkulichenko [via Apache Ignite Users]" <[hidden
> email] > wrote:
>
>> Hi vidhu,
>>
>> First of all, can you please subscribe to the mailing list so that
>> community receives email notifications? Please follow this simple
>> instruction:
>> http://apache-ignite-users.70518.x6.nabble.com/mailing_list/MailingListOptions.jtp?forum=1
>>
>> vidhu wrote
>> I am facing a problem where my heap reaches to about 2.8 gb when I am
>> storing 20 million objects. Size  of each object is approx. 462 bytes.
>> Structure of my object is a simple pojo having 101 fields each of int
>> type.
>>
>> I tried using following 3 marshalling techniques:
>>
>> 1.) Default marshaller of Ignite. (Size of heap ~ 2.8 gb, looking with
>> Visual VM find that most of objects on heap are byte array.)
>>
>> 2.) Optimized marshaller. (Again size was ~ 2.8 gb, looking with Visual
>> VM find that this time size of objects on heap of byte array is half +
>> there are lot of objects of class that I put in cache. )
>>
>> 3.) Tried implementing Binarylizable in my class that I am storing in
>> cache. I used rawWriter. This time size reduced considerably but I was not
>> able to read values out of cache, (getting null on read).
>>
>> I am implementing this in java version 1.8.
>>
>> 20,000,000 * 462 bytes is already around 8.6GB of memory, so 2.8 is not
>> enough even without Ignite's overhead. Did you try to allocate more heap
>> memory and/or start more nodes? With your sizings I would start with two
>> nodes, 8GB of memory per node.
>>
>> As for options you tried, I would recommend to use default binary
>> marshaller because it's much more compact. Also you should take a look at
>> offheap memory [1]. It can be useful if you're going to have a lot of data.
>>
>> [1] https://apacheignite.readme.io/docs/off-heap-memory
>>
>> -Val
>>
>> --
>> If you reply to this email, your message will be added to the discussion
>> below:
>>
>> http://apache-ignite-users.70518.x6.nabble.com/Huge-heap-size-for-ignite-cache-when-storing-20-million-objects-of-small-size-tp3049p3050.html
>> To unsubscribe from Huge heap size for ignite cache when storing 20
>> million objects of small size, click here.
>> NAML
>> 
>>
>
> --
> View this message in context: Re: Huge heap size for ignite cache when
> storing 20 million objects of small size
> 
>
> Sent from the Apache Ignite Users mailing list archive
>  at Nabble.com.
>

Re: Huge heap size for ignite cache when storing 20 million objects of small size

[vidhu]

Hi Vladimir,

We are using 2 server nodes of 5 GB each. Also keeping backup count 1.

We used binarylizable and raw mode which shrinked size but SQL queries are
not running on raw mode. Is there any way we can run SQL queries with raw
mode or we could use any external serializer with which SQL queries could
run?
On 17 Feb 2016 23:54, "Vladimir Ozerov [via Apache Ignite Users]" <
ml-node+s70518n305...@n6.nabble.com> wrote:

> Hi vidhu,
>
> If your object have 101 int field, each instance serialized with
> BinaryMarshaller (the default one) will be an array of 731 byte. This means
> 1.4GB of heap for 2M objects. How many nodes are there in your application?
> Do you have backups? It possible pleas share your configuration and source
> code.
>
> Using Binarylizable interface and raw mode could shrink size of
> serialized object to somewhat about 450 bytes per instance. It looks
> strange to me that you cannot get objects serialized in this way form
> cache. Again, you source code will help us to understand what the problem
> is.
>
> Vladimir.
>
> On Wed, Feb 17, 2016 at 8:18 AM, vidhu <[hidden email]
> > wrote:
>
>> Thanks for quick reply.
>>
>> I mistakenly write 20 million to earlier but objects I am putting in
>> cache are 2 million which means .9 GB approx.
>>
>> Sorry for putting wrong number of objects earlier.
>>
>> I am starting 2 server nodes of 5 GB each and a client node of 5 gb to
>> load data into cache.
>>
>> Apart from this cache I am having 2 more caches whose size combined is
>> less than 400 MB.
>> On 17 Feb 2016 10:44, "vkulichenko [via Apache Ignite Users]" <[hidden
>> email] > wrote:
>>
>>> Hi vidhu,
>>>
>>> First of all, can you please subscribe to the mailing list so that
>>> community receives email notifications? Please follow this simple
>>> instruction:
>>> http://apache-ignite-users.70518.x6.nabble.com/mailing_list/MailingListOptions.jtp?forum=1
>>>
>>> vidhu wrote
>>> I am facing a problem where my heap reaches to about 2.8 gb when I am
>>> storing 20 million objects. Size  of each object is approx. 462 bytes.
>>> Structure of my object is a simple pojo having 101 fields each of int
>>> type.
>>>
>>> I tried using following 3 marshalling techniques:
>>>
>>> 1.) Default marshaller of Ignite. (Size of heap ~ 2.8 gb, looking with
>>> Visual VM find that most of objects on heap are byte array.)
>>>
>>> 2.) Optimized marshaller. (Again size was ~ 2.8 gb, looking with Visual
>>> VM find that this time size of objects on heap of byte array is half +
>>> there are lot of objects of class that I put in cache. )
>>>
>>> 3.) Tried implementing Binarylizable in my class that I am storing in
>>> cache. I used rawWriter. This time size reduced considerably but I was not
>>> able to read values out of cache, (getting null on read).
>>>
>>> I am implementing this in java version 1.8.
>>>
>>> 20,000,000 * 462 bytes is already around 8.6GB of memory, so 2.8 is not
>>> enough even without Ignite's overhead. Did you try to allocate more heap
>>> memory and/or start more nodes? With your sizings I would start with two
>>> nodes, 8GB of memory per node.
>>>
>>> As for options you tried, I would recommend to use default binary
>>> marshaller because it's much more compact. Also you should take a look at
>>> offheap memory [1]. It can be useful if you're going to have a lot of data.
>>>
>>> [1] https://apacheignite.readme.io/docs/off-heap-memory
>>>
>>> -Val
>>>
>>> --
>>> If you reply to this email, your message will be added to the discussion
>>> below:
>>>
>>> http://apache-ignite-users.70518.x6.nabble.com/Huge-heap-size-for-ignite-cache-when-storing-20-million-objects-of-small-size-tp3049p3050.html
>>> To unsubscribe from Huge heap size for ignite cache when storing 20
>>> million objects of small size, click here.
>>> NAML
>>> 
>>>
>>
>> --
>> View this message in context: Re: Huge heap size for ignite cache when
>> storing 20 million objects of small size
>> 
>>
>> Sent from the Apache Ignite Users mailing list archive
>>  at Nabble.com.
>>
>
>
>
> --
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-ignite-users.70518.x6.nabble.com/Huge-heap-size-for-ignite-cache-when-storing-20-million-objects-of

Re: Stupide question about Queue ?

[Stolidedog]

So if I understand correctly you cannot store queue info in a DB currently
with Ignite V1.5?

We are a software vendor and want to build services that communicate between
each other.  The thing to note, is this is on-prem software and we don't
have anyway to know ahead of time how much nodes we can deploy.  Probably in
most cases we have two - four machines, so spinning up more nodes adding
more memory isn't "usually" a luxury afforded to us.  It is simply out of
our control.  

The ability to off load storage in a distributed cluster is what our
customers are requiring more and more (but not all of them).  Also we cannot
loose any messages we distribute work on and it is very likely the whole
cluster can go down either on purpose or not.  Backing up data in a storage
engine (probably for us a RDBMS or NoSQL) is something we just need to have
for our situation.

Hazelcast allows this, but I've been experimenting with Hz but seems like
transacted queues have been never really tested and mostly broken and I
don't get a good feeling about how fast they turn things around when issues
are reported.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Stupide-question-about-Queue-tp2557p3058.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

ExpiryPolicy based on cache element

[Christian Seiler]

Hello all,
is there an elegant way to base the expiry of a cache element on properties of  
the element itself? 
My solution would be to schedule a (Scan)Query with an arbitrary predicate and 
gather the elements to be expired.
I would then remove those elements using cache.removeAll(Set keys). 
Is there a way to combine these two steps into one and even co-locate that step 
with the data?
Any other options?

Regards
Christian

--
InteRes GmbH -   Dr. Christian Seiler
Rheinstrasse 44-46   64283 Darmstadt
fon: +49 6151 307 40 18  fax: -11
www.interes.com   christian.sei...@interes.com

Geschäftsführer:
Volker Herrmann

Registergericht: Amtsgericht Darmstadt
Registernummer: HRB 6831/Darmstadt
Steuernummer: DE 189 632 313 

ITB Berlin - Sie finden uns als Mitaussteller am DRV-Stand in Halle 10.1. 
Schauen Sie vorbei, wir freuen uns!

Re: Semaphore blocking on tryAcquire() while holding a cache-lock

[nyname00]

Vladimir, 
thanks for your efforts. Do you see any way around this problem? It feels
like I'm running out of options (see below).

I'm not sure this is related, but I tried to replace the cache lock with a
semaphore but things get even weirder. 

To reproduce 
  (1) start a new node (see code below) and wait for the "acquired" message
  (2) start another node and wait until it joins the cluster
  (3) kill the node created in (1) - the node created in (2) will take over
and print "acquired"
  (4) start yet another node and wait until it joins the cluster
  (5) kill the node created in (3) - when this node is trying to acquire the
semaphore the following ex is thrown:

SEVERE:  Failed to compare and set:
o.a.i.i.processors.datastructures.GridCacheSemaphoreImpl$Sync$2@b7b3649
class org.apache.ignite.IgniteCheckedException: Failed to find semaphore
with given name: sem1
at
org.apache.ignite.internal.processors.datastructures.GridCacheSemaphoreImpl$Sync$2.call(GridCacheSemaphoreImpl.java:333)
at
org.apache.ignite.internal.processors.datastructures.GridCacheSemaphoreImpl$Sync$2.call(GridCacheSemaphoreImpl.java:323)
at
org.apache.ignite.internal.processors.cache.GridCacheUtils$24.call(GridCacheUtils.java:1776)
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.outTx(GridCacheUtils.java:982)
at
org.apache.ignite.internal.processors.datastructures.GridCacheSemaphoreImpl$Sync.releaseFailedNode(GridCacheSemaphoreImpl.java:322)
at
org.apache.ignite.internal.processors.datastructures.GridCacheSemaphoreImpl.onNodeRemoved(GridCacheSemaphoreImpl.java:504)
at
org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor$1$1.call(DataStructuresProcessor.java:191)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6397)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:929)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

There seems to be no way to retry acquiring the semaphore - Ignite just
hangs ...
Note that, in a more complex test scenario I can reproduce this behavior
with just two nodes.

Regards, 
Mario


public static void main(String[] args) {
try {
Ignite ignite = Ignition.start(new
IgniteConfiguration().setGridName("1"));
System.out.println("started");

IgniteSemaphore sem = ignite.semaphore("sem1", 1, true, true);
System.out.println("created");

sem.acquire();
System.out.println("acquired");
} catch (Exception e) {
System.out.println("oh no");
}
}



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Semaphore-blocking-on-tryAcquire-while-holding-a-cache-lock-tp3031p3061.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: ExpiryPolicy based on cache element

[vkulichenko]

Hi Christian,

You can broadcast a closure which will use IgniteCache.localEntries() method
to iterate through the local data and remove all required data. This way you
will minimize the network utilization.

Will this work for you?

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/ExpiryPolicy-based-on-cache-element-tp3059p3062.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: package name needs to be taken into account when registering class

[vkulichenko]

Hi,

Can you please properly subscribe to the mailing list so that the community
receives email notifications? Please follow the instruction here:
http://apache-ignite-users.70518.x6.nabble.com/mailing_list/MailingListOptions.jtp?forum=1


pingvin wrote
> There is a bug in BinaryContext, line 555.
> 
> Only class name used to register a class,
> I got an exception for two classes with identical names and different
> packages.
> 
> Caused by: class org.apache.ignite.IgniteCheckedException: Duplicate ID
> [id=748942773, oldCls=..., newCls=...]

This is already fixed and the fix will be released in 1.6. You can build
Ignite from master and check if it works for you.

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/package-name-needs-to-be-taken-into-account-when-registering-class-tp3060p3063.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

changed cache configuration and restarted server nodes. Getting exception.

[vinshar]

Hi,

I faced an issue today and couldn't figure out whats wrong hence though of
asking on this forum.
I added expiration policy to 2 cacheConfigurations, stopped all cache server
nodes and then started one by one. My client nodes had near caches for the
these caches and i am not sure if this caused the issue. Issue was that i
started getting "org.apache.ignite.cache.CachePartialUpdateException: Failed
to update keys (retry update if possible)." exception in my apps that were
using these caches. 
I thought that may be there are some old entries in near caches where as
server caches are empty and this is causing issue somehow. I checked cache
statistics and all caches were empty. 
Still i tried to clear caches using visor and faced following exception.

visor> cache -clear -c=PROGRAMS
[16:43:42,883][SEVERE][mgmt-#22%null%][GridTaskWorker] Failed to reduce job
results due to undeclared user exception
[task=o.a.i.i.v.cache.VisorCacheClearTask@54656dd, err=class
o.a.i.IgniteException: Failed to deserialize object with given class loader:
WebappClassLoader
  context: /myWebService
  delegate: false
  repositories:
/WEB-INF/classes/
--> Parent Classloader:
java.net.URLClassLoader@2b71fc7e
]
class org.apache.ignite.IgniteException: Failed to deserialize object with
given class loader: WebappClassLoader
  context: /myWebService
  delegate: false
  repositories:
/WEB-INF/classes/
--> Parent Classloader:
java.net.URLClassLoader@2b71fc7e

at
org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:882)

..
..
..
..
..
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
deserialize object with given class loader: WebappClassLoader
..
..
..
Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
at
org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:523
..
..
..
Caused by: java.io.InvalidObjectException: Ignite instance with provided
name doesn't exist. Did you call Ignition.start(..) to start an Ignite
instance? [name=null]
at
org.apache.ignite.internal.processors.cache.GridCacheContext.readResolve(GridCacheContext.java:1999)
... 37 more
Caused by: class org.apache.ignite.IgniteIllegalStateException: Ignite
instance with provided name doesn't exist. Did you call Ignition.start(..)
to start an Ignite instance? [name=null]
at org.apache.ignite.internal.IgnitionEx.gridx(IgnitionEx.java:1267)
at
org.apache.ignite.internal.processors.cache.GridCacheContext.readResolve(GridCacheContext.java:1989)
... 37 more
(wrn) : class org.apache.ignite.IgniteException: Failed to
deserialize object with given class loader: WebappClassLoader


Regards,
Vinay Sharma



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/changed-cache-configuration-and-restarted-server-nodes-Getting-exception-tp3064.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

IGFS: Writing to a nonexistent file will not create the file?

[Kobe]

I am using Ignite 1.5.0.final on RHEL6 over JRE8.

I have an initialized IGFS filesystem with a seconday backup store as the
local filesystem
and have create a path (directory)  *a/b/c/d*.

I am trying to write content to a (non-existent) file file as follows:

*IgniteFileSystem fs = ...;
IgfsPath folder = new IgfsPath("a/b/c/d");
IgfsPath filex = new IgfsPath(folder, "myFile");

final Boolean create_file_if_not_existent = true;
/try (OutputStream out = fs.append(path, create_file_if_not_existent)) {  
// << Exception here
/out.write(data);
}
*

The fs.append(...) above issues an exception 
/   Failed to append to the file due to secondary file system exception:
/a/b/c/d/myFile
/

The full stack trace is as shown below.

What could be going wrong? Any hints appreciated.
*
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to append
to the file due to secondary file system exception: /a/b/c/d/myFile
at
org.apache.ignite.internal.processors.igfs.IgfsMetaManager$6.onFailure(IgfsMetaManager.java:2137)
at
org.apache.ignite.internal.processors.igfs.IgfsMetaManager$6.onFailure(IgfsMetaManager.java:2077)
at
org.apache.ignite.internal.processors.igfs.IgfsMetaManager.synchronizeAndExecute(IgfsMetaManager.java:2870)
at
org.apache.ignite.internal.processors.igfs.IgfsMetaManager.synchronizeAndExecute(IgfsMetaManager.java:2665)
at
org.apache.ignite.internal.processors.igfs.IgfsMetaManager.appendDual(IgfsMetaManager.java:2142)
at
org.apache.ignite.internal.processors.igfs.IgfsImpl$16.call(IgfsImpl.java:1066)
at
org.apache.ignite.internal.processors.igfs.IgfsImpl$16.call(IgfsImpl.java:1051)
at
org.apache.ignite.internal.processors.igfs.IgfsImpl.safeOp(IgfsImpl.java:1930)
... 17 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to create
path locally due to secondary file system exception: /a/b/c/d/myFile
at
org.apache.ignite.internal.processors.igfs.IgfsMetaManager.synchronize(IgfsMetaManager.java:2619)
at
org.apache.ignite.internal.processors.igfs.IgfsMetaManager.synchronizeAndExecute(IgfsMetaManager.java:2823)
... 22 more*

Thanks!

/kobe



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/IGFS-Writing-to-a-nonexistent-file-will-not-create-the-file-tp3065.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: changed cache configuration and restarted server nodes. Getting exception.

[vkulichenko]

Hi Vinay,

CachePartialUpdateException is thrown by an update operation (put, putAll,
remove, removeAll, ...) if updates for one or more keys involved in this
operation failed. This exception has failedKeys() method that tells you
which keys failed so that you can retry only them, no need to retry
successful ones.

Most likely you were getting these exceptions when there were no server
nodes in the topology. Is this the case?

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/changed-cache-configuration-and-restarted-server-nodes-Getting-exception-tp3064p3066.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: IGFS: Writing to a nonexistent file: "Failed to append to the file due to secondary file system exception"

[Kobe]

I see that this is a known issue:

   https://issues.apache.org/jira/browse/IGNITE-1631

and is planned to be resolved in 1.6

What is the workaround? Should I choose not to use DUAL_SYNC?

If I create the file using IgniteFileSystem.create() would that sidestep
this problem?

Thanks,

Rk
x77309



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/IGFS-Writing-to-a-nonexistent-file-Failed-to-append-to-the-file-due-to-secondary-file-system-excepti-tp3065p3067.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Can Queues be Transacted?

[Stolidedog]

>From what I'm able to get from the documentation, Transactions only apply to
caches?  Is it possible to use transactions in queues?

Basically I want to have one application offer items on a queue.  Another
application take them off the queue and process them, but if it crashes, I
want another node to get the item off the queue because the first taker
never committed.

I can do this with Hazelcast but there seems to be some bugs they are
working on.





--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Can-Queues-be-Transacted-tp3068.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Can Queues be Transacted?

[Dmitriy Setrakyan]

I don’t think transactions on queues are supported yet, however, I am not
sure why not. Seems to be for some legacy reasons. I think it should be a
fairly easy feature to add.

Can someone from the community comment please?

On Wed, Feb 17, 2016 at 4:04 PM, Stolidedog  wrote:

> From what I'm able to get from the documentation, Transactions only apply
> to
> caches?  Is it possible to use transactions in queues?
>
> Basically I want to have one application offer items on a queue.  Another
> application take them off the queue and process them, but if it crashes, I
> want another node to get the item off the queue because the first taker
> never committed.
>
> I can do this with Hazelcast but there seems to be some bugs they are
> working on.
>
>
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Can-Queues-be-Transacted-tp3068.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: changed cache configuration and restarted server nodes. Getting exception.

[vinshar]

Hi Val

At the time of this issue i checked topology through visor and all 3 client
and 2 server nodes were there. There were no items in any cache. I could
see caches created on all 5 nodes (near cache on 3 clients and replicated
cache on 2 servers). I also tried cleaning a cache through visor through
command "cache -clear -c=PROGRAMS" which caused exception trace as i
mentioned previously. I tried multiple times and got same error. I was
running visor on one on the hosts of server nodes.

Shouldn't a node be dropped from topology if its not accessible due to any
issue? Exception trace with class loader related exceptions, error in
uodate when cache is empty on all nodes, all nodes visible in visor
topology and getting same exception on repeated tries to clean a cache
seems like pointing to problem other than network or node accessibility
issue.

I restarted ignite server nodes but problem was still there. I had to stop
all ignite nodes including clients to resolve problem.

Even more interesting thing is that i did not faced any issue on my DEV and
QA environment when i did cache changes and restarted just server nodes. I
faced this problem on pre-prod where i had to restart all nodes.

Regards,
Vinay Sharma
On Feb 17, 2016 7:05 PM, "vkulichenko [via Apache Ignite Users]" <
ml-node+s70518n306...@n6.nabble.com> wrote:

> Hi Vinay,
>
> CachePartialUpdateException is thrown by an update operation (put, putAll,
> remove, removeAll, ...) if updates for one or more keys involved in this
> operation failed. This exception has failedKeys() method that tells you
> which keys failed so that you can retry only them, no need to retry
> successful ones.
>
> Most likely you were getting these exceptions when there were no server
> nodes in the topology. Is this the case?
>
> -Val
>
> --
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-ignite-users.70518.x6.nabble.com/changed-cache-configuration-and-restarted-server-nodes-Getting-exception-tp3064p3066.html
> To unsubscribe from changed cache configuration and restarted server
> nodes. Getting exception., click here
> 
> .
> NAML
> 
>




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/changed-cache-configuration-and-restarted-server-nodes-Getting-exception-tp3064p3070.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Pagination to fetch complete data

[sujay...@ymail.com]

Can we use pagination while fetching the complete cache data.. 

I am using below method for fetching complete Cache data, but it is taking
too much time :
ScanQuery scanQuery = new ScanQuery()
QueryCursor> cursor =
cache.query(scanQuery);


I tired using below method also, but i am getting exception:
 QueryCursor> result =
 cache.query(new SqlQuery([CacheObject].class,
"select * from  [CacheName]"));
result.getAll()


*Exception in thread "main" javax.cache.CacheException: Failed to run reduce
query locally.*
at
org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.query(GridReduceQueryExecutor.java:673)
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$2.iterator(IgniteH2Indexing.java:956)
at
org.apache.ignite.internal.processors.cache.QueryCursorImpl.iterator(QueryCursorImpl.java:61)
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$3.iterator(IgniteH2Indexing.java:990)
at
org.apache.ignite.internal.processors.cache.QueryCursorImpl.iterator(QueryCursorImpl.java:61)
at
org.apache.ignite.internal.processors.cache.QueryCursorImpl.getAll(QueryCursorImpl.java:73)
at com.test.ignite.query.QueryFetch2.queryPrices(QueryFetch2.java:83)
at com.test.ignite.query.QueryFetch2.main(QueryFetch2.java:52)
Caused by: java.lang.NullPointerException
at
org.apache.ignite.internal.processors.query.h2.twostep.GridMergeIndexUnsorted$1.hasNext(GridMergeIndexUnsorted.java:97)


Cache Configurations I am using:

CacheConfiguration  cacheConfig = new
CacheConfiguration<>(cacheName);
cacheConfig.setCacheMode(CacheMode.REPLICATED);
cacheConfig.setMemoryMode(CacheMemoryMode.OFFHEAP_TIERED);

cacheConfig.setExpiryPolicyFactory(CreatedExpiryPolicy.factoryOf(new
Duration(TimeUnit.MINUTES, 60)));
cacheConfig.setEvictionPolicy(new LruEvictionPolicy(5));
cacheConfig.setOffHeapMaxMemory(0);
cacheConfig.setSwapEnabled(false);
cacheConfig.setBackups(1);
cacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Pagination-to-fetch-complete-data-tp3071.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Huge heap size for ignite cache when storing 20 million objects of small size

[vkulichenko]

Most likely memory is consumed by indexes. How many of them do you have?

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Huge-heap-size-for-ignite-cache-when-storing-20-million-objects-of-small-size-tp3049p3072.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Pagination to fetch complete Cache data

[vkulichenko]

Hi,

Query results are already paginated by default. Page size is 1024 elements
by default and can be changed via Query.setPageSize() method. How many
entries in cache do you have? Iteration through the large data set is not
going to be fast anyway.

SQL should be used for indexed search, and they require additional
configuration. Refer to [1] for details.

[1] https://apacheignite.readme.io/docs/sql-queries

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Pagination-to-fetch-complete-Cache-data-tp3071p3073.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Stupide question about Queue ?

[Dmitriy Setrakyan]

At this point queues cannot be persisted yet. The ability to store any data
structure or any cache to disk is currently being designed. I believe that
it will become part of Ignite release by the end of March.

As far as transaction logic, I would like other community members to chime
in. How difficult will it be to make queues and all other data structures
transactional?

D.



On Wed, Feb 17, 2016 at 10:37 AM, Stolidedog  wrote:

> So if I understand correctly you cannot store queue info in a DB currently
> with Ignite V1.5?
>
> We are a software vendor and want to build services that communicate
> between
> each other.  The thing to note, is this is on-prem software and we don't
> have anyway to know ahead of time how much nodes we can deploy.  Probably
> in
> most cases we have two - four machines, so spinning up more nodes adding
> more memory isn't "usually" a luxury afforded to us.  It is simply out of
> our control.
>
> The ability to off load storage in a distributed cluster is what our
> customers are requiring more and more (but not all of them).  Also we
> cannot
> loose any messages we distribute work on and it is very likely the whole
> cluster can go down either on purpose or not.  Backing up data in a storage
> engine (probably for us a RDBMS or NoSQL) is something we just need to have
> for our situation.
>
> Hazelcast allows this, but I've been experimenting with Hz but seems like
> transacted queues have been never really tested and mostly broken and I
> don't get a good feeling about how fast they turn things around when issues
> are reported.
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Stupide-question-about-Queue-tp2557p3058.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>