Re: Performance Issue - Threads blocking

[Matt Hoffman]

(Inline)

On Fri, Apr 22, 2016, 4:26 PM vkulichenko 
wrote:
>
> Hi Matt,
>
> I'm confused. The locking does happen on per-entry level, otherwise it's
> impossible to guarantee data consistency. Two concurrent updates or reads
> for the same key will wait for each other on this lock. But this should
not
> cause performance degradation, unless you have very few keys and very high
> contention on them.
>

Based on his claim of a lot of threads waiting on the same locks, I assumed
that's what was happening -- high contention for a few cache keys. I don't
know his use case, but I can imagine cases with a fairly small number of
very "hot" entries.
It wouldn't necessarily require very few keys, right? Just high contention
on a few of them.

> The only thing I see here is that the value is deserialized on read. This
is
> done because JCache requires store-by-value semantics and thus we create a
> copy each time you get the value (by deserializing its binary
> representation). You can override this behavior by setting
> CacheConfiguration.setCopyOnRead(false) property, this should give you
> performance improvement. Only note that it's not safe to modify the
instance
> that you got from cache this way.
>

Do you think that would be a candidate for the "Performance tips" page in
the docs? I know I've referred to that page a few times recently myself.

> -Val
>
>
>
> --
> View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Performance-Issue-Threads-blocking-tp4433p4465.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Performance Issue - Threads blocking

[Matt Hoffman]

I'm assuming you're seeing a lot of threads that are BLOCKED waiting on
that locked GridLocalCacheEntry (<70d32489> in that example you pasted
above). Looking at the code, it looks like it does block on individual
cache entries (so two reads of the same key within the same JVM will
block). In your particular example above, the thread in question is
publishing an EVT_CACHE_OBJECT_EXPIRED event. If you don't need that,
turning it off (along with EVT_CACHE_OBJECT_READ) will speed up the time
that the cache entry spends blocking other reads (and speed things up,
generally).
It's locking to make sure it's deserialized from swap once and expired once
(if necessary; looks like it was in this particular case).

matt

On Fri, Apr 22, 2016 at 8:07 AM, Vladimir Ozerov 
wrote:

> Hi,
>
> Could you please explain why do you think that the thread is blocked? I
> see it is in a RUNNABLE state.
>
> Vladimir.
>
> On Fri, Apr 22, 2016 at 2:41 AM, ccanning  wrote:
>
>> We seem to be having some serious performance issues after adding Apache
>> Ignite Local cache to our APIs'. Looking at a heap dump, we seem to have a
>> bunch of threads blocked by this lock:
>>
>> "ajp-0.0.0.0-8009-70" - Thread t@641
>>java.lang.Thread.State: RUNNABLE
>> at
>>
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.(BinaryReaderExImpl.java:166)
>> at
>>
>> org.apache.ignite.internal.binary.BinaryUtils.doReadObject(BinaryUtils.java:1486)
>> at
>>
>> org.apache.ignite.internal.binary.BinaryUtils.deserializeOrUnmarshal(BinaryUtils.java:1830)
>> at
>>
>> org.apache.ignite.internal.binary.BinaryUtils.doReadMap(BinaryUtils.java:1813)
>> at
>>
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1597)
>> at
>>
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1646)
>> at
>>
>> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read(BinaryFieldAccessor.java:643)
>> at
>>
>> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:714)
>> at
>>
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1450)
>> at
>>
>> org.apache.ignite.internal.binary.BinaryObjectImpl.deserializeValue(BinaryObjectImpl.java:537)
>> at
>>
>> org.apache.ignite.internal.binary.BinaryObjectImpl.value(BinaryObjectImpl.java:117)
>> at
>>
>> org.apache.ignite.internal.processors.cache.CacheObjectContext.unwrapBinary(CacheObjectContext.java:280)
>> at
>>
>> org.apache.ignite.internal.processors.cache.CacheObjectContext.unwrapBinaryIfNeeded(CacheObjectContext.java:145)
>> at
>>
>> org.apache.ignite.internal.processors.cache.GridCacheEventManager.addEvent(GridCacheEventManager.java:276)
>> at
>>
>> org.apache.ignite.internal.processors.cache.GridCacheEventManager.addEvent(GridCacheEventManager.java:159)
>> at
>>
>> org.apache.ignite.internal.processors.cache.GridCacheEventManager.addEvent(GridCacheEventManager.java:92)
>> at
>>
>> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerGet0(GridCacheMapEntry.java:862)
>> - locked <70d32489> (a
>> org.apache.ignite.internal.processors.cache.local.GridLocalCacheEntry)
>> at
>>
>> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerGet(GridCacheMapEntry.java:669)
>> at
>>
>> org.apache.ignite.internal.processors.cache.local.atomic.GridLocalAtomicCache.getAllInternal(GridLocalAtomicCache.java:587)
>> at
>>
>> org.apache.ignite.internal.processors.cache.local.atomic.GridLocalAtomicCache.get(GridLocalAtomicCache.java:483)
>> at
>>
>> org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:1378)
>> at
>>
>> org.apache.ignite.internal.processors.cache.IgniteCacheProxy.get(IgniteCacheProxy.java:864)
>> at
>> org.apache.ignite.cache.spring.SpringCache.get(SpringCache.java:52)
>>
>>  - locked <70d32489> (a
>> org.apache.ignite.internal.processors.cache.local.GridLocalCacheEntry)
>>
>> Should this be causing blocking in a high-throughput API? Do you have any
>> pointers in how we could solve this issue?
>>
>> Thanks.
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-ignite-users.70518.x6.nabble.com/Performance-Issue-Threads-blocking-tp4433.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>

Re: Ignite "bugs" ?

[Matt Hoffman]

No, I understand that you can't control what other projects do, and it's
not always possible to fix bugs at their source.

I imagine the best thing for Ignite to do in this case is to iterate
through the properties and just ignore anything that isn't a String. It
wouldn't be meaningful when deserialized anyway.

On Wed, Jan 13, 2016 at 10:26 AM, Yann BLAZART <
yann.blaz...@externe.bnpparibas.com> wrote:

> Well, TomEE is using System.properties to store its IVmContext… I can’t
> make change all open source projects on this point, even if I’m agree with
> you.
>
>
>
> I have only one Ignite instance started in this case. This is really
> strange.
>
>
>
> *From:* Matt Hoffman [mailto:m...@mhoffman.org]
> *Sent:* mercredi 13 janvier 2016 15:31
> *To:* user@ignite.apache.org
> *Subject:* Re: Ignite "bugs" ?
>
>
>
> Using System.properties() to store non-strings is really poor behavior. It
> violates the contract of the System.setProperty and System.getProperty, as
> well as the contract of the Properties object itself, from right at the top
> of the javadoc:  "Each key and its corresponding value in the property
> list is a string."
>
> I've hit something like that in Hibernate before, years ago; I'm surprised
> they haven't fixed it yet. It's a long-standing bug in Hibernate. Not to
> say Ignite shouldn't have a workaround for badly-behaved libraries that do
> things like that, but it's definitely a Hibernate bug.
>
>
>
> Someone else will have to talk the locking behavior you're seeing. Are you
> starting Ignite more than once in parallel in that case?
>
>
>
> On Wed, Jan 13, 2016 at 8:23 AM, Yann BLAZART <
> yann.blaz...@externe.bnpparibas.com> wrote:
>
> Hello everybody.
>
>
>
> I’m currently evaluation Ignite vs Hazelcast on a poc.
>
>
>
> I’m facing some issues.
>
> I’m coding some integration/unit test using the ApplicationComposer of
> TomEE 7.0.0. I have no problem with Hazelcast on it.
>
>
>
> · The first one is when I’m start Ignite in the
> applicationComposer (with a CDI @Produces), Ignite complain about some
> System.properties that are not String :
>
>
>
> it try to "serialize" System.properties by using
> System.getProperties().store(new PrintWriter(sw));
>
>
>
> In fact, TomEE (as other framework like hibernate) use System.properties
> to store prop objects.
>
>
>
> So I made something to remove this properties before Ignite start and
> recover its after. Well. Pehraps it will be nice to change the way to
> “serialize” System.properties.
>
>
>
> · The second problem is strange, very strange. If I’m making a
> call to Ignite.start in @Before or using a @Inject in the test class (that
> use the @Produce that make the call to start(), everything is ok. But if
> the start is called in the @Test method (so after ApplicationComposer has
> made some things), Ignite is “locked”.
>
> o   Precisely in IgniteKernel.java:917  :
>
>
>
>
> *// Start discovery manager last to make sure that grid is fully
> initialized. *startManager(discoMgr);
>
>
>
> The call to  this method never exit.
>
>
>
> Any idea to help me to understand ? Anybody has tried to use Ignite with
> EE or CDI ?
>
>
>
> Regards
>
>
>
>
>
>
>
> This message and any attachments (the "message") is
> intended solely for the intended addressees and is confidential.
> If you receive this message in error,or are not the intended recipient(s),
> please delete it and any copies from your systems and immediately notify
> the sender. Any unauthorized view, use that does not comply with its
> purpose,
> dissemination or disclosure, either whole or partial, is prohibited. Since
> the internet
> cannot guarantee the integrity of this message which may not be reliable,
> BNP PARIBAS
> (and its subsidiaries) shall not be liable for the message if modified,
> changed or falsified.
> Do not print this message unless it is necessary,consider the environment.
>
>
> --
>
> Ce message et toutes les pieces jointes (ci-apres le "message")
> sont etablis a l'intention exclusive de ses destinataires et sont
> confidentiels.
> Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
> merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
> immediatement l'expediteur. Toute lecture non autorisee, toute utilisation
> de
> ce message qui n'est pas conforme a sa destination, toute diffusion ou
> toute
> publication, totale ou partielle, est interdite. L'Internet ne permettant
> pas d'assurer
> l'integrite de ce message electronique susceptible d'alteration, BNP
> Paribas
> (et ses filiales) decline(nt) toute responsabilite au titre de ce message
> dans l'hypothese
> ou il aurait ete modifie, deforme ou falsifie.
> N'imprimez ce message que si necessaire, pensez a l'environnement.
>
>
>

Changing node attributes at runtime

[Matt Hoffman]

This was asked about a month ago, but the discussion ended up going a
different direction. I have a use case involving targeting computation to
nodes where the most natural answer _seems_ to be to be able to change node
attributes at runtime. I'm aware that right now node attributes can't be
changed at runtime; I'm curious if a.) there is a technical limitation why
this couldn't be supported, and b.) if there's perhaps a better way for me
to solve my problem.

I have a cluster of nodes, which can each have a list of tags indicating
whether a job should run on them. So I would like to be able to target jobs
to only those services that have a particular tag.
However, users can edit which tags apply to which nodes at runtime through
a UI. I can't restart nodes when tags are edited. I'm flexible about how I
store the tags -- I could store them in a cache or another central store,
for example. So the only alternative I can think of to having attributes
editable at runtime is to have a map of tags to cluster node IDs in a
central location, and explicitly build a ClusterGroup from that when
launching compute jobs. Does that sound reasonable? Is there a better way
to handle this kind of thing?


Thanks,


Matt