Possible starvation in striped pool in one of the node

[Lo, Marcus]

Hi,

We uses Ignite 2.10 with 5 nodes, and we can see one of the nodes with the 
below warning message, and the cluster becomes unresponsive at the same time. 
May I know what's the possible reason for that, and how we can resolve the 
issue? Thanks.

thread=grid-timeout-worker-#22%Ignite% 
class=org.apache.ignite.internal.util.typedef.G >>> Possible starvation in 
striped pool.
Thread name: sys-stripe-7-#8%Ignite%
Queue: [Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, 
topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, 
msg=GridDhtAtomicSingleUpdateRequest [key=BinaryObjectImpl [arr= true, 
ctx=false, start=0], val=BinaryObjectImpl [arr= true, ctx=false, start=0], 
prevVal=null, super=GridDhtAtomicAbstractUpdateRequest [onRes=false, 
nearNodeId=null, nearFutId=0, flags=keepBinary, Message closure 
[msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, 
timeout=0, skipOnTimeout=false, msg=GridDhtAtomicUpdateRequest [keys=ArrayList 
[BinaryObjectImpl [arr= true, ctx=false, start=0], BinaryObjectImpl [arr= true, 
ctx=false, start=0], BinaryObjectImpl [arr= true, ctx=false, start=0], 
BinaryObjectImpl [arr= true, ctx=false, start=0], BinaryObjectImpl [arr= true, 
ctx=false, start=0]], vals=ArrayList [BinaryObjectImpl [arr= true, ctx=false, 
start=0], BinaryObjectImpl [arr= true, ctx=false, start=0], BinaryObjectImpl 
[arr= true, ctx=false, start=0], BinaryObjectImpl [arr= true, ctx=false, 
start=0], BinaryObjectImpl [arr= true, ctx=false, start=0]], prevVals=null, 
ttls=null, conflictExpireTimes=null, nearTtls=null, nearExpireTimes=null, 
nearKeys=null, nearVals=null, obsoleteIndexes=null, 
forceTransformBackups=false, updateCntrs=GridLongList [idx=5, 
arr=[16537140,13196110,15013490,15241357,13947459]], 
super=GridDhtAtomicAbstractUpdateRequest [onRes=false, nearNodeId=null, 
nearFutId=0, flags=keepBinary, Message closure [msg=GridIoMessage [plc=2, 
topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, 
msg=GridDhtAtomicDeferredUpdateResponse [futIds=GridLongList [idx=1, 
arr=[1475340], Message closure [msg=GridIoMessage [plc=2, 
topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, 
msg=GridDhtAtomicDeferredUpdateResponse [futIds=GridLongList [idx=1, 
arr=[1475340], Message closure [msg=GridIoMessage [plc=2, 
topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, 
msg=GridDhtAtomicUpdateRequest [keys=ArrayList [BinaryObjectImpl [arr= true, 
ctx=false, start=0], BinaryObjectImpl [arr= true, ctx=false, start=0], 
BinaryObjectImpl [arr= true, ctx=false, start=0], BinaryObjectImpl [arr= true, 
ctx=false, start=0], BinaryObjectImpl [arr= true, ctx=false, start=0], 
BinaryObjectImpl [arr= true, ctx=false, start=0], BinaryObjectImpl [arr= true, 
ctx=false, start=0], BinaryObjectImpl [arr= true, ctx=false, start=0], 
BinaryObjectImpl [arr= true, ctx=false, start=0], BinaryObjectImpl [arr= true, 
ctx=false, start=0], BinaryObjectImpl [arr= true, ctx=false, start=0], 
BinaryObjectImpl [arr= true, ctx=false, start=0], BinaryObjectImpl [arr= true, 
ctx=false, start=0], BinaryObjectImpl [arr= true, ctx=false, start=0], 
BinaryObjectImpl [arr= true, ctx=false, start=0], BinaryObjectImpl [arr= true, 
ctx=false, start=0], BinaryObjectImpl [arr= true, ctx=false, start=0], 
BinaryObjectImpl [arr= true, ctx=false, start=0], BinaryObjectImpl [arr= true, 
ctx=false, start=0], BinaryObjectImpl [arr= true, ctx=false, start=0], 
BinaryObjectImpl [arr= true, ctx=false, start=0], BinaryObjectImpl [arr= true, 
ctx=false, start=0], BinaryObjectImpl [arr= true, ctx=false, start=0], 
BinaryObjectImpl [arr= true, ctx=false, start=0], BinaryObjectImpl [arr= true, 
ctx=false, start=0], BinaryObjectImpl [arr= true, ctx=false, start=0], 
BinaryObjectImpl [arr= true, ctx=false, start=0], BinaryObjectImpl [arr= true, 
ctx=false, start=0], BinaryObjectImpl [arr= true, ctx=false, start=0], 
BinaryObjectImpl [arr= true, ctx=false, start=0], BinaryObjectImpl [arr= true, 
ctx=false, start=0], BinaryObjectImpl [arr= true, ctx=false, start=0], 
BinaryObjectImpl [arr= true, ctx=false, start=0], BinaryObjectImpl [arr= true, 
ctx=false, start=0], BinaryObjectImpl [arr= true, ctx=false, start=0], 
BinaryObjectImpl [arr= true, ctx=false, start=0], BinaryObjectImpl [arr= true, 
ctx=false, start=0], BinaryObjectImpl [arr= true, ctx=false, start=0], 
BinaryObjectImpl [arr= true, ctx=false, start=0], BinaryObjectImpl [arr= true, 
ctx=false, start=0], BinaryObjectImpl [arr= true, ctx=false, start=0], 
BinaryObjectImpl [arr= true, ctx=false, start=0], BinaryObjectImpl [arr= true, 
ctx=false, start=0], BinaryObjectImpl [arr= true, ctx=false, start=0], 
BinaryObjectImpl [arr= true, ctx=false, start=0], BinaryObjectImpl [arr= true, 
ctx=false, start=0], BinaryObjectImpl [arr= true, ctx=false, start=0], 
BinaryObjectImpl [arr= true, ctx=false, start=0], BinaryObjectImpl [arr= true, 
ctx=false, start=0], BinaryObjectImpl [arr= true,

Re: Gridgain ultimate 2.8.18 : can't write to work directory

[Surinder Mehra]

Thanks Stephen. Will check with them

On Wed, 8 Jun 2022, 15:18 Stephen Darlington, <
stephen.darling...@gridgain.com> wrote:

> This is a mailing list for Apache ignite users. Please contact GridGain
> for support of their software.
>
> On 8 Jun 2022, at 09:53, Surinder Mehra  wrote:
>
> Hi,
> I deployed gridgain ultimate 2.8.18 to kubernetes with mount point for
> work directory to store ignite data as per steps mentioned  below
>
>
> https://www.gridgain.com/docs/latest/installation-guide/kubernetes/amazon-eks-deployment
>
> It works if I don't use an external  mount point for the work directory
> but  when I use the below configuration, it throws the below error
> saying it cannot write to the work directory.
> mounted volumes as mounted as root with below permission set
> *drwxr-xr-x -> root*
> But the container starts with user gridgain.
>
> Do those volumes need explicit permission or I am missing any other steps
> here ?
>
>
>
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> class org.apache.ignite.IgniteException: Cannot write to work directory:
> /opt/gridgain/work
> at
> org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:1130)
> at org.apache.ignite.Ignition.start(Ignition.java:347)
> at
> org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:358)
> Caused by: class org.apache.ignite.IgniteCheckedException: Cannot write to
> work directory: /opt/gridgain/work
> at
> org.apache.ignite.internal.util.IgniteUtils.workDirectory(IgniteUtils.java:10126)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.initializeConfiguration(IgnitionEx.java:1874)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1700)
> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1143)
> at
> org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1058)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:944)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:843)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:713)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:682)
> at org.apache.ignite.Ignition.start(Ignition.java:344)
> ... 1 more
> Failed to start grid: Cannot write to work directory: /opt/gridgain/work
>
>
>
>
> config:
>
> 
>
>
> 
>
> 
>  class="org.apache.ignite.configuration.DataStorageConfiguration">
> 
>  value="/opt/gridgain/walarchive"/>
> 
> 
> 
>  class="org.gridgain.grid.configuration.GridGainConfiguration">
> 
>  class="org.gridgain.grid.configuration.SnapshotConfiguration">
> 
> 
> 
> 
> 
> 
>  class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
>
> 
> 
>
> 
>
> statefulset config:
> 
> image: gridgain/ultimate:8.8.18-openjdk11-slim
> volumeMounts:
> - mountPath: /opt/gridgain/config
>   mountPropagation: None
>   name: config-vol
> - mountPath: /opt/gridgain/work
>   mountPropagation: None
>   name: work-vol
>
>
>

Re: LEFT, RIGHT JOIN not working

[Konstantin Orlov]

Hi, Surinder! Thank you for pointing out to the problem.

Seems we have a bug in subquery rewriter. I've filed a ticket for this case. [1]

As a possible workaround you could disable join rewriting by setting the system 
property IGNITE_ENABLE_SUBQUERY_REWRITE_OPTIMIZATION to false.


[1] https://issues.apache.org/jira/browse/IGNITE-17131

-- 
Regards,
Konstantin Orlov




> On 8 Jun 2022, at 09:08, Surinder Mehra  wrote:
> 
> Hi,
> Could you please provide an update on this.
> 
> On Mon, Jun 6, 2022 at 11:48 AM Zhenya Stanilovsky  > wrote:
> 
> Hi ! thanks for example, i hope some updates will be here in a short time.
> 
> 
>  
> Hi,
> Just wondering if you had an opportunity to look into this.
>  
> On Thu, Jun 2, 2022 at 2:52 PM Surinder Mehra  > wrote:
> Hi,
> Please find the attached java file which reproduces the issue. As you can 
> see, the cache key is used as a join condition but LEFT join is still giving 
> only common values.
>  
> output:
> [2, Keyboard, 2]
> Size of actual output 1
> Expected size 3 is not equal to Actual size 1
>  
>  
> On Thu, Jun 2, 2022 at 11:48 AM Zhenya Stanilovsky  > wrote:
> Hi, Surinder Mehra ! I check your sql and it work correct for me.
> You no need to define AffinityKeyMapped for Key, check additionally [1], you 
> can simple modify [2] according to your case
> I problem still exist somehow, plz attach some code example.
> thanks !
>  
> [1] 
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/affinity/AffinityKeyMapped.html
>  
> 
> [2] 
> https://github.com/apache/ignite/blob/master/modules/indexing/src/test/java/org/apache/ignite/internal/processors/cache/IgniteCacheJoinPartitionedAndReplicatedTest.java#L160
>  
> 
>  
>  
> Hi,
> I have the following sample code to demo issue in SQL joins. I have created 
> an affinity key and value as shown below and added some sample data to it. 
> When I try LEFT self join on this table it always gives me common rows   
> irrespective of LEFT or RIGHT JOIN
> Could you please help me find what am I doing wrong here.
>  
> cache Key :
> 
> public class OrderAffinityKey {
> Integer id;
> @AffinityKeyMapped
> Integer customerId;
> }
> 
> 
> cache value:
> 
> public class Order implements Serializable {
> @QuerySqlField
> Integer id;
> 
> @AffinityKeyMapped
> @QuerySqlField Integer customerId;
> @QuerySqlField String product;
> }
> 
> 
> Table C: (select customerID, product FROM "orderCache"."ORDER" WHERE 
> CUSTOMERID IN ( 1, 2))
> 
> 1 keyboard
> 2 Laptop
> 
> 
> Table O: (select customerID, product FROM "orderCache"."ORDER" WHERE 
> CUSTOMERID IN ( 3, 2))
> 
> 2 laptop
> 3 mouse
> 
> 
> 
> JOIN:
> 
> Query :
> select DISTINCT C.customerID, C.product, O.customerID
> FROM
>  (select customerID, product FROM "orderCache"."ORDER" WHERE CUSTOMERID IN ( 
> 1, 2)) C
>  LEFT JOIN
> (select customerID, product FROM "orderCache"."ORDER" WHERE CUSTOMERID IN ( 
> 3, 2)) O
> ON
> C.customerId = O.customerId
> 
> 
> Output:
> 
> 2 laptop   2
> 3 mouse   3
> 
> Expected output:
> 
> 1 keyboard   null
> 2 laptop   2
> 3 mouse   3
>  
>  
>  
>  
>  
>  
>  
>  

Re: Gridgain ultimate 2.8.18 : can't write to work directory

[Stephen Darlington]

This is a mailing list for Apache ignite users. Please contact GridGain for 
support of their software.

> On 8 Jun 2022, at 09:53, Surinder Mehra  wrote:
> 
> Hi,
> I deployed gridgain ultimate 2.8.18 to kubernetes with mount point for work 
> directory to store ignite data as per steps mentioned  below
> 
> https://www.gridgain.com/docs/latest/installation-guide/kubernetes/amazon-eks-deployment
>  
> 
> 
> It works if I don't use an external  mount point for the work directory but  
> when I use the below configuration, it throws the below error saying it 
> cannot write to the work directory. 
> mounted volumes as mounted as root with below permission set
> drwxr-xr-x -> root
> But the container starts with user gridgain.
> 
> Do those volumes need explicit permission or I am missing any other steps 
> here ?
> 
> 
> 
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder 
>  for further details.
> class org.apache.ignite.IgniteException: Cannot write to work directory: 
> /opt/gridgain/work
> at 
> org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:1130)
> at org.apache.ignite.Ignition.start(Ignition.java:347)
> at 
> org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:358)
> Caused by: class org.apache.ignite.IgniteCheckedException: Cannot write to 
> work directory: /opt/gridgain/work
> at 
> org.apache.ignite.internal.util.IgniteUtils.workDirectory(IgniteUtils.java:10126)
> at 
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.initializeConfiguration(IgnitionEx.java:1874)
> at 
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1700)
> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1143)
> at 
> org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1058)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:944)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:843)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:713)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:682)
> at org.apache.ignite.Ignition.start(Ignition.java:344)
> ... 1 more
> Failed to start grid: Cannot write to work directory: /opt/gridgain/work
> 
> 
> 
> 
> config:
> 
> 
> 
> 
> 
> 
> 
>  class="org.apache.ignite.configuration.DataStorageConfiguration">
> 
>  value="/opt/gridgain/walarchive"/>
> 
> 
> 
>  class="org.gridgain.grid.configuration.GridGainConfiguration">
> 
>  class="org.gridgain.grid.configuration.SnapshotConfiguration">
> 
> 
> 
> 
> 
> 
> 
>
> 
> 
> 
> 
> 
> statefulset config:
> 
> image: gridgain/ultimate:8.8.18-openjdk11-slim
> volumeMounts:
> - mountPath: /opt/gridgain/config
>   mountPropagation: None
>   name: config-vol
> - mountPath: /opt/gridgain/work
>   mountPropagation: None
>   name: work-vol

Gridgain ultimate 2.8.18 : can't write to work directory

[Surinder Mehra]

Hi,
I deployed gridgain ultimate 2.8.18 to kubernetes with mount point for work
directory to store ignite data as per steps mentioned  below

https://www.gridgain.com/docs/latest/installation-guide/kubernetes/amazon-eks-deployment

It works if I don't use an external  mount point for the work directory
but  when I use the below configuration, it throws the below error
saying it cannot write to the work directory.
mounted volumes as mounted as root with below permission set
*drwxr-xr-x -> root*
But the container starts with user gridgain.

Do those volumes need explicit permission or I am missing any other steps
here ?



SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
class org.apache.ignite.IgniteException: Cannot write to work directory:
/opt/gridgain/work
at
org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:1130)
at org.apache.ignite.Ignition.start(Ignition.java:347)
at
org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:358)
Caused by: class org.apache.ignite.IgniteCheckedException: Cannot write to
work directory: /opt/gridgain/work
at
org.apache.ignite.internal.util.IgniteUtils.workDirectory(IgniteUtils.java:10126)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.initializeConfiguration(IgnitionEx.java:1874)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1700)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1143)
at
org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1058)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:944)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:843)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:713)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:682)
at org.apache.ignite.Ignition.start(Ignition.java:344)
... 1 more
Failed to start grid: Cannot write to work directory: /opt/gridgain/work




config:



   



















   





statefulset config:

image: gridgain/ultimate:8.8.18-openjdk11-slim
volumeMounts:
- mountPath: /opt/gridgain/config
  mountPropagation: None
  name: config-vol
- mountPath: /opt/gridgain/work
  mountPropagation: None
  name: work-vol