[jira] [Assigned] (IGNITE-3994) Client buffer CacheContinuousQueryEntry on pendingEvts after reconnect to alive cluster

[Vladislav Pyatkov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladislav Pyatkov reassigned IGNITE-3994:
-

Assignee: Vladislav Pyatkov

> Client buffer CacheContinuousQueryEntry on pendingEvts after reconnect to 
> alive cluster
> ---
>
> Key: IGNITE-3994
> URL: https://issues.apache.org/jira/browse/IGNITE-3994
> Project: Ignite
>  Issue Type: Bug
>Reporter: Vladislav Pyatkov
>Assignee: Vladislav Pyatkov
> Attachments: ClientReconnectContinuousQueryTest.java
>
>
> The CacheContinuousQueryEntry will accumulate until buffer will not full.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4205) CassandraCacheStore should start IgniteThread threads in loadCache() method

[Igor Rudyak (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724582#comment-15724582
 ] 

Igor Rudyak commented on IGNITE-4205:
-

Valentin I can easily do this, but it looks like IgniteThread implementation 
doesn't have anything special related to marchalling/unmarshalling - just only 
a number of getters and setters. Thus it looks like it shouldn't work. Am I 
wrong?

> CassandraCacheStore should start IgniteThread threads in loadCache() method
> ---
>
> Key: IGNITE-4205
> URL: https://issues.apache.org/jira/browse/IGNITE-4205
> Project: Ignite
>  Issue Type: Bug
>  Components: cache
>Affects Versions: 1.7
>Reporter: Valentin Kulichenko
>Assignee: Igor Rudyak
>
> {{CassandraCacheStore.loadCache()}} method starts a generic thread pool for 
> parallel data load. Threads in this thread pool can't deserialize Ignite 
> internal objects (e.g. {{IgniteKernal}}) which can cause unexpected behavior. 
> Here is one of the scenarios:
> * There is column in Cassandra which stores an object as BLOB using 
> {{JavaSerializer}}.
> * {{CacheConfiguration.storeKeepBinary}} is {{true}}.
> * When an object is saved, it's passed to the store as an instance of 
> {{BinaryObject}} which is converted to a byte array and saved in Cassandra.
> * When the same object is loaded in {{loadCache}}, the store takes the byte 
> array and tries to convert it to {{BinaryObject}}. But it can't because this 
> implies calling {{IgnitionEx.localIgnite()}} from non-Ignite thread.
> To fix this we need to provide a thread factory that will create instances of 
> {{IgniteThread}} and use it in the pool that loads the data.
> Most likely the same issue exists in {{CacheAbstractJdbcStore}}.
> And in general, any threads created by Ignite internals should be 
> {{IgniteThread}}-s. This should be revisited.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-4371) Neat TX finish request processing may fall into sync wait of dht finish response

[Yakov Zhdanov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yakov Zhdanov updated IGNITE-4371:
--
Description: 
After fixing this please remove TODO and put proper partition ID to 
org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxFinishResponse#partition

{noformat}
Thread 
[name="sys-stripe-1-#30%dht.GridCacheColocatedTxExceptionSelfTest1%", id=46, 
state=WAITING, blockCnt=0, waitCnt=7]
Lock 
[object=o.a.i.i.processors.cache.distributed.dht.GridDhtTxFinishFuture@212874c9,
 ownerName=null, ownerId=-1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at 
o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:159)
at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:117)
at 
o.a.i.i.processors.cache.distributed.dht.GridDhtTxFinishFuture.onError(GridDhtTxFinishFuture.java:183)
at 
o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.finishCommit(GridDhtTxLocal.java:543)
at 
o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.commitAsync(GridDhtTxLocal.java:580)
at 
o.a.i.i.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:849)
at 
o.a.i.i.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:728)
at 
o.a.i.i.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:687)
at 
o.a.i.i.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:157)
at 
o.a.i.i.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:155)
at 
o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:758)
at 
o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:363)
at 
o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:287)
at 
o.a.i.i.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:89)
at 
o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:232)
at 
o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1212)
at 
o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:840)
at 
o.a.i.i.managers.communication.GridIoManager.access$2100(GridIoManager.java:110)
at 
o.a.i.i.managers.communication.GridIoManager$6.run(GridIoManager.java:785)
at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:362)
at java.lang.Thread.run(Thread.java:724)
{noformat}

  was:
{noformat}
Thread 
[name="sys-stripe-1-#30%dht.GridCacheColocatedTxExceptionSelfTest1%", id=46, 
state=WAITING, blockCnt=0, waitCnt=7]
Lock 
[object=o.a.i.i.processors.cache.distributed.dht.GridDhtTxFinishFuture@212874c9,
 ownerName=null, ownerId=-1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at 
o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:159)
at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:117)
at 
o.a.i.i.processors.cache.distributed.dht.GridDhtTxFinishFuture.onError(GridDhtTxFinishFuture.java:183)
at 
o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.finishCommit(GridDhtTxLocal.java:543)
at 
o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.commitAsync(GridDhtTxLocal.java:580)
at 
o.a.i.i.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:849)
at 
o.a.i.i.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:728)
at 
o.a.i.i.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:687)
at 
o.a.i.i.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:157)
at 
o.a.i.i.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:155)
at 

[jira] [Created] (IGNITE-4371) Neat TX finish request processing may fall into sync wait of dht finish response

[Yakov Zhdanov (JIRA)]

Yakov Zhdanov created IGNITE-4371:
-

 Summary: Neat TX finish request processing may fall into sync wait 
of dht finish response
 Key: IGNITE-4371
 URL: https://issues.apache.org/jira/browse/IGNITE-4371
 Project: Ignite
  Issue Type: Bug
  Components: cache
Reporter: Yakov Zhdanov
Assignee: Semen Boikov


{noformat}
Thread 
[name="sys-stripe-1-#30%dht.GridCacheColocatedTxExceptionSelfTest1%", id=46, 
state=WAITING, blockCnt=0, waitCnt=7]
Lock 
[object=o.a.i.i.processors.cache.distributed.dht.GridDhtTxFinishFuture@212874c9,
 ownerName=null, ownerId=-1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at 
o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:159)
at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:117)
at 
o.a.i.i.processors.cache.distributed.dht.GridDhtTxFinishFuture.onError(GridDhtTxFinishFuture.java:183)
at 
o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.finishCommit(GridDhtTxLocal.java:543)
at 
o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.commitAsync(GridDhtTxLocal.java:580)
at 
o.a.i.i.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:849)
at 
o.a.i.i.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:728)
at 
o.a.i.i.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:687)
at 
o.a.i.i.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:157)
at 
o.a.i.i.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:155)
at 
o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:758)
at 
o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:363)
at 
o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:287)
at 
o.a.i.i.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:89)
at 
o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:232)
at 
o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1212)
at 
o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:840)
at 
o.a.i.i.managers.communication.GridIoManager.access$2100(GridIoManager.java:110)
at 
o.a.i.i.managers.communication.GridIoManager$6.run(GridIoManager.java:785)
at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:362)
at java.lang.Thread.run(Thread.java:724)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-4369) ReentrantLocks hangs on unlock when node is stopped.

[Yakov Zhdanov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yakov Zhdanov updated IGNITE-4369:
--
Summary: ReentrantLocks hangs on unlock when node is stopped.  (was: 
ReentranLocks hands on unlock when node is stopped.)

> ReentrantLocks hangs on unlock when node is stopped.
> 
>
> Key: IGNITE-4369
> URL: https://issues.apache.org/jira/browse/IGNITE-4369
> Project: Ignite
>  Issue Type: Bug
>  Components: cache
>Affects Versions: 1.7
>Reporter: Taras Ledkov
> Fix For: 1.9
>
>  Time Spent: 4m
>  Remaining Estimate: 0h
>
> The thread hangs on GridCacheLockImpl.unlock when other node is stopped.
> {code}
> at java.lang.Thread.yield(Native Method)
>   at 
> org.apache.ignite.internal.processors.datastructures.GridCacheLockImpl$Sync.tryRelease(GridCacheLockImpl.java:469)
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1260)
>   at 
> org.apache.ignite.internal.processors.datastructures.GridCacheLockImpl.unlock(GridCacheLockImpl.java:1296)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (IGNITE-1050) Document dynamic caches on readme.io

[Denis Magda (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-1050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denis Magda resolved IGNITE-1050.
-
Resolution: Fixed

> Document dynamic caches on readme.io
> 
>
> Key: IGNITE-1050
> URL: https://issues.apache.org/jira/browse/IGNITE-1050
> Project: Ignite
>  Issue Type: Task
>  Components: documentation
>Reporter: Yakov Zhdanov
>Priority: Critical
>
> http://apacheignite.readme.io/
> create new page under Data Grid section - Dynamic Caches



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (IGNITE-1050) Document dynamic caches on readme.io

[Denis Magda (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-1050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denis Magda closed IGNITE-1050.
---

> Document dynamic caches on readme.io
> 
>
> Key: IGNITE-1050
> URL: https://issues.apache.org/jira/browse/IGNITE-1050
> Project: Ignite
>  Issue Type: Task
>  Components: documentation
>Reporter: Yakov Zhdanov
>Priority: Critical
>
> http://apacheignite.readme.io/
> create new page under Data Grid section - Dynamic Caches



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-1050) Document dynamic caches on readme.io

[Denis Magda (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-1050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722968#comment-15722968
 ] 

Denis Magda commented on IGNITE-1050:
-

There is already a section about this capability
https://apacheignite.readme.io/docs/jcache#section-dynamic-cache

> Document dynamic caches on readme.io
> 
>
> Key: IGNITE-1050
> URL: https://issues.apache.org/jira/browse/IGNITE-1050
> Project: Ignite
>  Issue Type: Task
>  Components: documentation
>Reporter: Yakov Zhdanov
>Priority: Critical
>
> http://apacheignite.readme.io/
> create new page under Data Grid section - Dynamic Caches



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-4358) Better error reporting in case of unmarshallable classes.

[Rohit Mohta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohit Mohta updated IGNITE-4358:

Attachment: IGNITE-4358-Exceptionlog-05Dec2016.txt

> Better error reporting in case of unmarshallable classes.
> -
>
> Key: IGNITE-4358
> URL: https://issues.apache.org/jira/browse/IGNITE-4358
> Project: Ignite
>  Issue Type: Improvement
>  Components: compute, messaging, newbie
>Affects Versions: 1.6
>Reporter: Alexei Scherbakov
>Assignee: Rohit Mohta
>Priority: Trivial
>  Labels: newbie
> Fix For: 2,0
>
> Attachments: IGNITE-4358-Exceptionlog-05Dec2016.txt, 
> IGNITE-4358-GridClosureProcessor-05Dec2016.patch, PureIgniteRunTest.java
>
>
> When trying to execute Thread's derived class implementing IgniteRunnable 
> using compute API, it silently serializes to null because Thread 
> serialization is prohibited in MarshallerExclusions and throws NPE on 
> executing node.
> We need to throw more informative exception for such case.
> Reproducer in the attachment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (IGNITE-1049) Document deployment SPI on readme.io

[Denis Magda (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-1049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denis Magda resolved IGNITE-1049.
-
Resolution: Fixed

> Document deployment SPI on readme.io
> 
>
> Key: IGNITE-1049
> URL: https://issues.apache.org/jira/browse/IGNITE-1049
> Project: Ignite
>  Issue Type: Task
>  Components: documentation
>Reporter: Yakov Zhdanov
>Priority: Critical
>
> http://apacheignite.readme.io/
> create new page under Compute Grid section - Code Deloyment



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-1049) Document deployment SPI on readme.io

[Denis Magda (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-1049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722961#comment-15722961
 ] 

Denis Magda commented on IGNITE-1049:
-

The page is already there
https://apacheignite.readme.io/docs/deployment-spi

> Document deployment SPI on readme.io
> 
>
> Key: IGNITE-1049
> URL: https://issues.apache.org/jira/browse/IGNITE-1049
> Project: Ignite
>  Issue Type: Task
>  Components: documentation
>Reporter: Yakov Zhdanov
>Priority: Critical
>
> http://apacheignite.readme.io/
> create new page under Compute Grid section - Code Deloyment



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4039) Track and Improve SQL Performance Against Alternatives

[Denis Magda (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722958#comment-15722958
 ] 

Denis Magda commented on IGNITE-4039:
-

This is being constantly done. No need to keep this ticket opened.

> Track and Improve SQL Performance Against Alternatives
> --
>
> Key: IGNITE-4039
> URL: https://issues.apache.org/jira/browse/IGNITE-4039
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Suminda Dharmasena
>Priority: Blocker
>
> Can you track and improve performance against:
>  - https://www.questdb.org/
>  - https://clickhouse.yandex/benchmark.html
>  - http://www.scylladb.com/
>  - http://www.aerospike.com/
>  - http://kudu.apache.org/
>  - http://traildb.io/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (IGNITE-4039) Track and Improve SQL Performance Against Alternatives

[Denis Magda (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denis Magda closed IGNITE-4039.
---

> Track and Improve SQL Performance Against Alternatives
> --
>
> Key: IGNITE-4039
> URL: https://issues.apache.org/jira/browse/IGNITE-4039
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Suminda Dharmasena
>Priority: Blocker
>
> Can you track and improve performance against:
>  - https://www.questdb.org/
>  - https://clickhouse.yandex/benchmark.html
>  - http://www.scylladb.com/
>  - http://www.aerospike.com/
>  - http://kudu.apache.org/
>  - http://traildb.io/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (IGNITE-4039) Track and Improve SQL Performance Against Alternatives

[Denis Magda (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denis Magda resolved IGNITE-4039.
-
Resolution: Fixed

> Track and Improve SQL Performance Against Alternatives
> --
>
> Key: IGNITE-4039
> URL: https://issues.apache.org/jira/browse/IGNITE-4039
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Suminda Dharmasena
>Priority: Blocker
>
> Can you track and improve performance against:
>  - https://www.questdb.org/
>  - https://clickhouse.yandex/benchmark.html
>  - http://www.scylladb.com/
>  - http://www.aerospike.com/
>  - http://kudu.apache.org/
>  - http://traildb.io/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-4358) Better error reporting in case of unmarshallable classes.

[Rohit Mohta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohit Mohta updated IGNITE-4358:

Attachment: IGNITE-4358-GridClosureProcessor-05Dec2016.patch

Validate non-null runnable. Custom message to give more information, why is the 
user seeing NPE

> Better error reporting in case of unmarshallable classes.
> -
>
> Key: IGNITE-4358
> URL: https://issues.apache.org/jira/browse/IGNITE-4358
> Project: Ignite
>  Issue Type: Improvement
>  Components: compute, messaging, newbie
>Affects Versions: 1.6
>Reporter: Alexei Scherbakov
>Assignee: Rohit Mohta
>Priority: Trivial
>  Labels: newbie
> Fix For: 2,0
>
> Attachments: IGNITE-4358-GridClosureProcessor-05Dec2016.patch, 
> PureIgniteRunTest.java
>
>
> When trying to execute Thread's derived class implementing IgniteRunnable 
> using compute API, it silently serializes to null because Thread 
> serialization is prohibited in MarshallerExclusions and throws NPE on 
> executing node.
> We need to throw more informative exception for such case.
> Reproducer in the attachment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4358) Better error reporting in case of unmarshallable classes.

[Rohit Mohta (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722931#comment-15722931
 ] 

Rohit Mohta commented on IGNITE-4358:
-

[~ascherbakov] Is the below message helpful?

{color:red}
Caused by: java.lang.NullPointerException: received null runnable in Grid. Did 
we serialize only excluded classes? See MarshallerExclusions for the exclusion 
list.
{color}

> Better error reporting in case of unmarshallable classes.
> -
>
> Key: IGNITE-4358
> URL: https://issues.apache.org/jira/browse/IGNITE-4358
> Project: Ignite
>  Issue Type: Improvement
>  Components: compute, messaging, newbie
>Affects Versions: 1.6
>Reporter: Alexei Scherbakov
>Assignee: Rohit Mohta
>Priority: Trivial
>  Labels: newbie
> Fix For: 2,0
>
> Attachments: PureIgniteRunTest.java
>
>
> When trying to execute Thread's derived class implementing IgniteRunnable 
> using compute API, it silently serializes to null because Thread 
> serialization is prohibited in MarshallerExclusions and throws NPE on 
> executing node.
> We need to throw more informative exception for such case.
> Reproducer in the attachment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4045) .NET: Support DML API

[Pavel Tupitsyn (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722918#comment-15722918
 ] 

Pavel Tupitsyn commented on IGNITE-4045:


Identity resolvers implemented. In some cases the resolver is ignored, 
investigating.

> .NET: Support DML API
> -
>
> Key: IGNITE-4045
> URL: https://issues.apache.org/jira/browse/IGNITE-4045
> Project: Ignite
>  Issue Type: Task
>  Components: platforms
>Reporter: Denis Magda
>Assignee: Pavel Tupitsyn
>  Labels: roadmap
> Fix For: 2.0
>
>
> Ignite's Java component will provide support for DML soon (IGNITE-2294). At 
> she same time DML will be supported at the level of ODBC and JDBC drivers.
> As the next step we should include the similar functionality into Ignite.NET 
> by doing the following:
> - Implement DML API;
> - Enhance {{QueryExample.cs}} by doing INSERTs instead of cache.puts and 
> adding UPDATE and DELETE operation examples.
> - Add documentation to Ignite.NET readme.io covering the feature. Most like 
> most of the content can be take from the general documentation when this 
> ticket IGNITE-4018 is ready 
> (https://apacheignite.readme.io/docs/distributed-dml).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4106) SQL: parallelize sql queries over cache local partitions

[Andrew Mashenkov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722743#comment-15722743
 ] 

Andrew Mashenkov commented on IGNITE-4106:
--

Segmented tree index implementation moved to separate class. 
ReduceQueryExecutor now sends only one query request per node. 
MapQueryExecutor bothered with starting query threads if needed.
Corresponding configuration options added.



> SQL: parallelize sql queries over cache local partitions
> 
>
> Key: IGNITE-4106
> URL: https://issues.apache.org/jira/browse/IGNITE-4106
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.6, 1.7
>Reporter: Andrew Mashenkov
>Assignee: Andrew Mashenkov
>  Labels: performance, sql
> Fix For: 2.0
>
> Attachments: 1node-4thread.jfr, 4node-1thread.jfr
>
>
> If we run SQL query on cache partitioned over several cluster nodes, it will 
> be split into several queries running in parallel. But really we will have 
> one thread per query on each node.
> So, for now, to improve SQL query performance we need to run more Ignite 
> instances or split caches manually.
> It seems to be better to split local SQL queries over cache partitions, so we 
> would be able to parallelize SQL query on every single node and utilize CPU 
> more efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-3292) Yardstick: add logging of preloading progress

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-3292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722611#comment-15722611
 ] 

ASF GitHub Bot commented on IGNITE-3292:


GitHub user oleg-ostanin opened a pull request:

https://github.com/apache/ignite/pull/1317

IGNITE-3292 Yardstick: add logging of preloading progress



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-3292

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1317.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1317


commit e60dadda47f66902132cbf22f49743741d01c9a5
Author: oleg-ostanin 
Date:   2016-12-05T15:41:26Z

IGNITE-3292 Yardstick: add logging of preloading progress




> Yardstick: add logging of preloading progress
> -
>
> Key: IGNITE-3292
> URL: https://issues.apache.org/jira/browse/IGNITE-3292
> Project: Ignite
>  Issue Type: Wish
>  Components: yardstick
>Reporter: Ksenia Rybakova
>Assignee: Oleg Ostanin
> Fix For: 1.9
>
>
> Please, add logging of preloading progress. This is really useful when we 
> load a lot of entries or they are large. 
> For instance: "Preloading: 200 of 1000 loaded".
> Also adding an option that controls frequency of such output makes sense. For 
> instance, this might be a step (number of entries loaded) - if entries are 
> large, we set small step, if entries are integers,  the step will be large.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-4370) ODBC: Implement DML operations with parameters in batch.

[Igor Sapego (JIRA)]

Igor Sapego created IGNITE-4370:
---

 Summary: ODBC: Implement DML operations with parameters in batch.
 Key: IGNITE-4370
 URL: https://issues.apache.org/jira/browse/IGNITE-4370
 Project: Ignite
  Issue Type: Task
  Components: odbc
Affects Versions: 1.7
Reporter: Igor Sapego
Assignee: Igor Sapego
 Fix For: 2.0


Currently, if user wants for example to insert 20k records they need to call 
{{SQLExecute}} for the 20k times. More than that, internally, we transmit and 
execute the same SQL query 20k times. This is a huge overhead. We should only 
transfer sql query once, transfer all the parameters in a batch and then 
execute it once using some fast, possibly internal API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-3558) Affinity task hangs when Collision SPI produces a lot of job rejections & Failover SPI produces many attempts

[Taras Ledkov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-3558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722540#comment-15722540
 ] 

Taras Ledkov commented on IGNITE-3558:
--

Pull request to run tests: 
[pull/1316|https://github.com/apache/ignite/pull/1316]

> Affinity task hangs when Collision SPI produces a lot of job rejections & 
> Failover SPI produces many attempts
> -
>
> Key: IGNITE-3558
> URL: https://issues.apache.org/jira/browse/IGNITE-3558
> Project: Ignite
>  Issue Type: Bug
>  Components: compute
>Reporter: Taras Ledkov
>Assignee: Taras Ledkov
> Fix For: 2.0
>
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> The test to reproduce:
> IgniteCacheLockPartitionOnAffinityRunWithCollisionSpiTest#testJobFinishing
> *Root cause*
> GridJobExecuteResponse isn't set from target node because there is a 
> confusion with GridJobWorker instances in the CollisionContext.
> *Suggestion*
> The method GridJobProcessor.CollisionJobContext.cancel()
> use passiveJobs.remove(jobWorker.getJobId(), jobWorker). 
> *passiveJobs* is a ConcurrentHashMap and GridJobWorker.equals() implements as 
> a equation of jobId.
> So, when two thread try to cancel the two workers with *the same jobIds* we 
> have the case:
> - thread0 remove jobWorker0 & cancel jobWorker0.
> - thread0 put jobWorker1 (because jobWorker0 already removed);
> - thread1: (has a copy of jobWorker0) and try to cancel it.
> - thread1: remove jobWorker1 instead of jobWorker0 (because jobId is used to 
> identify);
> - thread1: doesn't send ExecuteResponse because jobWorker0 has been canceled.
> *Proposal*
> Try to use system default equals for the GridJobWorker



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4341) Add TeraSort example as a unit test to Ignite

[Ivan Veselovsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722519#comment-15722519
 ] 

Ivan Veselovsky commented on IGNITE-4341:
-

2 problems were solved while incorporating this test:
1) distributed cache mechanism was not working if the Hadoop job submitted with 
grid().hadoop().submit(); This is related to the fact that real Hadoop client 
sets job property {code}MRJobConfig.MAPREDUCE_JOB_DIR{code} , while test job 
submission does not. As a result, 
org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2JobResourceManager#prepareJobEnvironment
 method did not copy the distributed resources to the local directories, while 
it should: the "mapreduce.job.dir" property only defines the staging directory, 
it is not directly related to distributed caching mechanism.
2) Special local file system implementation 
(org.apache.ignite.internal.processors.hadoop.impl.fs.HadoopLocalFileSystemV1) 
used in most of Hadoop tests, while original  
org.apache.hadoop.fs.LocalFileSystem is used in terasort test. But since the 
instance of file system is cached in FileSystem.CACHE, an instance from 
previous tests were actually used. This is fixed by adding 
org.apache.ignite.internal.processors.hadoop.impl.fs.HadoopFileSystemsUtils#clearFileSystemCache
 method and invoking it from #beforeTestsStarted and #beforeTest in Hadoop test 
suites. 

> Add TeraSort example as a unit test to Ignite
> -
>
> Key: IGNITE-4341
> URL: https://issues.apache.org/jira/browse/IGNITE-4341
> Project: Ignite
>  Issue Type: Test
>  Components: hadoop
>Affects Versions: 1.7
>Reporter: Ivan Veselovsky
>Assignee: Ivan Veselovsky
> Fix For: 2.0
>
>
> Add canonical TeraSort example as a unit test. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4368) Data Streamer closed exception in Kafka Streamer

[Roman Shtykh (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722391#comment-15722391
 ] 

Roman Shtykh commented on IGNITE-4368:
--

[~adasari] Do you have a code to reproduce the issue?

> Data Streamer closed exception in Kafka Streamer
> 
>
> Key: IGNITE-4368
> URL: https://issues.apache.org/jira/browse/IGNITE-4368
> Project: Ignite
>  Issue Type: Bug
>  Components: streaming
>Affects Versions: 1.7
>Reporter: Anil
>
> There is a possibility of data streamer closed exception in the following case
> 1. Lets says Kafka streamer is created when application is stared.
> 2. application node join the ignite cluster
> 3. Node disconnects from cluster 
> 4. Node joins the cluster after failure connect time and network timeout time.
> Data streamer closed exception happens between #3 and #4. still an issue post 
> #4 ?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (IGNITE-4281) Hadoop: decouple mapper and reducer maps.

[Vladimir Ozerov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov resolved IGNITE-4281.
-
Resolution: Fixed

> Hadoop: decouple mapper and reducer maps.
> -
>
> Key: IGNITE-4281
> URL: https://issues.apache.org/jira/browse/IGNITE-4281
> Project: Ignite
>  Issue Type: Sub-task
>  Components: hadoop
>Affects Versions: 1.7, 1.8
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
>  Labels: performance
> Fix For: 2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4281) Hadoop: decouple mapper and reducer maps.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722357#comment-15722357
 ] 

ASF GitHub Bot commented on IGNITE-4281:


Github user devozerov closed the pull request at:

https://github.com/apache/ignite/pull/1315


> Hadoop: decouple mapper and reducer maps.
> -
>
> Key: IGNITE-4281
> URL: https://issues.apache.org/jira/browse/IGNITE-4281
> Project: Ignite
>  Issue Type: Sub-task
>  Components: hadoop
>Affects Versions: 1.7, 1.8
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
>  Labels: performance
> Fix For: 2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-4301) Hadoop: send only one shuffle ack at the very end of shuffle processing.

[Vladimir Ozerov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov updated IGNITE-4301:

Labels: performance  (was: )

> Hadoop: send only one shuffle ack at the very end of shuffle processing.
> 
>
> Key: IGNITE-4301
> URL: https://issues.apache.org/jira/browse/IGNITE-4301
> Project: Ignite
>  Issue Type: Sub-task
>  Components: general, hadoop
>Affects Versions: 1.7, 1.8
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
>  Labels: performance
> Fix For: 2.0
>
>
> This shuffle acks will be processed ASAP, hence minimizing wait during 
> completion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-4355) Hadoop: eliminate map threads pauses during startup

[Vladimir Ozerov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov updated IGNITE-4355:

Labels: performance  (was: )

> Hadoop: eliminate map threads pauses during startup
> ---
>
> Key: IGNITE-4355
> URL: https://issues.apache.org/jira/browse/IGNITE-4355
> Project: Ignite
>  Issue Type: Sub-task
>  Components: hadoop
>Reporter: Ivan Veselovsky
>Assignee: Vladimir Ozerov
>  Labels: performance
> Fix For: 2.0
>
>
> Pauses in all Map threads but one are observed in the beginning . This is 
> caused by waiting on future.get()  in 
> HadoopV2Job.getTaskContext(HadoopTaskInfo) .
> {code}
>  at sun.misc.Unsafe.park(boolean, long)
>  at java.util.concurrent.locks.LockSupport.park(Object)
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int)
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int)
>  at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(boolean)
>  at org.apache.ignite.internal.util.future.GridFutureAdapter.get()
>  at 
> org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2Job.getTaskContext(HadoopTaskInfo)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffleJob.(Object,
>  IgniteLogger, HadoopJob, GridUnsafeMemory, int, int[], int)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffle.newJob(HadoopJobId)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffle.job(HadoopJobId)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffle.output(HadoopTaskContext)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopEmbeddedTaskExecutor$1.createOutput(HadoopTaskContext)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.createOutputInternal(HadoopTaskContext)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopPerformanceCounter)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call0()
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask$1.call()
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask$1.call()
>  at 
> org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2TaskContext.runAsJobOwner(Callable)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call()
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call()
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body()
>  at org.apache.ignite.internal.util.worker.GridWorker.run()
>  at java.lang.Thread.run()
> {code}
> while the working thread initializes the context:
> {code}
> Java Monitor Wait
>  at java.lang.Object.wait(long)
>  at java.lang.Thread.join(long)
>  at java.lang.Thread.join()
>  at org.apache.hadoop.util.Shell.joinThread(Thread)
>  at org.apache.hadoop.util.Shell.runCommand()
>  at org.apache.hadoop.util.Shell.run()
>  at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute()
>  at org.apache.hadoop.util.Shell.isSetsidSupported()
>  at org.apache.hadoop.util.Shell.()
>  at org.apache.hadoop.util.StringUtils.()
>  at 
> org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(Configuration)
>  at org.apache.hadoop.security.UserGroupInformation.initialize(Configuration, 
> boolean)
>  at org.apache.hadoop.security.UserGroupInformation.ensureInitialized()
>  at 
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(Subject)
>  at org.apache.hadoop.security.UserGroupInformation.getLoginUser()
>  at org.apache.hadoop.security.UserGroupInformation.getCurrentUser()
>  at org.apache.hadoop.mapreduce.task.JobContextImpl.(Configuration, 
> JobID)
>  at org.apache.hadoop.mapred.JobContextImpl.(JobConf, JobID, 
> Progressable)
>  at org.apache.hadoop.mapred.JobContextImpl.(JobConf, JobID)
>  at 
> org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2TaskContext.(HadoopTaskInfo,
>  HadoopJob, HadoopJobId, UUID, DataInput)
>  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Constructor, 
> Object[])
>  at sun.reflect.NativeConstructorAccessorImpl.newInstance(Object[])
>  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Object[])
>  at java.lang.reflect.Constructor.newInstance(Object[])
>  at 
> org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2Job.getTaskContext(HadoopTaskInfo)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffleJob.(Object,
>  IgniteLogger, HadoopJob, GridUnsafeMemory, int, int[], 

[jira] [Updated] (IGNITE-4276) Hadoop: control shuffle jobs "sleep" backpressure with property.

[Vladimir Ozerov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov updated IGNITE-4276:

Labels: performance  (was: )

> Hadoop: control shuffle jobs "sleep" backpressure with property.
> 
>
> Key: IGNITE-4276
> URL: https://issues.apache.org/jira/browse/IGNITE-4276
> Project: Ignite
>  Issue Type: Sub-task
>  Components: hadoop
>Affects Versions: 1.8
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
>  Labels: performance
> Fix For: 2.0
>
>
> Currently we simply sleep between shuffle job iterations for 5 milliseconds 
> (hard-coded). Let's get more control around it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-4281) Hadoop: decouple mapper and reducer maps.

[Vladimir Ozerov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov updated IGNITE-4281:

Labels: performance  (was: )

> Hadoop: decouple mapper and reducer maps.
> -
>
> Key: IGNITE-4281
> URL: https://issues.apache.org/jira/browse/IGNITE-4281
> Project: Ignite
>  Issue Type: Sub-task
>  Components: hadoop
>Affects Versions: 1.7, 1.8
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
>  Labels: performance
> Fix For: 2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-4274) Hadoop: control shuffle message buffer size through property.

[Vladimir Ozerov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov updated IGNITE-4274:

Labels: performance  (was: )

> Hadoop: control shuffle message buffer size through property.
> -
>
> Key: IGNITE-4274
> URL: https://issues.apache.org/jira/browse/IGNITE-4274
> Project: Ignite
>  Issue Type: Sub-task
>  Components: hadoop
>Affects Versions: 1.8
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
>  Labels: performance
> Fix For: 2.0
>
>
> Currently it is hard-coded to 128Kb.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-4263) Hadoop: abstract out offheap/heap memory management.

[Vladimir Ozerov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov updated IGNITE-4263:

Labels: performance  (was: )

> Hadoop: abstract out offheap/heap memory management.
> 
>
> Key: IGNITE-4263
> URL: https://issues.apache.org/jira/browse/IGNITE-4263
> Project: Ignite
>  Issue Type: Sub-task
>  Components: hadoop
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Taras Ledkov
>  Labels: performance
> Fix For: 2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-4270) Hadoop: optionally stripe mapper output for every partition.

[Vladimir Ozerov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov updated IGNITE-4270:

Labels: performance  (was: )

> Hadoop: optionally stripe mapper output for every partition.
> 
>
> Key: IGNITE-4270
> URL: https://issues.apache.org/jira/browse/IGNITE-4270
> Project: Ignite
>  Issue Type: Sub-task
>  Components: hadoop
>Affects Versions: 1.8
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
>  Labels: performance
> Fix For: 2.0
>
>
> Currently we have R maps for M mappers, where R is number of reducers. For 
> this reason many mappers writes to concurrent offheap data structure, loosing 
> time on concurrency burden.
> Let's add an option to create R * M maps, so that every mapper has dedicated 
> map for every reducer. This will eliminate almost all concurrency overhead.
> Design:
> 1) Every mapper works with it's own set of "remote" output maps;
> 2) These maps are essentially not "maps", but IO messages, which we fill up 
> to certain threshold;
> 3) Once filled, message is sent to remote node.
> 4) Async shuffle thread is no longer need in this architecture.
> As a result we decrease concurrency, removes slowdown from a single shuffle 
> thread which is not able to send messages fast enough, and removes 
> unnecessary intermediate sorting.
> NB! Be careful with "combiner" case and with "external" execution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-4271) Hadoop messages must use "direct marshallable" infrastructure.

[Vladimir Ozerov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov updated IGNITE-4271:

Labels: performance  (was: )

> Hadoop messages must use "direct marshallable" infrastructure.
> --
>
> Key: IGNITE-4271
> URL: https://issues.apache.org/jira/browse/IGNITE-4271
> Project: Ignite
>  Issue Type: Sub-task
>  Components: hadoop
>Affects Versions: 1.7, 1.8
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
>  Labels: performance
> Fix For: 2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-4277) Hadoop: support raw comparator.

[Vladimir Ozerov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov updated IGNITE-4277:

Labels: performance  (was: )

> Hadoop: support raw comparator.
> ---
>
> Key: IGNITE-4277
> URL: https://issues.apache.org/jira/browse/IGNITE-4277
> Project: Ignite
>  Issue Type: Sub-task
>  Components: hadoop
>Affects Versions: 1.8
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
>  Labels: performance
> Fix For: 2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-4278) Hadoop: investigate why small array copying is slow on PPC.

[Vladimir Ozerov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov updated IGNITE-4278:

Issue Type: Task  (was: Sub-task)
Parent: (was: IGNITE-4262)

> Hadoop: investigate why small array copying is slow on PPC.
> ---
>
> Key: IGNITE-4278
> URL: https://issues.apache.org/jira/browse/IGNITE-4278
> Project: Ignite
>  Issue Type: Task
>  Components: hadoop
>Affects Versions: 1.8
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
> Fix For: 2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (IGNITE-4274) Hadoop: control shuffle message buffer size through property.

[Vladimir Ozerov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov resolved IGNITE-4274.
-

> Hadoop: control shuffle message buffer size through property.
> -
>
> Key: IGNITE-4274
> URL: https://issues.apache.org/jira/browse/IGNITE-4274
> Project: Ignite
>  Issue Type: Sub-task
>  Components: hadoop
>Affects Versions: 1.8
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
> Fix For: 2.0
>
>
> Currently it is hard-coded to 128Kb.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (IGNITE-4355) Hadoop: eliminate map threads pauses during startup

[Vladimir Ozerov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov closed IGNITE-4355.
---

> Hadoop: eliminate map threads pauses during startup
> ---
>
> Key: IGNITE-4355
> URL: https://issues.apache.org/jira/browse/IGNITE-4355
> Project: Ignite
>  Issue Type: Sub-task
>  Components: hadoop
>Reporter: Ivan Veselovsky
>Assignee: Vladimir Ozerov
> Fix For: 2.0
>
>
> Pauses in all Map threads but one are observed in the beginning . This is 
> caused by waiting on future.get()  in 
> HadoopV2Job.getTaskContext(HadoopTaskInfo) .
> {code}
>  at sun.misc.Unsafe.park(boolean, long)
>  at java.util.concurrent.locks.LockSupport.park(Object)
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int)
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int)
>  at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(boolean)
>  at org.apache.ignite.internal.util.future.GridFutureAdapter.get()
>  at 
> org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2Job.getTaskContext(HadoopTaskInfo)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffleJob.(Object,
>  IgniteLogger, HadoopJob, GridUnsafeMemory, int, int[], int)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffle.newJob(HadoopJobId)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffle.job(HadoopJobId)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffle.output(HadoopTaskContext)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopEmbeddedTaskExecutor$1.createOutput(HadoopTaskContext)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.createOutputInternal(HadoopTaskContext)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopPerformanceCounter)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call0()
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask$1.call()
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask$1.call()
>  at 
> org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2TaskContext.runAsJobOwner(Callable)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call()
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call()
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body()
>  at org.apache.ignite.internal.util.worker.GridWorker.run()
>  at java.lang.Thread.run()
> {code}
> while the working thread initializes the context:
> {code}
> Java Monitor Wait
>  at java.lang.Object.wait(long)
>  at java.lang.Thread.join(long)
>  at java.lang.Thread.join()
>  at org.apache.hadoop.util.Shell.joinThread(Thread)
>  at org.apache.hadoop.util.Shell.runCommand()
>  at org.apache.hadoop.util.Shell.run()
>  at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute()
>  at org.apache.hadoop.util.Shell.isSetsidSupported()
>  at org.apache.hadoop.util.Shell.()
>  at org.apache.hadoop.util.StringUtils.()
>  at 
> org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(Configuration)
>  at org.apache.hadoop.security.UserGroupInformation.initialize(Configuration, 
> boolean)
>  at org.apache.hadoop.security.UserGroupInformation.ensureInitialized()
>  at 
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(Subject)
>  at org.apache.hadoop.security.UserGroupInformation.getLoginUser()
>  at org.apache.hadoop.security.UserGroupInformation.getCurrentUser()
>  at org.apache.hadoop.mapreduce.task.JobContextImpl.(Configuration, 
> JobID)
>  at org.apache.hadoop.mapred.JobContextImpl.(JobConf, JobID, 
> Progressable)
>  at org.apache.hadoop.mapred.JobContextImpl.(JobConf, JobID)
>  at 
> org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2TaskContext.(HadoopTaskInfo,
>  HadoopJob, HadoopJobId, UUID, DataInput)
>  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Constructor, 
> Object[])
>  at sun.reflect.NativeConstructorAccessorImpl.newInstance(Object[])
>  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Object[])
>  at java.lang.reflect.Constructor.newInstance(Object[])
>  at 
> org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2Job.getTaskContext(HadoopTaskInfo)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffleJob.(Object,
>  IgniteLogger, HadoopJob, GridUnsafeMemory, int, int[], int)
>  at 
> 

[jira] [Closed] (IGNITE-4271) Hadoop messages must use "direct marshallable" infrastructure.

[Vladimir Ozerov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov closed IGNITE-4271.
---

> Hadoop messages must use "direct marshallable" infrastructure.
> --
>
> Key: IGNITE-4271
> URL: https://issues.apache.org/jira/browse/IGNITE-4271
> Project: Ignite
>  Issue Type: Sub-task
>  Components: hadoop
>Affects Versions: 1.7, 1.8
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
> Fix For: 2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4301) Hadoop: send only one shuffle ack at the very end of shuffle processing.

[Vladimir Ozerov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722325#comment-15722325
 ] 

Vladimir Ozerov commented on IGNITE-4301:
-

Design:

1) Send shuffle messages as usual.
2) When shuffle is finished on particular node ("last mapper" situation), send 
single "shuffle finished" message.
3) Remote node accepts this message and sends response when all shuffle tasks 
for original node is finished.

This way we will always send only 1 shuffle ack for N shuffle messages, as 
opposed to N acks now. Moreover, we will not have to store shuffle messages 
in-memory.

> Hadoop: send only one shuffle ack at the very end of shuffle processing.
> 
>
> Key: IGNITE-4301
> URL: https://issues.apache.org/jira/browse/IGNITE-4301
> Project: Ignite
>  Issue Type: Sub-task
>  Components: general, hadoop
>Affects Versions: 1.7, 1.8
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
> Fix For: 2.0
>
>
> This shuffle acks will be processed ASAP, hence minimizing wait during 
> completion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (IGNITE-4276) Hadoop: control shuffle jobs "sleep" backpressure with property.

[Vladimir Ozerov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov closed IGNITE-4276.
---

> Hadoop: control shuffle jobs "sleep" backpressure with property.
> 
>
> Key: IGNITE-4276
> URL: https://issues.apache.org/jira/browse/IGNITE-4276
> Project: Ignite
>  Issue Type: Sub-task
>  Components: hadoop
>Affects Versions: 1.8
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
> Fix For: 2.0
>
>
> Currently we simply sleep between shuffle job iterations for 5 milliseconds 
> (hard-coded). Let's get more control around it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-4301) Hadoop: send only one shuffle ack at the very end of shuffle processing.

[Vladimir Ozerov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov updated IGNITE-4301:

Summary: Hadoop: send only one shuffle ack at the very end of shuffle 
processing.  (was: Hadoop: put shuffle ack to the top of send queue.)

> Hadoop: send only one shuffle ack at the very end of shuffle processing.
> 
>
> Key: IGNITE-4301
> URL: https://issues.apache.org/jira/browse/IGNITE-4301
> Project: Ignite
>  Issue Type: Sub-task
>  Components: general, hadoop
>Affects Versions: 1.7, 1.8
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
> Fix For: 2.0
>
>
> This shuffle acks will be processed ASAP, hence minimizing wait during 
> completion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (IGNITE-4283) Hadoop: implement "striped" shuffle mode.

[Vladimir Ozerov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov closed IGNITE-4283.
---

> Hadoop: implement "striped" shuffle mode.
> -
>
> Key: IGNITE-4283
> URL: https://issues.apache.org/jira/browse/IGNITE-4283
> Project: Ignite
>  Issue Type: Task
>  Components: hadoop
>Affects Versions: 1.7, 1.8
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
> Fix For: 2.0
>
>
> No need for this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (IGNITE-4283) Hadoop: implement "striped" shuffle mode.

[Vladimir Ozerov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov resolved IGNITE-4283.
-
Resolution: Duplicate

> Hadoop: implement "striped" shuffle mode.
> -
>
> Key: IGNITE-4283
> URL: https://issues.apache.org/jira/browse/IGNITE-4283
> Project: Ignite
>  Issue Type: Task
>  Components: hadoop
>Affects Versions: 1.7, 1.8
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
> Fix For: 2.0
>
>
> No need for this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-4270) Hadoop: optionally stripe mapper output for every partition.

[Vladimir Ozerov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov updated IGNITE-4270:

Description: 
Currently we have R maps for M mappers, where R is number of reducers. For this 
reason many mappers writes to concurrent offheap data structure, loosing time 
on concurrency burden.
Let's add an option to create R * M maps, so that every mapper has dedicated 
map for every reducer. This will eliminate almost all concurrency overhead.

Design:
1) Every mapper works with it's own set of "remote" output maps;
2) These maps are essentially not "maps", but IO messages, which we fill up to 
certain threshold;
3) Once filled, message is sent to remote node.
4) Async shuffle thread is no longer need in this architecture.

As a result we decrease concurrency, removes slowdown from a single shuffle 
thread which is not able to send messages fast enough, and removes unnecessary 
intermediate sorting.

NB! Be careful with "combiner" case and with "external" execution.

  was:
Currently we have R maps for M mappers, where R is number of reducers. For this 
reason many mappers writes to concurrent offheap data structure, loosing time 
on concurrency burden.

Let's add an option to create R * M maps, so that every mapper has dedicated 
map for every reducer. This will eliminate almost all concurrency overhead.


> Hadoop: optionally stripe mapper output for every partition.
> 
>
> Key: IGNITE-4270
> URL: https://issues.apache.org/jira/browse/IGNITE-4270
> Project: Ignite
>  Issue Type: Sub-task
>  Components: hadoop
>Affects Versions: 1.8
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
> Fix For: 2.0
>
>
> Currently we have R maps for M mappers, where R is number of reducers. For 
> this reason many mappers writes to concurrent offheap data structure, loosing 
> time on concurrency burden.
> Let's add an option to create R * M maps, so that every mapper has dedicated 
> map for every reducer. This will eliminate almost all concurrency overhead.
> Design:
> 1) Every mapper works with it's own set of "remote" output maps;
> 2) These maps are essentially not "maps", but IO messages, which we fill up 
> to certain threshold;
> 3) Once filled, message is sent to remote node.
> 4) Async shuffle thread is no longer need in this architecture.
> As a result we decrease concurrency, removes slowdown from a single shuffle 
> thread which is not able to send messages fast enough, and removes 
> unnecessary intermediate sorting.
> NB! Be careful with "combiner" case and with "external" execution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-4283) Hadoop: implement "striped" shuffle mode.

[Vladimir Ozerov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov updated IGNITE-4283:

Issue Type: Task  (was: Sub-task)
Parent: (was: IGNITE-4262)

> Hadoop: implement "striped" shuffle mode.
> -
>
> Key: IGNITE-4283
> URL: https://issues.apache.org/jira/browse/IGNITE-4283
> Project: Ignite
>  Issue Type: Task
>  Components: hadoop
>Affects Versions: 1.7, 1.8
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
> Fix For: 2.0
>
>
> No need for this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-4300) Hadoop: get rid of shuffle job.

[Vladimir Ozerov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov updated IGNITE-4300:

Issue Type: Task  (was: Sub-task)
Parent: (was: IGNITE-4262)

> Hadoop: get rid of shuffle job.
> ---
>
> Key: IGNITE-4300
> URL: https://issues.apache.org/jira/browse/IGNITE-4300
> Project: Ignite
>  Issue Type: Task
>  Components: hadoop
>Affects Versions: 1.8
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
> Fix For: 2.0
>
>
> 1) When mapper detects that 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (IGNITE-4300) Hadoop: get rid of shuffle job.

[Vladimir Ozerov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov closed IGNITE-4300.
---

> Hadoop: get rid of shuffle job.
> ---
>
> Key: IGNITE-4300
> URL: https://issues.apache.org/jira/browse/IGNITE-4300
> Project: Ignite
>  Issue Type: Sub-task
>  Components: hadoop
>Affects Versions: 1.8
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
> Fix For: 2.0
>
>
> 1) When mapper detects that 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-4283) Hadoop: implement "striped" shuffle mode.

[Vladimir Ozerov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov updated IGNITE-4283:

Summary: Hadoop: implement "striped" shuffle mode.  (was: Hadoop: do not 
sort mapper output for remote reducers.)

> Hadoop: implement "striped" shuffle mode.
> -
>
> Key: IGNITE-4283
> URL: https://issues.apache.org/jira/browse/IGNITE-4283
> Project: Ignite
>  Issue Type: Sub-task
>  Components: hadoop
>Affects Versions: 1.7, 1.8
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
> Fix For: 2.0
>
>
> No need for this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4281) Hadoop: decouple mapper and reducer maps.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722294#comment-15722294
 ] 

ASF GitHub Bot commented on IGNITE-4281:


GitHub user devozerov opened a pull request:

https://github.com/apache/ignite/pull/1315

IGNITE-4281

…her optimizations.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-4281

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1315.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1315


commit d23f802f7f2d95e1a0dcd5b0f3c557a616fde1d0
Author: devozerov 
Date:   2016-12-05T13:40:48Z

IGNITE-4281: Hadoop: decoupled remote and local maps to simplify further 
optimizations.




> Hadoop: decouple mapper and reducer maps.
> -
>
> Key: IGNITE-4281
> URL: https://issues.apache.org/jira/browse/IGNITE-4281
> Project: Ignite
>  Issue Type: Sub-task
>  Components: hadoop
>Affects Versions: 1.7, 1.8
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
> Fix For: 2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4369) ReentranLocks hands on unlock when node is stopped.

[Taras Ledkov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722276#comment-15722276
 ] 

Taras Ledkov commented on IGNITE-4369:
--

The test to reproduce is available at the [pull 
request|https://github.com/apache/ignite/pull/1314].

> ReentranLocks hands on unlock when node is stopped.
> ---
>
> Key: IGNITE-4369
> URL: https://issues.apache.org/jira/browse/IGNITE-4369
> Project: Ignite
>  Issue Type: Bug
>  Components: cache
>Affects Versions: 1.7
>Reporter: Taras Ledkov
> Fix For: 1.9
>
>
> The thread hangs on GridCacheLockImpl.unlock when other node is stopped.
> {code}
> at java.lang.Thread.yield(Native Method)
>   at 
> org.apache.ignite.internal.processors.datastructures.GridCacheLockImpl$Sync.tryRelease(GridCacheLockImpl.java:469)
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1260)
>   at 
> org.apache.ignite.internal.processors.datastructures.GridCacheLockImpl.unlock(GridCacheLockImpl.java:1296)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4369) ReentranLocks hands on unlock when node is stopped.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722273#comment-15722273
 ] 

ASF GitHub Bot commented on IGNITE-4369:


GitHub user tledkov-gridgain opened a pull request:

https://github.com/apache/ignite/pull/1314

IGNITE-4369  ReentranLocks hands on unlock when node is stopped.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-4369

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1314.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1314


commit acf20b32d8fb68e42b904b091fb2b943f4558cef
Author: sboikov 
Date:   2016-12-05T11:01:28Z

ignite-4296 Optimize GridDhtPartitionsSingleMessage processing
- optimized processing of GridDhtPartitionsSingleMessage
- minor optimizations for RendezvousAffinityFunction
 - fixed heartbeats sending in tcp discovery

commit 3ddceb1e8b46eb09c1b717471772ce37043f6949
Author: tledkov-gridgain 
Date:   2016-12-05T13:31:34Z

IGNITE-4369: add test to reproduce




> ReentranLocks hands on unlock when node is stopped.
> ---
>
> Key: IGNITE-4369
> URL: https://issues.apache.org/jira/browse/IGNITE-4369
> Project: Ignite
>  Issue Type: Bug
>  Components: cache
>Affects Versions: 1.7
>Reporter: Taras Ledkov
> Fix For: 1.9
>
>
> The thread hangs on GridCacheLockImpl.unlock when other node is stopped.
> {code}
> at java.lang.Thread.yield(Native Method)
>   at 
> org.apache.ignite.internal.processors.datastructures.GridCacheLockImpl$Sync.tryRelease(GridCacheLockImpl.java:469)
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1260)
>   at 
> org.apache.ignite.internal.processors.datastructures.GridCacheLockImpl.unlock(GridCacheLockImpl.java:1296)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4355) Hadoop: eliminate map threads pauses during startup

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722261#comment-15722261
 ] 

ASF GitHub Bot commented on IGNITE-4355:


Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/1310


> Hadoop: eliminate map threads pauses during startup
> ---
>
> Key: IGNITE-4355
> URL: https://issues.apache.org/jira/browse/IGNITE-4355
> Project: Ignite
>  Issue Type: Sub-task
>  Components: hadoop
>Reporter: Ivan Veselovsky
>Assignee: Vladimir Ozerov
> Fix For: 2.0
>
>
> Pauses in all Map threads but one are observed in the beginning . This is 
> caused by waiting on future.get()  in 
> HadoopV2Job.getTaskContext(HadoopTaskInfo) .
> {code}
>  at sun.misc.Unsafe.park(boolean, long)
>  at java.util.concurrent.locks.LockSupport.park(Object)
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int)
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int)
>  at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(boolean)
>  at org.apache.ignite.internal.util.future.GridFutureAdapter.get()
>  at 
> org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2Job.getTaskContext(HadoopTaskInfo)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffleJob.(Object,
>  IgniteLogger, HadoopJob, GridUnsafeMemory, int, int[], int)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffle.newJob(HadoopJobId)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffle.job(HadoopJobId)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffle.output(HadoopTaskContext)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopEmbeddedTaskExecutor$1.createOutput(HadoopTaskContext)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.createOutputInternal(HadoopTaskContext)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopPerformanceCounter)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call0()
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask$1.call()
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask$1.call()
>  at 
> org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2TaskContext.runAsJobOwner(Callable)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call()
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call()
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body()
>  at org.apache.ignite.internal.util.worker.GridWorker.run()
>  at java.lang.Thread.run()
> {code}
> while the working thread initializes the context:
> {code}
> Java Monitor Wait
>  at java.lang.Object.wait(long)
>  at java.lang.Thread.join(long)
>  at java.lang.Thread.join()
>  at org.apache.hadoop.util.Shell.joinThread(Thread)
>  at org.apache.hadoop.util.Shell.runCommand()
>  at org.apache.hadoop.util.Shell.run()
>  at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute()
>  at org.apache.hadoop.util.Shell.isSetsidSupported()
>  at org.apache.hadoop.util.Shell.()
>  at org.apache.hadoop.util.StringUtils.()
>  at 
> org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(Configuration)
>  at org.apache.hadoop.security.UserGroupInformation.initialize(Configuration, 
> boolean)
>  at org.apache.hadoop.security.UserGroupInformation.ensureInitialized()
>  at 
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(Subject)
>  at org.apache.hadoop.security.UserGroupInformation.getLoginUser()
>  at org.apache.hadoop.security.UserGroupInformation.getCurrentUser()
>  at org.apache.hadoop.mapreduce.task.JobContextImpl.(Configuration, 
> JobID)
>  at org.apache.hadoop.mapred.JobContextImpl.(JobConf, JobID, 
> Progressable)
>  at org.apache.hadoop.mapred.JobContextImpl.(JobConf, JobID)
>  at 
> org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2TaskContext.(HadoopTaskInfo,
>  HadoopJob, HadoopJobId, UUID, DataInput)
>  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Constructor, 
> Object[])
>  at sun.reflect.NativeConstructorAccessorImpl.newInstance(Object[])
>  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Object[])
>  at java.lang.reflect.Constructor.newInstance(Object[])
>  at 
> org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2Job.getTaskContext(HadoopTaskInfo)
>  at 
> 

[jira] [Commented] (IGNITE-4355) Hadoop: eliminate map threads pauses during startup

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722260#comment-15722260
 ] 

ASF GitHub Bot commented on IGNITE-4355:


Github user devozerov closed the pull request at:

https://github.com/apache/ignite/pull/1313


> Hadoop: eliminate map threads pauses during startup
> ---
>
> Key: IGNITE-4355
> URL: https://issues.apache.org/jira/browse/IGNITE-4355
> Project: Ignite
>  Issue Type: Sub-task
>  Components: hadoop
>Reporter: Ivan Veselovsky
>Assignee: Vladimir Ozerov
> Fix For: 2.0
>
>
> Pauses in all Map threads but one are observed in the beginning . This is 
> caused by waiting on future.get()  in 
> HadoopV2Job.getTaskContext(HadoopTaskInfo) .
> {code}
>  at sun.misc.Unsafe.park(boolean, long)
>  at java.util.concurrent.locks.LockSupport.park(Object)
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int)
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int)
>  at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(boolean)
>  at org.apache.ignite.internal.util.future.GridFutureAdapter.get()
>  at 
> org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2Job.getTaskContext(HadoopTaskInfo)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffleJob.(Object,
>  IgniteLogger, HadoopJob, GridUnsafeMemory, int, int[], int)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffle.newJob(HadoopJobId)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffle.job(HadoopJobId)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffle.output(HadoopTaskContext)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopEmbeddedTaskExecutor$1.createOutput(HadoopTaskContext)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.createOutputInternal(HadoopTaskContext)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopPerformanceCounter)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call0()
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask$1.call()
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask$1.call()
>  at 
> org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2TaskContext.runAsJobOwner(Callable)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call()
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call()
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body()
>  at org.apache.ignite.internal.util.worker.GridWorker.run()
>  at java.lang.Thread.run()
> {code}
> while the working thread initializes the context:
> {code}
> Java Monitor Wait
>  at java.lang.Object.wait(long)
>  at java.lang.Thread.join(long)
>  at java.lang.Thread.join()
>  at org.apache.hadoop.util.Shell.joinThread(Thread)
>  at org.apache.hadoop.util.Shell.runCommand()
>  at org.apache.hadoop.util.Shell.run()
>  at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute()
>  at org.apache.hadoop.util.Shell.isSetsidSupported()
>  at org.apache.hadoop.util.Shell.()
>  at org.apache.hadoop.util.StringUtils.()
>  at 
> org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(Configuration)
>  at org.apache.hadoop.security.UserGroupInformation.initialize(Configuration, 
> boolean)
>  at org.apache.hadoop.security.UserGroupInformation.ensureInitialized()
>  at 
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(Subject)
>  at org.apache.hadoop.security.UserGroupInformation.getLoginUser()
>  at org.apache.hadoop.security.UserGroupInformation.getCurrentUser()
>  at org.apache.hadoop.mapreduce.task.JobContextImpl.(Configuration, 
> JobID)
>  at org.apache.hadoop.mapred.JobContextImpl.(JobConf, JobID, 
> Progressable)
>  at org.apache.hadoop.mapred.JobContextImpl.(JobConf, JobID)
>  at 
> org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2TaskContext.(HadoopTaskInfo,
>  HadoopJob, HadoopJobId, UUID, DataInput)
>  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Constructor, 
> Object[])
>  at sun.reflect.NativeConstructorAccessorImpl.newInstance(Object[])
>  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Object[])
>  at java.lang.reflect.Constructor.newInstance(Object[])
>  at 
> org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2Job.getTaskContext(HadoopTaskInfo)
>  at 
> 

[jira] [Updated] (IGNITE-4368) Data Streamer closed exception in Kafka Streamer

[Anil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anil updated IGNITE-4368:
-
Description: 
There is a possibility of data streamer closed exception in the following case

1. Lets says Kafka streamer is created when application is stared.
2. application node join the ignite cluster
3. Node disconnects from cluster 
4. Node joins the cluster after failure connect time and network timeout time.

Data streamer closed exception happens between #3 and #4. still an issue post 
#4 ?

  was:
There is a possibility of data streamer closed exception in the following case

1. Lets says Kafka streamer is created when application is stared.
2. application node join the ignite cluster
3. Node disconnects from cluster 
4. Node joins the cluster after failure connect time and network timeout time.

Data streamer closed exception happens between #3 and #4


> Data Streamer closed exception in Kafka Streamer
> 
>
> Key: IGNITE-4368
> URL: https://issues.apache.org/jira/browse/IGNITE-4368
> Project: Ignite
>  Issue Type: Bug
>  Components: streaming
>Affects Versions: 1.7
>Reporter: Anil
>
> There is a possibility of data streamer closed exception in the following case
> 1. Lets says Kafka streamer is created when application is stared.
> 2. application node join the ignite cluster
> 3. Node disconnects from cluster 
> 4. Node joins the cluster after failure connect time and network timeout time.
> Data streamer closed exception happens between #3 and #4. still an issue post 
> #4 ?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-4355) Hadoop: eliminate map threads pauses during startup

[Vladimir Ozerov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov updated IGNITE-4355:

Summary: Hadoop: eliminate map threads pauses during startup  (was: Hadoop: 
Eliminate map threads pauses during startup)

> Hadoop: eliminate map threads pauses during startup
> ---
>
> Key: IGNITE-4355
> URL: https://issues.apache.org/jira/browse/IGNITE-4355
> Project: Ignite
>  Issue Type: Sub-task
>  Components: hadoop
>Reporter: Ivan Veselovsky
>Assignee: Vladimir Ozerov
> Fix For: 2.0
>
>
> Pauses in all Map threads but one are observed in the beginning . This is 
> caused by waiting on future.get()  in 
> HadoopV2Job.getTaskContext(HadoopTaskInfo) .
> {code}
>  at sun.misc.Unsafe.park(boolean, long)
>  at java.util.concurrent.locks.LockSupport.park(Object)
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int)
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int)
>  at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(boolean)
>  at org.apache.ignite.internal.util.future.GridFutureAdapter.get()
>  at 
> org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2Job.getTaskContext(HadoopTaskInfo)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffleJob.(Object,
>  IgniteLogger, HadoopJob, GridUnsafeMemory, int, int[], int)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffle.newJob(HadoopJobId)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffle.job(HadoopJobId)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffle.output(HadoopTaskContext)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopEmbeddedTaskExecutor$1.createOutput(HadoopTaskContext)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.createOutputInternal(HadoopTaskContext)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopPerformanceCounter)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call0()
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask$1.call()
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask$1.call()
>  at 
> org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2TaskContext.runAsJobOwner(Callable)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call()
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call()
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body()
>  at org.apache.ignite.internal.util.worker.GridWorker.run()
>  at java.lang.Thread.run()
> {code}
> while the working thread initializes the context:
> {code}
> Java Monitor Wait
>  at java.lang.Object.wait(long)
>  at java.lang.Thread.join(long)
>  at java.lang.Thread.join()
>  at org.apache.hadoop.util.Shell.joinThread(Thread)
>  at org.apache.hadoop.util.Shell.runCommand()
>  at org.apache.hadoop.util.Shell.run()
>  at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute()
>  at org.apache.hadoop.util.Shell.isSetsidSupported()
>  at org.apache.hadoop.util.Shell.()
>  at org.apache.hadoop.util.StringUtils.()
>  at 
> org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(Configuration)
>  at org.apache.hadoop.security.UserGroupInformation.initialize(Configuration, 
> boolean)
>  at org.apache.hadoop.security.UserGroupInformation.ensureInitialized()
>  at 
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(Subject)
>  at org.apache.hadoop.security.UserGroupInformation.getLoginUser()
>  at org.apache.hadoop.security.UserGroupInformation.getCurrentUser()
>  at org.apache.hadoop.mapreduce.task.JobContextImpl.(Configuration, 
> JobID)
>  at org.apache.hadoop.mapred.JobContextImpl.(JobConf, JobID, 
> Progressable)
>  at org.apache.hadoop.mapred.JobContextImpl.(JobConf, JobID)
>  at 
> org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2TaskContext.(HadoopTaskInfo,
>  HadoopJob, HadoopJobId, UUID, DataInput)
>  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Constructor, 
> Object[])
>  at sun.reflect.NativeConstructorAccessorImpl.newInstance(Object[])
>  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Object[])
>  at java.lang.reflect.Constructor.newInstance(Object[])
>  at 
> org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2Job.getTaskContext(HadoopTaskInfo)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffleJob.(Object,
> 

[jira] [Commented] (IGNITE-4355) Hadoop: Eliminate map threads pauses during startup

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1574#comment-1574
 ] 

ASF GitHub Bot commented on IGNITE-4355:


GitHub user devozerov opened a pull request:

https://github.com/apache/ignite/pull/1313

IGNITE-4355



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-4355-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1313.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1313


commit abe6a6b679bf728648c30c30de745b2c3e446f6f
Author: devozerov 
Date:   2016-12-05T13:05:32Z

Done.




> Hadoop: Eliminate map threads pauses during startup
> ---
>
> Key: IGNITE-4355
> URL: https://issues.apache.org/jira/browse/IGNITE-4355
> Project: Ignite
>  Issue Type: Sub-task
>  Components: hadoop
>Reporter: Ivan Veselovsky
>Assignee: Vladimir Ozerov
> Fix For: 2.0
>
>
> Pauses in all Map threads but one are observed in the beginning . This is 
> caused by waiting on future.get()  in 
> HadoopV2Job.getTaskContext(HadoopTaskInfo) .
> {code}
>  at sun.misc.Unsafe.park(boolean, long)
>  at java.util.concurrent.locks.LockSupport.park(Object)
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int)
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int)
>  at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(boolean)
>  at org.apache.ignite.internal.util.future.GridFutureAdapter.get()
>  at 
> org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2Job.getTaskContext(HadoopTaskInfo)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffleJob.(Object,
>  IgniteLogger, HadoopJob, GridUnsafeMemory, int, int[], int)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffle.newJob(HadoopJobId)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffle.job(HadoopJobId)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffle.output(HadoopTaskContext)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopEmbeddedTaskExecutor$1.createOutput(HadoopTaskContext)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.createOutputInternal(HadoopTaskContext)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopPerformanceCounter)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call0()
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask$1.call()
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask$1.call()
>  at 
> org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2TaskContext.runAsJobOwner(Callable)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call()
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call()
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body()
>  at org.apache.ignite.internal.util.worker.GridWorker.run()
>  at java.lang.Thread.run()
> {code}
> while the working thread initializes the context:
> {code}
> Java Monitor Wait
>  at java.lang.Object.wait(long)
>  at java.lang.Thread.join(long)
>  at java.lang.Thread.join()
>  at org.apache.hadoop.util.Shell.joinThread(Thread)
>  at org.apache.hadoop.util.Shell.runCommand()
>  at org.apache.hadoop.util.Shell.run()
>  at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute()
>  at org.apache.hadoop.util.Shell.isSetsidSupported()
>  at org.apache.hadoop.util.Shell.()
>  at org.apache.hadoop.util.StringUtils.()
>  at 
> org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(Configuration)
>  at org.apache.hadoop.security.UserGroupInformation.initialize(Configuration, 
> boolean)
>  at org.apache.hadoop.security.UserGroupInformation.ensureInitialized()
>  at 
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(Subject)
>  at org.apache.hadoop.security.UserGroupInformation.getLoginUser()
>  at org.apache.hadoop.security.UserGroupInformation.getCurrentUser()
>  at org.apache.hadoop.mapreduce.task.JobContextImpl.(Configuration, 
> JobID)
>  at org.apache.hadoop.mapred.JobContextImpl.(JobConf, JobID, 
> Progressable)
>  at org.apache.hadoop.mapred.JobContextImpl.(JobConf, JobID)
>  at 
> 

[jira] [Updated] (IGNITE-4355) Hadoop: Eliminate map threads pauses during startup

[Vladimir Ozerov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov updated IGNITE-4355:

Summary: Hadoop: Eliminate map threads pauses during startup  (was: 
Eliminate map threads pauses during startup)

> Hadoop: Eliminate map threads pauses during startup
> ---
>
> Key: IGNITE-4355
> URL: https://issues.apache.org/jira/browse/IGNITE-4355
> Project: Ignite
>  Issue Type: Sub-task
>  Components: hadoop
>Reporter: Ivan Veselovsky
>Assignee: Vladimir Ozerov
> Fix For: 2.0
>
>
> Pauses in all Map threads but one are observed in the beginning . This is 
> caused by waiting on future.get()  in 
> HadoopV2Job.getTaskContext(HadoopTaskInfo) .
> {code}
>  at sun.misc.Unsafe.park(boolean, long)
>  at java.util.concurrent.locks.LockSupport.park(Object)
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int)
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int)
>  at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(boolean)
>  at org.apache.ignite.internal.util.future.GridFutureAdapter.get()
>  at 
> org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2Job.getTaskContext(HadoopTaskInfo)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffleJob.(Object,
>  IgniteLogger, HadoopJob, GridUnsafeMemory, int, int[], int)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffle.newJob(HadoopJobId)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffle.job(HadoopJobId)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffle.output(HadoopTaskContext)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopEmbeddedTaskExecutor$1.createOutput(HadoopTaskContext)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.createOutputInternal(HadoopTaskContext)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopPerformanceCounter)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call0()
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask$1.call()
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask$1.call()
>  at 
> org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2TaskContext.runAsJobOwner(Callable)
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call()
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call()
>  at 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body()
>  at org.apache.ignite.internal.util.worker.GridWorker.run()
>  at java.lang.Thread.run()
> {code}
> while the working thread initializes the context:
> {code}
> Java Monitor Wait
>  at java.lang.Object.wait(long)
>  at java.lang.Thread.join(long)
>  at java.lang.Thread.join()
>  at org.apache.hadoop.util.Shell.joinThread(Thread)
>  at org.apache.hadoop.util.Shell.runCommand()
>  at org.apache.hadoop.util.Shell.run()
>  at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute()
>  at org.apache.hadoop.util.Shell.isSetsidSupported()
>  at org.apache.hadoop.util.Shell.()
>  at org.apache.hadoop.util.StringUtils.()
>  at 
> org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(Configuration)
>  at org.apache.hadoop.security.UserGroupInformation.initialize(Configuration, 
> boolean)
>  at org.apache.hadoop.security.UserGroupInformation.ensureInitialized()
>  at 
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(Subject)
>  at org.apache.hadoop.security.UserGroupInformation.getLoginUser()
>  at org.apache.hadoop.security.UserGroupInformation.getCurrentUser()
>  at org.apache.hadoop.mapreduce.task.JobContextImpl.(Configuration, 
> JobID)
>  at org.apache.hadoop.mapred.JobContextImpl.(JobConf, JobID, 
> Progressable)
>  at org.apache.hadoop.mapred.JobContextImpl.(JobConf, JobID)
>  at 
> org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2TaskContext.(HadoopTaskInfo,
>  HadoopJob, HadoopJobId, UUID, DataInput)
>  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Constructor, 
> Object[])
>  at sun.reflect.NativeConstructorAccessorImpl.newInstance(Object[])
>  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Object[])
>  at java.lang.reflect.Constructor.newInstance(Object[])
>  at 
> org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2Job.getTaskContext(HadoopTaskInfo)
>  at 
> org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffleJob.(Object,
>  

[jira] [Created] (IGNITE-4369) ReentranLocks hands on unlock when node is stopped.

[Taras Ledkov (JIRA)]

Taras Ledkov created IGNITE-4369:


 Summary: ReentranLocks hands on unlock when node is stopped.
 Key: IGNITE-4369
 URL: https://issues.apache.org/jira/browse/IGNITE-4369
 Project: Ignite
  Issue Type: Bug
  Components: cache
Affects Versions: 1.7
Reporter: Taras Ledkov
 Fix For: 1.9


The thread hangs on GridCacheLockImpl.unlock when other node is stopped.
{code}
at java.lang.Thread.yield(Native Method)
at 
org.apache.ignite.internal.processors.datastructures.GridCacheLockImpl$Sync.tryRelease(GridCacheLockImpl.java:469)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1260)
at 
org.apache.ignite.internal.processors.datastructures.GridCacheLockImpl.unlock(GridCacheLockImpl.java:1296)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-4368) Data Streamer closed exception in Kafka Streamer

[Anil (JIRA)]

Anil created IGNITE-4368:


 Summary: Data Streamer closed exception in Kafka Streamer
 Key: IGNITE-4368
 URL: https://issues.apache.org/jira/browse/IGNITE-4368
 Project: Ignite
  Issue Type: Bug
  Components: streaming
Affects Versions: 1.7
Reporter: Anil


There is a possibility of data streamer closed exception in the following case

1. Lets says Kafka streamer is created when application is stared.
2. application node join the ignite cluster
3. Node disconnects from cluster 
4. Node joins the cluster after failure connect time and network timeout time.

Data streamer closed exception happens between #3 and #4



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4271) Hadoop messages must use "direct marshallable" infrastructure.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722193#comment-15722193
 ] 

ASF GitHub Bot commented on IGNITE-4271:


Github user devozerov closed the pull request at:

https://github.com/apache/ignite/pull/1312


> Hadoop messages must use "direct marshallable" infrastructure.
> --
>
> Key: IGNITE-4271
> URL: https://issues.apache.org/jira/browse/IGNITE-4271
> Project: Ignite
>  Issue Type: Sub-task
>  Components: hadoop
>Affects Versions: 1.7, 1.8
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
> Fix For: 2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (IGNITE-4271) Hadoop messages must use "direct marshallable" infrastructure.

[Vladimir Ozerov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov resolved IGNITE-4271.
-
Resolution: Fixed

> Hadoop messages must use "direct marshallable" infrastructure.
> --
>
> Key: IGNITE-4271
> URL: https://issues.apache.org/jira/browse/IGNITE-4271
> Project: Ignite
>  Issue Type: Sub-task
>  Components: hadoop
>Affects Versions: 1.7, 1.8
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
> Fix For: 2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4295) GridUnsafe: implement specialized methods for every kind of copy operation.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722189#comment-15722189
 ] 

ASF GitHub Bot commented on IGNITE-4295:


Github user devozerov closed the pull request at:

https://github.com/apache/ignite/pull/1282


> GridUnsafe: implement specialized methods for every kind of copy operation.
> ---
>
> Key: IGNITE-4295
> URL: https://issues.apache.org/jira/browse/IGNITE-4295
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 1.8
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
>Priority: Minor
>  Labels: performance
> Fix For: 2.0
>
>
> 1) copy(OFFHEAP, HEAP)
> 2) copy(HEAP, OFFHEAP)
> 3) copy(OFFHEAP, OFFHEAP)
> 4) copy(HEAP, HEAP) 
> 5) copy(T[], T[]) must be avoided and replaced with System.arrayCopy().
> 6) Add optional threshold. If we copy too small memory chunk which size is 
> below the threshold, then resort to byte-by-byte copying, as it will be 
> faster.
> E.g. on PowerPC with OpenJDK 8, copying of <100 bytes of data is faster on 
> byte-by-byte basis.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4308) Make QueryEntity.keyFields optional for caches having SQL types as keys

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722157#comment-15722157
 ] 

ASF GitHub Bot commented on IGNITE-4308:


Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/1286


> Make QueryEntity.keyFields optional for caches having SQL types as keys
> ---
>
> Key: IGNITE-4308
> URL: https://issues.apache.org/jira/browse/IGNITE-4308
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.8
>Reporter: Alexander Paschenko
>Assignee: Vladimir Ozerov
> Fix For: 1.8
>
>
> Initial implementation of DML requires the user to specify keyFields in 
> configuration explicitly, even if it's empty, in cases when DML is used and 
> binary marshaller is used. As [~vozerov] noted, it could affect usability, if 
> even a little, so this should be fixed - when a primitive/simple SQL type is 
> used as a key, it's obvious that there's no key fields and hence it's not 
> necessary to specify empty keyFields in this case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4340) Implicitly cast new column values to expected types on SQL UPDATE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722158#comment-15722158
 ] 

ASF GitHub Bot commented on IGNITE-4340:


Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/1303


> Implicitly cast new column values to expected types on SQL UPDATE
> -
>
> Key: IGNITE-4340
> URL: https://issues.apache.org/jira/browse/IGNITE-4340
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.8
>Reporter: Alexander Paschenko
>Assignee: Vladimir Ozerov
> Fix For: 1.8
>
>
> When the following query is run,
> {code:sql}
> update AllTypes set longCol = 1 where _key = ?
> {code}
> it fails with exception
> {noformat}
> Suppressed: java.lang.ClassCastException: java.lang.Integer cannot be cast to 
> java.lang.Long
>   at 
> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$RowDescriptor.wrap(IgniteH2Indexing.java:2960)
>   at 
> org.apache.ignite.internal.processors.query.h2.opt.GridH2AbstractKeyValueRow.getValue(GridH2AbstractKeyValueRow.java:316)
>   at 
> org.h2.index.BaseIndex.compareRows(BaseIndex.java:294)
>   at 
> org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex$2.compare(GridH2TreeIndex.java:103)
>   at 
> org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex$2.compare(GridH2TreeIndex.java:95)
>   at 
> java.util.concurrent.ConcurrentSkipListMap$ComparableUsingComparator.compareTo(ConcurrentSkipListMap.java:647)
>   at 
> java.util.concurrent.ConcurrentSkipListMap.findPredecessor(ConcurrentSkipListMap.java:727)
>   at 
> java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMap.java:850)
>   at 
> java.util.concurrent.ConcurrentSkipListMap.put(ConcurrentSkipListMap.java:1645)
>   at 
> org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex.put(GridH2TreeIndex.java:362)
>   at 
> org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.doUpdate(GridH2Table.java:566)
>   at 
> org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:495)
>   at 
> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:603)
>   at 
> org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:737)
>   at 
> org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:431)
>   at 
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.updateIndex(GridCacheMapEntry.java:4019)
>   at 
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2458)
>   at 
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2385)
>   at 
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1787)
> {noformat}
> It's due to that UPDATE's SELECT part selects 1 as int, and that's what we're 
> setting to field. Problem can be solved by casting SELECTed values to the 
> types that columns expect.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4096) ODBC and DML: Add tests with DML and ODBC.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722166#comment-15722166
 ] 

ASF GitHub Bot commented on IGNITE-4096:


Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/1178


> ODBC and DML: Add tests with DML and ODBC.
> --
>
> Key: IGNITE-4096
> URL: https://issues.apache.org/jira/browse/IGNITE-4096
> Project: Ignite
>  Issue Type: Task
>  Components: odbc
>Affects Versions: 1.7
>Reporter: Igor Sapego
>Assignee: Igor Sapego
>  Labels: odbc
> Fix For: 1.8
>
>
> In Apache Ignite 1.8 the community is planning to release DML support.
> We need to make sure that ODBC is ready for that. Thus we need to add ODBC 
> test that involve DML.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4287) DML: DELETE statement failed: Invalid number of query parameters. Cannot find 2 parameter.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722162#comment-15722162
 ] 

ASF GitHub Bot commented on IGNITE-4287:


Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/1280


> DML: DELETE statement failed: Invalid number of query parameters. Cannot find 
> 2 parameter.
> --
>
> Key: IGNITE-4287
> URL: https://issues.apache.org/jira/browse/IGNITE-4287
> Project: Ignite
>  Issue Type: Bug
>  Components: general
>Affects Versions: 1.8
>Reporter: Sergey Kozlov
>Assignee: Sergey Kozlov
> Fix For: 1.8
>
>
> {code:title=test.java|borderStyle=solid}
>   IgniteCache orgCache = 
> Ignition.ignite().cache(ORG_CACHE);
> orgCache.clear();
> for (int z=100; z < 10; z++) {
> orgCache.query(new SqlFieldsQuery("insert into Organization 
> (_key, name) values (?, ?)").setArgs(z, "Org " + Integer.toString(z)));
> }
> for (int z=100; z < 10; z++) {
> if (z > 0 && z % 10 == 0)
> System.out.println("Delete " + z);
> orgCache.query(new SqlFieldsQuery("delete from Organization where 
> _key >= ?").setArgs(z));
> }
> {code}
> The code above failed with IgniteException
> {noformat}
> Exception in thread "main" javax.cache.CacheException: class 
> org.apache.ignite.IgniteException: Invalid number of query parameters. Cannot 
> find 2 parameter.
>   at 
> org.apache.ignite.internal.processors.cache.IgniteCacheProxy.query(IgniteCacheProxy.java:761)
>   at 
> org.apache.ignite.examples.datagrid.CacheQueryExample.initialize(CacheQueryExample.java:336)
>   at 
> org.apache.ignite.examples.datagrid.CacheQueryExample.main(CacheQueryExample.java:110)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
> Caused by: class org.apache.ignite.IgniteException: Invalid number of query 
> parameters. Cannot find 2 parameter.
>   at 
> org.apache.ignite.internal.processors.query.GridQueryProcessor.queryTwoStep(GridQueryProcessor.java:818)
>   at 
> org.apache.ignite.internal.processors.cache.IgniteCacheProxy.query(IgniteCacheProxy.java:749)
>   ... 7 more
> Caused by: class org.apache.ignite.IgniteCheckedException: Invalid number of 
> query parameters. Cannot find 2 parameter.
>   at 
> org.apache.ignite.internal.processors.query.GridQueryProcessor.executeQuery(GridQueryProcessor.java:1789)
>   at 
> org.apache.ignite.internal.processors.query.GridQueryProcessor.queryTwoStep(GridQueryProcessor.java:811)
>   ... 8 more
> Caused by: class org.apache.ignite.IgniteException: Invalid number of query 
> parameters. Cannot find 2 parameter.
>   at 
> org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.findParams(GridSqlQuerySplitter.java:634)
>   at 
> org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.findParams(GridSqlQuerySplitter.java:650)
>   at 
> org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.findParams(GridSqlQuerySplitter.java:650)
>   at 
> org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.findParams(GridSqlQuerySplitter.java:604)
>   at 
> org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.split(GridSqlQuerySplitter.java:403)
>   at 
> org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.split(GridSqlQuerySplitter.java:184)
>   at 
> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.queryTwoStep(IgniteH2Indexing.java:1277)
>   at 
> org.apache.ignite.internal.processors.query.h2.DmlStatementsProcessor.executeUpdateStatement(DmlStatementsProcessor.java:226)
>   at 
> org.apache.ignite.internal.processors.query.h2.DmlStatementsProcessor.updateSqlFields(DmlStatementsProcessor.java:134)
>   at 
> org.apache.ignite.internal.processors.query.h2.DmlStatementsProcessor.updateSqlFieldsTwoStep(DmlStatementsProcessor.java:160)
>   at 
> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.queryTwoStep(IgniteH2Indexing.java:1266)
>   at 
> org.apache.ignite.internal.processors.query.GridQueryProcessor$4.applyx(GridQueryProcessor.java:813)
>   at 
> org.apache.ignite.internal.processors.query.GridQueryProcessor$4.applyx(GridQueryProcessor.java:811)
>   at 
> 

[jira] [Commented] (IGNITE-4319) Fix IgniteCacheAbstractSqlDmlQuerySelfTest

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722159#comment-15722159
 ] 

ASF GitHub Bot commented on IGNITE-4319:


Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/1291


> Fix IgniteCacheAbstractSqlDmlQuerySelfTest
> --
>
> Key: IGNITE-4319
> URL: https://issues.apache.org/jira/browse/IGNITE-4319
> Project: Ignite
>  Issue Type: Test
>  Components: SQL
>Affects Versions: 1.8
>Reporter: Alexander Paschenko
>Assignee: Vladimir Ozerov
> Fix For: 1.8
>
>
> Somehow it's green on TC in non binary mode but fails locally. Purely test 
> problem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4343) Check mutable entries for existence properly inside DML entry processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722168#comment-15722168
 ] 

ASF GitHub Bot commented on IGNITE-4343:


Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/1304


> Check mutable entries for existence properly inside DML entry processors
> 
>
> Key: IGNITE-4343
> URL: https://issues.apache.org/jira/browse/IGNITE-4343
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.8
>Reporter: Alexander Paschenko
>Assignee: Vladimir Ozerov
> Fix For: 1.8
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4280) Fix failing tests in IgniteBinaryCacheQueryTestSuite

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722155#comment-15722155
 ] 

ASF GitHub Bot commented on IGNITE-4280:


Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/1275


> Fix failing tests in IgniteBinaryCacheQueryTestSuite
> 
>
> Key: IGNITE-4280
> URL: https://issues.apache.org/jira/browse/IGNITE-4280
> Project: Ignite
>  Issue Type: Test
>Reporter: Alexander Paschenko
>Assignee: Vladimir Ozerov
> Fix For: 1.8
>
>
> Details: 
> https://issues.apache.org/jira/browse/IGNITE-2294?focusedCommentId=15683377



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-3910) ODBC: PDO always passes parameters as strings.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-3910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722167#comment-15722167
 ] 

ASF GitHub Bot commented on IGNITE-3910:


Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/1269


> ODBC: PDO always passes parameters as strings.
> --
>
> Key: IGNITE-3910
> URL: https://issues.apache.org/jira/browse/IGNITE-3910
> Project: Ignite
>  Issue Type: Bug
>  Components: odbc
>Affects Versions: 1.7
>Reporter: Igor Sapego
>Assignee: Vladimir Ozerov
>  Labels: odbc
> Fix For: 1.8
>
>
> By some reason, PDO always passes parameters as strings to our ODBC driver. 
> Need to investigate and find a solution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4299) Remove useless cin.get() in query_example.cpp

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722156#comment-15722156
 ] 

ASF GitHub Bot commented on IGNITE-4299:


Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/1277


> Remove useless cin.get() in query_example.cpp
> -
>
> Key: IGNITE-4299
> URL: https://issues.apache.org/jira/browse/IGNITE-4299
> Project: Ignite
>  Issue Type: Bug
>  Components: platforms
>Reporter: Ksenia Rybakova
>Assignee: Igor Sapego
>Priority: Minor
> Fix For: 1.8
>
>
> Remove useless cin.get() in query_example.cpp:
> {noformat}
> std::cout << ">>> Ready" << std::endl;
> std::cout << std::endl;
> std::cin.get();
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4016) ODBC: Enhance ODBC example by using DML statements

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722169#comment-15722169
 ] 

ASF GitHub Bot commented on IGNITE-4016:


Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/1265


> ODBC: Enhance ODBC example by using DML statements
> --
>
> Key: IGNITE-4016
> URL: https://issues.apache.org/jira/browse/IGNITE-4016
> Project: Ignite
>  Issue Type: Task
>  Components: odbc
>Reporter: Denis Magda
>Assignee: Vladimir Ozerov
> Fix For: 1.8
>
>
> In Apache Ignite 1.8 the community is planning to release DML support. 
> To adopt DML usage we need to improve existed or add additional examples.
> On ODBC side we can improve existed odbc_example by implementing the 
> following scenario:
> - fill up a cache using INSERT commands instead of cache.puts().
> - execute existed SELECT statements.
> - perform cache update using UPDATE statements.
> - execute SELECT statements showing that the data has been changed.
> - remove a part of the data from cache using DELETE command.
> - execute SELECTs again.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4288) ODBC: Fix ODBC and DML interoperability.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722164#comment-15722164
 ] 

ASF GitHub Bot commented on IGNITE-4288:


Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/1272


> ODBC: Fix ODBC and DML interoperability.
> 
>
> Key: IGNITE-4288
> URL: https://issues.apache.org/jira/browse/IGNITE-4288
> Project: Ignite
>  Issue Type: Bug
>  Components: odbc
>Affects Versions: 1.7
>Reporter: Igor Sapego
>Assignee: Vladimir Ozerov
> Fix For: 1.8
>
>
> Need to fix ODBC-DML interoperability issues on ignite-1.8 branch.
> Failing tests list:
> * odbc-tests/QueriesTestSuite/TestInsertDeleteSelect  
> * odbc-tests/QueriesTestSuite/TestInsertMergeSelect   
> * odbc-tests/QueriesTestSuite/TestInsertSelect
> * odbc-tests/QueriesTestSuite/TestInsertUpdateSelect 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4347) ODBC: NPE when cache name is different from the one configured in DSN

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722163#comment-15722163
 ] 

ASF GitHub Bot commented on IGNITE-4347:


Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/1305


> ODBC: NPE when cache name is different from the one configured in DSN
> -
>
> Key: IGNITE-4347
> URL: https://issues.apache.org/jira/browse/IGNITE-4347
> Project: Ignite
>  Issue Type: Bug
>  Components: odbc
>Reporter: Denis Magda
>Assignee: Vladimir Ozerov
>Priority: Critical
> Fix For: 1.8
>
> Attachments: exception.png
>
>
> The following query was executed from PHP PDO side
> {code}
> $dbs = $dbh->prepare('INSERT INTO Person (_key, firstName, lastName, resume, 
> salary) 
> VALUES (?, ?, ?, ?, ?)');
> {code}
> The cache name in Spring XML configuration was "Person" while the DSN was 
> configured to use "PersonCache" as a default cache name.
> As a result, I was getting NPE shown in the attached screenshot. Only after I 
> sorted out the root cause of the NPE referring to the source code I could 
> finally fix the issue.
> Let's provide more meaningful explanation rather than throwing NPE saying 
> something like "The cache named {cache_name} has not been found. Make sure 
> that ODBC connection string or DSN is configured properly."



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4279) Make DML related JDBC driver tests pass within suite

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722165#comment-15722165
 ] 

ASF GitHub Bot commented on IGNITE-4279:


Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/1284


> Make DML related JDBC driver tests pass within suite
> 
>
> Key: IGNITE-4279
> URL: https://issues.apache.org/jira/browse/IGNITE-4279
> Project: Ignite
>  Issue Type: Test
>  Components: jdbc-driver, SQL
>Affects Versions: 1.8
>Reporter: Alexander Paschenko
>Assignee: Alexander Paschenko
> Fix For: 1.8
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4017) DML: Add Java Example

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722161#comment-15722161
 ] 

ASF GitHub Bot commented on IGNITE-4017:


Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/1293


> DML: Add Java Example
> -
>
> Key: IGNITE-4017
> URL: https://issues.apache.org/jira/browse/IGNITE-4017
> Project: Ignite
>  Issue Type: Task
>  Components: SQL
>Reporter: Denis Magda
>Assignee: Vladimir Ozerov
> Fix For: 1.8
>
>
> In Apache Ignite 1.8 the community is planning to release DML support.
> To adopt DML usage we need to improve existed or add additional examples.
> We need to add {{CacheDmlExample}} doing the following with DML's *GridGain 
> Java API*:
> - fill up a cache using INSERT commands.
> - execute SELECT statements. There are should be queries with joins.
> - perform cache update using UPDATE and MERGE statements.
> - execute SELECT statements. There are should be queries with joins.
> - remove a part of the data from cache using DELETE command.
> execute SELECTs again.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4320) Minor fixes in DmlStatementsProcessor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722160#comment-15722160
 ] 

ASF GitHub Bot commented on IGNITE-4320:


Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/1292


> Minor fixes in DmlStatementsProcessor
> -
>
> Key: IGNITE-4320
> URL: https://issues.apache.org/jira/browse/IGNITE-4320
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.8
>Reporter: Alexander Paschenko
>Assignee: Vladimir Ozerov
> Fix For: 1.8
>
>
> 1. Important local variable renamed to more sane name
> 2. Removed duplicate operation manipulations in doUpdate
> 3. Small optimization in doUpdate (don't read key fields as we won't use them 
> anyway)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (IGNITE-4274) Hadoop: control shuffle message buffer size through property.

[Vladimir Ozerov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov resolved IGNITE-4274.
-

> Hadoop: control shuffle message buffer size through property.
> -
>
> Key: IGNITE-4274
> URL: https://issues.apache.org/jira/browse/IGNITE-4274
> Project: Ignite
>  Issue Type: Sub-task
>  Components: hadoop
>Affects Versions: 1.8
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
> Fix For: 2.0
>
>
> Currently it is hard-coded to 128Kb.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (IGNITE-4274) Hadoop: control shuffle message buffer size through property.

[Vladimir Ozerov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov closed IGNITE-4274.
---

> Hadoop: control shuffle message buffer size through property.
> -
>
> Key: IGNITE-4274
> URL: https://issues.apache.org/jira/browse/IGNITE-4274
> Project: Ignite
>  Issue Type: Sub-task
>  Components: hadoop
>Affects Versions: 1.8
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
> Fix For: 2.0
>
>
> Currently it is hard-coded to 128Kb.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4271) Hadoop messages must use "direct marshallable" infrastructure.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722125#comment-15722125
 ] 

ASF GitHub Bot commented on IGNITE-4271:


GitHub user devozerov opened a pull request:

https://github.com/apache/ignite/pull/1312

IGNITE-4271



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-4271

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1312.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1312


commit 56e30e895ebbd4466a7e7afd3dcfd2762b9e9b9c
Author: devozerov 
Date:   2016-12-05T12:21:26Z

IGNITE-4271: Done.




> Hadoop messages must use "direct marshallable" infrastructure.
> --
>
> Key: IGNITE-4271
> URL: https://issues.apache.org/jira/browse/IGNITE-4271
> Project: Ignite
>  Issue Type: Sub-task
>  Components: hadoop
>Affects Versions: 1.7, 1.8
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
> Fix For: 2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-4367) .NET: Fix flaky tests

[Pavel Tupitsyn (JIRA)]

Pavel Tupitsyn created IGNITE-4367:
--

 Summary: .NET: Fix flaky tests
 Key: IGNITE-4367
 URL: https://issues.apache.org/jira/browse/IGNITE-4367
 Project: Ignite
  Issue Type: Task
  Components: platforms
Reporter: Pavel Tupitsyn
Assignee: Pavel Tupitsyn
Priority: Trivial
 Fix For: 1.9


TeamCity has detected a number of flaky tests in .NET: 
http://ci.ignite.apache.org/project.html?projectId=IgniteTests=flakyTests



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4026) BinaryObjectBuilder.build() can fail if one of the fields is Externalizable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722078#comment-15722078
 ] 

ASF GitHub Bot commented on IGNITE-4026:


Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/1289


> BinaryObjectBuilder.build() can fail if one of the fields is Externalizable
> ---
>
> Key: IGNITE-4026
> URL: https://issues.apache.org/jira/browse/IGNITE-4026
> Project: Ignite
>  Issue Type: Bug
>  Components: binary
>Affects Versions: 1.7
>Reporter: Valentin Kulichenko
>Assignee: Vladimir Ozerov
>Priority: Critical
> Fix For: 1.8
>
> Attachments: BuilderTest.java, BuilderTest2.java
>
>
> Test reproducing the issue is attached.
> Scenario is the following:
> # Create a binary object with an {{Externalizable}} field.
> # Create a builder from this object using {{toBuilder()}} method.
> # Do some modifications.
> # Call {{build()}}, get exception below.
> {noformat}
> Exception in thread "main" class 
> org.apache.ignite.binary.BinaryObjectException: Invalid flag value: -2
>   at 
> org.apache.ignite.internal.binary.builder.BinaryBuilderReader.parseValue(BinaryBuilderReader.java:761)
>   at 
> org.apache.ignite.internal.binary.builder.BinaryObjectBuilderImpl.serializeTo(BinaryObjectBuilderImpl.java:281)
>   at 
> org.apache.ignite.internal.binary.builder.BinaryObjectBuilderImpl.build(BinaryObjectBuilderImpl.java:183)
>   at BuilderTest.main(BuilderTest.java:16)
> {noformat}
> Similar issue exists with enums. An attempt to set enum field to builder with 
> a value which was read from a binary object fails with this exception:
> {noformat}
> Exception in thread "main" class 
> org.apache.ignite.binary.BinaryObjectException: Wrong value has been set 
> [typeName=MyType, fieldName=enum, fieldType=Enum, assignedValueType=Object]
>   at 
> org.apache.ignite.internal.binary.builder.BinaryObjectBuilderImpl.checkMetadata(BinaryObjectBuilderImpl.java:401)
>   at 
> org.apache.ignite.internal.binary.builder.BinaryObjectBuilderImpl.serializeTo(BinaryObjectBuilderImpl.java:316)
>   at 
> org.apache.ignite.internal.binary.builder.BinaryObjectBuilderImpl.build(BinaryObjectBuilderImpl.java:183)
>   at BuilderTest2.main(BuilderTest2.java:15)
> {noformat}
> The letter is reproduced in {{BinaryTest2.java}} (attached).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-2662) .NET Standard support

[Pavel Tupitsyn (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15721956#comment-15721956
 ] 

Pavel Tupitsyn commented on IGNITE-2662:


Denis, it depends on IGNITE-3568 implementation a lot. Will it be optional? 
Which IPC mechanism will be used?

Ideally, IGNITE-3568 will get rid of C++ dependency completely, which will 
eliminate the main obstacle for this ticket.

As for the users - I don't expect any API changes.

> .NET Standard support
> -
>
> Key: IGNITE-2662
> URL: https://issues.apache.org/jira/browse/IGNITE-2662
> Project: Ignite
>  Issue Type: New Feature
>  Components: platforms
>Affects Versions: 1.1.4
>Reporter: Pavel Tupitsyn
>  Labels: .net
>
> Ignite.NET should target .NET Standard so it is available on maximum number 
> of platforms, see
> https://blogs.msdn.microsoft.com/dotnet/2016/09/26/introducing-net-standard/
> https://weblog.west-wind.com/posts/2016/Nov/23/NET-Standard-20-Making-Sense-of-NET-Again
> This will allow us to run on Windows, OSX, and Linux, and target .NET Core in 
> additional to good old regular .NET.
> Possible difficulties:
> * JNI interop. Core has dllImport and it works on linux, and our C++ client 
> works on linux, so it should be possible
> * Reflection. We use it a lot, and API has changed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (IGNITE-4285) Optimize locks for read-only keys in optimistic/serializable transactions

[Semen Boikov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Semen Boikov resolved IGNITE-4285.
--
Resolution: Fixed

Implemented read locks optimization: for serializable txs lock candidate for 
'read' can be added in pedning queue without version check if before it there 
are only others 'read' lock candidates. Locks re-assignment marks as owners all 
ready 'read' candidates in the queue head.

> Optimize locks for read-only keys in optimistic/serializable transactions
> -
>
> Key: IGNITE-4285
> URL: https://issues.apache.org/jira/browse/IGNITE-4285
> Project: Ignite
>  Issue Type: Task
>  Components: cache
>Reporter: Semen Boikov
>Assignee: Semen Boikov
> Fix For: 2,0
>
>
> Currently, when optimistic transaction acquires lock for key which is not 
> modified by any other tx, TransactionOptimisticException is still possible if 
> read lock has already been acquired by any other tx with higher tx version.
> Following optimization is possible to avoid TransactionOptimisticException 
> for mostly read keys: when tx tries to add lock for 'read' key, and there are 
> only 'read' locks in entry's queue, then lock can be acquired immediately 
> without tx version check. Standard version comparison should still take place 
> in case any writer is in queue which may result in optimistic exception.
> Having this implemented:
> # Ignite will not throw optimistic exception for read only keys
> # Ignite will be able to commit multiple transactions in parallel if these 
> transactions intersect only by their "reading" sets.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-4366) deadlock in continuous queries

[Yakov Zhdanov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yakov Zhdanov updated IGNITE-4366:
--
Affects Version/s: 1.7

> deadlock in continuous queries
> --
>
> Key: IGNITE-4366
> URL: https://issues.apache.org/jira/browse/IGNITE-4366
> Project: Ignite
>  Issue Type: Bug
>  Components: cache
>Affects Versions: 1.7
>Reporter: Yakov Zhdanov
> Attachments: threaddump-ignitequeries.txt
>
>
> see attached threaddump
> there is definitely a deadlock caused by 
> exchange-worker-#14935%continuous.GridCacheContinuousQueryConcurrentTest0%
> and 
> sys-stripe-7-#14847%continuous.GridCacheContinuousQueryConcurrentTest0%



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-4366) deadlock in continuous queries

[Yakov Zhdanov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yakov Zhdanov updated IGNITE-4366:
--
Attachment: threaddump-ignitequeries.txt

> deadlock in continuous queries
> --
>
> Key: IGNITE-4366
> URL: https://issues.apache.org/jira/browse/IGNITE-4366
> Project: Ignite
>  Issue Type: Bug
>  Components: cache
>Affects Versions: 1.7
>Reporter: Yakov Zhdanov
> Attachments: threaddump-ignitequeries.txt
>
>
> see attached threaddump
> there is definitely a deadlock caused by 
> exchange-worker-#14935%continuous.GridCacheContinuousQueryConcurrentTest0%
> and 
> sys-stripe-7-#14847%continuous.GridCacheContinuousQueryConcurrentTest0%



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-4366) deadlock in continuous queries

[Yakov Zhdanov (JIRA)]

Yakov Zhdanov created IGNITE-4366:
-

 Summary: deadlock in continuous queries
 Key: IGNITE-4366
 URL: https://issues.apache.org/jira/browse/IGNITE-4366
 Project: Ignite
  Issue Type: Bug
  Components: cache
Reporter: Yakov Zhdanov


see attached threaddump
there is definitely a deadlock caused by 

exchange-worker-#14935%continuous.GridCacheContinuousQueryConcurrentTest0%
and 
sys-stripe-7-#14847%continuous.GridCacheContinuousQueryConcurrentTest0%



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4263) Hadoop: abstract out offheap/heap memory management.

[Taras Ledkov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15721716#comment-15721716
 ] 

Taras Ledkov commented on IGNITE-4263:
--

Code 
review:[IGNT-CR-28|http://reviews.ignite.apache.org/ignite/review/IGNT-CR-28]

> Hadoop: abstract out offheap/heap memory management.
> 
>
> Key: IGNITE-4263
> URL: https://issues.apache.org/jira/browse/IGNITE-4263
> Project: Ignite
>  Issue Type: Sub-task
>  Components: hadoop
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Taras Ledkov
> Fix For: 2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-75) Revisit node stop procedure and busy locking

[Yakov Zhdanov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-75?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yakov Zhdanov updated IGNITE-75:

   Assignee: Yakov Zhdanov
Description: 
I think we need to eliminate all the gateways and busy locks in order to reduce 
inter-thread operations and get closer to share nothing pattern.

Here is the original description (not sure if it still makes sense):

# Have 64 or 128 threads preforming operations from public API (e.g. 
cache.put(..))
# Have 1 thread to stop current node. Stop will most likely hang due to busy 
lock on kernal gateway (since it is busy locked by threads performing puts)

Comments:
# Why we have multiple busy locks instead of using one from KernalGateway?
# All ongoing user initiated operations should fail, no new allowed and then 
stop procedure should proceed.
# One can use {{GridCacheStopSelfTest}} to debug this, but create a new test 
which should involve 2 or more nodes.

  was:
# Have 64 or 128 threads preforming operations from public API (e.g. 
cache.put(..))
# Have 1 thread to stop current node. Stop will most likely hang due to busy 
lock on kernal gateway (since it is busy locked by threads performing puts)

Comments:
# Why we have multiple busy locks instead of using one from KernalGateway?
# All ongoing user initiated operations should fail, no new allowed and then 
stop procedure should proceed.
# One can use {{GridCacheStopSelfTest}} to debug this, but create a new test 
which should involve 2 or more nodes.

 Issue Type: Improvement  (was: Bug)

> Revisit node stop procedure and busy locking
> 
>
> Key: IGNITE-75
> URL: https://issues.apache.org/jira/browse/IGNITE-75
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Yakov Zhdanov
>Assignee: Yakov Zhdanov
>
> I think we need to eliminate all the gateways and busy locks in order to 
> reduce inter-thread operations and get closer to share nothing pattern.
> Here is the original description (not sure if it still makes sense):
> # Have 64 or 128 threads preforming operations from public API (e.g. 
> cache.put(..))
> # Have 1 thread to stop current node. Stop will most likely hang due to busy 
> lock on kernal gateway (since it is busy locked by threads performing puts)
> Comments:
> # Why we have multiple busy locks instead of using one from KernalGateway?
> # All ongoing user initiated operations should fail, no new allowed and then 
> stop procedure should proceed.
> # One can use {{GridCacheStopSelfTest}} to debug this, but create a new test 
> which should involve 2 or more nodes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-4365) Data grid in deadlock on stop

[Yakov Zhdanov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yakov Zhdanov updated IGNITE-4365:
--
Attachment: thread-dump.txt

> Data grid in deadlock on stop
> -
>
> Key: IGNITE-4365
> URL: https://issues.apache.org/jira/browse/IGNITE-4365
> Project: Ignite
>  Issue Type: Bug
>  Components: cache
>Reporter: Yakov Zhdanov
>  Labels: busylock, gateway, performance
> Attachments: thread-dump.txt
>
>
> Attached is the threaddump describing the problem.
> # several public threads wait for new cache topology version
> # onExchangeFinish() tries to stop the gateway, but cannot do it due to 
> public threads waiting inside the GW.
> # grid stopping thread waits for job requests to complete



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-4365) Data grid in deadlock on stop

[Yakov Zhdanov (JIRA)]

Yakov Zhdanov created IGNITE-4365:
-

 Summary: Data grid in deadlock on stop
 Key: IGNITE-4365
 URL: https://issues.apache.org/jira/browse/IGNITE-4365
 Project: Ignite
  Issue Type: Bug
  Components: cache
Reporter: Yakov Zhdanov


Attached is the threaddump describing the problem.

# several public threads wait for new cache topology version
# onExchangeFinish() tries to stop the gateway, but cannot do it due to public 
threads waiting inside the GW.
# grid stopping thread waits for job requests to complete



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)