[jira] [Closed] (IGNITE-3175) BigDecimal fields are not supported if query is executed from IgniteRDD

2016-05-22 Thread Semen Boikov (JIRA) 

 [ 
https://issues.apache.org/jira/browse/IGNITE-3175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Semen Boikov closed IGNITE-3175.


> BigDecimal fields are not supported if query is executed from IgniteRDD
> ---
>
> Key: IGNITE-3175
> URL: https://issues.apache.org/jira/browse/IGNITE-3175
> Project: Ignite
>  Issue Type: Bug
>  Components: Ignite RDD
>Affects Versions: 1.5.0.final
>Reporter: Valentin Kulichenko
> Fix For: 1.7
>
>
> If one of the fields participating in the query is {{BigDecimal}}, the query 
> will fail when executed from {{IgniteRDD}} with the following error:
> {noformat}
> scala.MatchError: 1124757 (of class java.math.BigDecimal)
>   at 
> org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toCatalystImpl(CatalystTypeConverters.scala:255)
>   at 
> org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toCatalystImpl(CatalystTypeConverters.scala:250)
>   at 
> org.apache.spark.sql.catalyst.CatalystTypeConverters$CatalystTypeConverter.toCatalyst(CatalystTypeConverters.scala:102)
>   at 
> org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toCatalystImpl(CatalystTypeConverters.scala:260)
>   at 
> org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toCatalystImpl(CatalystTypeConverters.scala:250)
>   at 
> org.apache.spark.sql.catalyst.CatalystTypeConverters$CatalystTypeConverter.toCatalyst(CatalystTypeConverters.scala:102)
>   at 
> org.apache.spark.sql.catalyst.CatalystTypeConverters$$anonfun$createToCatalystConverter$2.apply(CatalystTypeConverters.scala:401)
>   at 
> org.apache.spark.sql.SQLContext$$anonfun$6.apply(SQLContext.scala:492)
>   at 
> org.apache.spark.sql.SQLContext$$anonfun$6.apply(SQLContext.scala:492)
>   at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>   at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>   at 
> org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.processInputs(TungstenAggregationIterator.scala:505)
>   at 
> org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.(TungstenAggregationIterator.scala:686)
>   at 
> org.apache.spark.sql.execution.aggregate.TungstenAggregate$$anonfun$doExecute$1$$anonfun$2.apply(TungstenAggregate.scala:95)
>   at 
> org.apache.spark.sql.execution.aggregate.TungstenAggregate$$anonfun$doExecute$1$$anonfun$2.apply(TungstenAggregate.scala:86)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
>   at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>   at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
>   at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
>   at org.apache.spark.scheduler.Task.run(Task.scala:89)
>   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Most likely this is caused by the fact that {{IgniteRDD.dataType()}} method 
> doesn't honor {{BigDecimal}} and returns {{StructType}} by default. We should 
> fix this and check other possible types as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-2969) Optimistic transactions support in deadlock detection

2016-05-22 Thread Andrey Gura (JIRA) 

[ 
https://issues.apache.org/jira/browse/IGNITE-2969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295725#comment-15295725
 ] 

Andrey Gura commented on IGNITE-2969:
-

One more problem found. In cases when some transaction wasn't timed out during 
prepare stage system discovers timeout during finishing and exception throws:

{noformat}
[20:40:47,872][ERROR][sys-#1229%transactions.TxOptimisticDeadlockDetectionTest3%][GridNearTxLocal]
 Failed to commit transaction: GridDhtTxLocal 
[nearNodeId=6d640bd9-19c9-4fc2-b3fc-ae84e805, 
nearFutId=66b7989d451-ee1a655e-73f8-4db0-a2f5-6f3d7221dad8, 
nearMiniId=8db7989d451-ee1a655e-73f8-4db0-a2f5-6f3d7221dad8, 
nearFinFutId=e6b7989d451-ee1a655e-73f8-4db0-a2f5-6f3d7221dad8, 
nearFinMiniId=efb7989d451-ee1a655e-73f8-4db0-a2f5-6f3d7221dad8, 
nearXidVer=GridCacheVersion [topVer=75418777, nodeOrderDrId=6, 
globalTime=1463938847069, order=1463938774976], super=GridDhtTxLocalAdapter 
[nearOnOriginatingNode=false, nearNodes=[], 
dhtNodes=[1ba1738d-6bc1-4581-9c70-c3a823f0], explicitLock=false, 
super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false, 
depEnabled=true, txState=IgniteTxStateImpl [activeCacheIds=GridLongList [idx=1, 
arr=[94416770]], txMap={IgniteTxKey [key=KeyCacheObjectImpl [val=317, 
hasValBytes=true], cacheId=94416770]=IgniteTxEntry [key=KeyCacheObjectImpl 
[val=317, hasValBytes=true], cacheId=94416770, txKey=IgniteTxKey 
[key=KeyCacheObjectImpl [val=317, hasValBytes=true], cacheId=94416770], 
val=[op=CREATE, val=CacheObjectImpl [val=2, hasValBytes=true]], 
prevVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, 
conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, 
filters=[], filtersPassed=false, filtersSet=false, 
entry=GridDhtColocatedCacheEntry [super=GridDhtCacheEntry [rdrs=[], 
locPart=GridDhtLocalPartition [id=317, mapPubSize=1, 
rmvQueue=GridCircularBuffer [sizeMask=31, idxGen=0], cntr=0, state=OWNING, 
reservations=0, empty=false, createTime=05/22/2016 20:40:46, mapPubSize=1], 
super=GridDistributedCacheEntry [super=GridCacheMapEntry 
[key=KeyCacheObjectImpl [val=317, hasValBytes=true], val=null, 
startVer=1463938775005, ver=GridCacheVersion [topVer=75418777, nodeOrderDrId=4, 
globalTime=1463938847857, order=1463938775005], hash=1633955401, 
extras=GridCacheMvccEntryExtras [mvcc=GridCacheMvcc 
[locs=[GridCacheMvccCandidate [nodeId=906a9b13-3558-4b6f-9bed-73346e13, 
ver=GridCacheVersion [topVer=75418777, nodeOrderDrId=4, 
globalTime=1463938847857, order=1463938775004], timeout=13, ts=1463938847850, 
threadId=2753, id=4202, topVer=AffinityTopologyVersion [topVer=8, 
minorTopVer=39], reentry=null, 
otherNodeId=6d640bd9-19c9-4fc2-b3fc-ae84e805, otherVer=GridCacheVersion 
[topVer=75418777, nodeOrderDrId=6, globalTime=1463938847069, 
order=1463938774976], mappedDhtNodes=null, mappedNearNodes=null, ownerVer=null, 
serOrder=null, key=KeyCacheObjectImpl [val=317, hasValBytes=true], 
masks=local=1|owner=1|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0,
 prevVer=null, nextVer=null]], rmts=null]], flags=0, prepared=true, 
locked=false, nodeId=null, locMapped=false, expiryPlc=null, 
transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, 
xidVer=null], IgniteTxKey [key=KeyCacheObjectImpl [val=319, hasValBytes=true], 
cacheId=94416770]=IgniteTxEntry [key=KeyCacheObjectImpl [val=319, 
hasValBytes=true], cacheId=94416770, txKey=IgniteTxKey [key=KeyCacheObjectImpl 
[val=319, hasValBytes=true], cacheId=94416770], val=[op=CREATE, 
val=CacheObjectImpl [val=2, hasValBytes=true]], prevVal=[op=NOOP, val=null], 
entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, 
explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, 
filtersSet=false, entry=GridDhtColocatedCacheEntry [super=GridDhtCacheEntry 
[rdrs=[], locPart=GridDhtLocalPartition [id=319, mapPubSize=1, 
rmvQueue=GridCircularBuffer [sizeMask=31, idxGen=0], cntr=0, state=OWNING, 
reservations=0, empty=false, createTime=05/22/2016 20:40:46, mapPubSize=1], 
super=GridDistributedCacheEntry [super=GridCacheMapEntry 
[key=KeyCacheObjectImpl [val=319, hasValBytes=true], val=null, 
startVer=1463938775007, ver=GridCacheVersion [topVer=75418777, nodeOrderDrId=4, 
globalTime=1463938847863, order=1463938775007], hash=802152124, 
extras=GridCacheMvccEntryExtras [mvcc=GridCacheMvcc 
[locs=[GridCacheMvccCandidate [nodeId=906a9b13-3558-4b6f-9bed-73346e13, 
ver=GridCacheVersion [topVer=75418777, nodeOrderDrId=4, 
globalTime=1463938847857, order=1463938775004], timeout=3, ts=1463938847860, 
threadId=2753, id=4207, topVer=AffinityTopologyVersion [topVer=8, 
minorTopVer=39], reentry=null, 
otherNodeId=6d640bd9-19c9-4fc2-b3fc-ae84e805, otherVer=GridCacheVersion 
[topVer=75418777, nodeOrderDrId=6, globalTime=1463938847069, 
order=1463938774976], mappedDhtNodes=null, mappedNearNodes=null, ownerVer=null, 
serOrder=null,

[jira] [Commented] (IGNITE-2538) RDD.isEmpty method should be overriden in IgniteRDD to use IgniteCache API

2016-05-22 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/IGNITE-2538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295527#comment-15295527
 ] 

ASF GitHub Bot commented on IGNITE-2538:


GitHub user f7753 opened a pull request:

https://github.com/apache/ignite/pull/741

IGNITE-2538 RDD.isEmpty method should be overriden in IgniteRDD to use 
IgniteCache API

Currently calling `IgniteRDD.isEmpty` ends up in execution of 1024 Spark 
jobs, which is not fast. We should override this method and use native 
IgniteCache API there.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/f7753/ignite igniterdd-native

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/741.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #741


commit 0f15e7ce21cfb02526b1d2bcfccd8850422b4f43
Author: MaBiao 
Date:   2016-05-22T08:15:19Z

overrride  method isEmpty

commit 5c5062f8917db13e1c9c50a30640298b8c40cf34
Author: MaBiao 
Date:   2016-05-22T11:47:30Z

modified the IgniteRDD.isEmpty method, using 'native' method to implement




> RDD.isEmpty method should be overriden in IgniteRDD to use IgniteCache API
> --
>
> Key: IGNITE-2538
> URL: https://issues.apache.org/jira/browse/IGNITE-2538
> Project: Ignite
>  Issue Type: Improvement
>  Components: Ignite RDD
>Reporter: Valentin Kulichenko
>
> Currently calling {{IgniteRDD.isEmpty}} ends up in execution of 1024 Spark 
> jobs, which is not fast. We should override this method and use native 
> {{IgniteCache}} API there. Other methods should be also revisited.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)