Re: what version of spark is supported by ignite 2.5

2018-07-18 Thread wt 
here is my rmdbs integration server config






http://www.springframework.org/schema/beans;
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
   xmlns:util="http://www.springframework.org/schema/util;
   xsi:schemaLocation="http://www.springframework.org/schema/beans
  
http://www.springframework.org/schema/beans/spring-beans.xsd
   http://www.springframework.org/schema/util
  
http://www.springframework.org/schema/util/spring-util.xsd;>





















127.0.0.1:47500..47510

Re: what version of spark is supported by ignite 2.5

2018-07-18 Thread wt 
could it be that spark is not compatible with ignite rmdbs integration? If i
run an example without that it works (writing to and reading from caches and
tables)



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: what version of spark is supported by ignite 2.5

2018-07-18 Thread aealexsandrov 
So you said that if I will try to start provided example I will see the same
error? I mean that I can try to investigate the problem in case if I will be
able to reproduce the same behavior.

Let me some time to take a look at this example.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: what version of spark is supported by ignite 2.5

2018-07-18 Thread aealexsandrov 
Hi,

First of all, you shouldn't use spark 2.1. With Ignite because you could
have conflicts of spark versions. 

>From your log when you used ignite spark (that used spark 2.2) I see that
you have the problem with spring configuration:

class org.apache.ignite.IgniteException: Spring application context resource
is not injected. 

I can't say why you face it using the provided code lines. 

Could you please provide the reproducer example on GitHub (or analog) to
analyze?

BR,
Andrei



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: what version of spark is supported by ignite 2.5

2018-07-18 Thread wt 
If i run the same code on spark 2.1 i get the following error (same jars in
classpath)

Welcome to
    __
 / __/__  ___ _/ /__
_\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.0
  /_/

Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java
1.8.0_151)
Type in expressions to have them evaluated.
Type :help for more information.

scala> import org.apache.ignite.spark.IgniteDataFrameSettings._
import org.apache.ignite.spark.IgniteDataFrameSettings._

scala>import org.apache.ignite.spark._
import org.apache.ignite.spark._

scala>
 |val df =
spark.read.format(FORMAT_IGNITE).option(OPTION_TABLE,"ComplianceSummaryCache.ComplianceSummary").option(OPTION_CONFIG_FILE,"stoxx-server.xml").load()
18/07/18 10:56:10 WARN GenericApplicationContext: Exception encountered
during context initialization - cancelling refresh attempt:
org.springframework.beans.factory.BeanCreationException: Error creating bean
with name 'org.apache.ignite.configuration.IgniteConfiguration#0' defined in
URL [file:/C:/spark_2.1/conf/stoxx-server.xm
org.apache.ignite.IgniteCheckedException: Failed to instantiate Spring XML
application context (make sure all classes used in Spring configuration are
present at CLASSPATH) [springUrl=file:/C:/spark_2.1/conf/stoxx-server.xml]
  at
org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:387)
  at
org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:104)
  at
org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:98)
  at
org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:744)
  at
org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:783)
  at
org.apache.ignite.internal.IgnitionEx.loadConfiguration(IgnitionEx.java:823)
  at
org.apache.ignite.spark.impl.IgniteRelationProvider$$anonfun$configProvider$1$1.apply(IgniteRelationProvider.scala:216)
  at
org.apache.ignite.spark.impl.IgniteRelationProvider$$anonfun$configProvider$1$1.apply(IgniteRelationProvider.scala:213)
  at org.apache.ignite.spark.Once.apply(IgniteContext.scala:222)
  at org.apache.ignite.spark.IgniteContext.ignite(IgniteContext.scala:144)
  at org.apache.ignite.spark.IgniteContext.(IgniteContext.scala:63)
  at org.apache.ignite.spark.IgniteContext$.apply(IgniteContext.scala:192)
  at
org.apache.ignite.spark.impl.IgniteRelationProvider.igniteContext(IgniteRelationProvider.scala:236)
  at
org.apache.ignite.spark.impl.IgniteRelationProvider.createRelation(IgniteRelationProvider.scala:62)
  at
org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:330)
  at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:152)
  at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:125)
  ... 53 elided
Caused by: org.springframework.beans.factory.BeanCreationException: Error
creating bean with name
'org.apache.ignite.configuration.IgniteConfiguration#0' defined in URL
[file:/C:/spark_2.1/conf/stoxx-server.xml]: Initialization of bean failed;
nested exception is java.lang.NoClassDefFoundError:
javax/cache/configuration/Factory
  at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:564)
  at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
  at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
  at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
  at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
  at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
  at
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
  at
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:866)
  at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542)
  at
org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:381)
  ... 69 more
Caused by: java.lang.NoClassDefFoundError: javax/cache/configuration/Factory
  at java.lang.Class.getDeclaredMethods0(Native Method)
  at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
  at java.lang.Class.privateGetPublicMethods(Class.java:2902)
  at java.lang.Class.getMethods(Class.java:1615)
  at
org.springframework.beans.ExtendedBeanInfoFactory.supports(ExtendedBeanInfoFactory.java:54)
  at

Re: what version of spark is supported by ignite 2.5

2018-07-18 Thread wt 
thanks but that doesnt work

I have the folowing jars in spark classpath for spark 2.2

cache-api-1.0.0.jar
spring-expression-4.3.7.RELEASE.jar
spring-context-4.3.7.RELEASE.jar
spring-core-4.3.7.RELEASE.jar
spring-beans-4.3.7.RELEASE.jar
ignite-spring-2.5.0.jar
ignite-core-2.5.0.jar
ignite-spark-2.5.0.jar


running spark in local mode i execute the following code:

import org.apache.ignite.spark.IgniteDataFrameSettings._
   import org.apache.ignite.spark._
   
   val df =
spark.read.format(FORMAT_IGNITE).option(OPTION_TABLE,"ComplianceSummaryCache.ComplianceSummary").option(OPTION_CONFIG_FILE,"stoxx-server.xml").load()

I get the following error 


Welcome to
    __
 / __/__  ___ _/ /__
_\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.0
  /_/

Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java
1.8.0_151)
Type in expressions to have them evaluated.
Type :help for more information.

scala> import org.apache.ignite.spark.IgniteDataFrameSettings._
import org.apache.ignite.spark.IgniteDataFrameSettings._

scala>import org.apache.ignite.spark._
import org.apache.ignite.spark._

scala>
 |val df =
spark.read.format(FORMAT_IGNITE).option(OPTION_TABLE,"ComplianceSummaryCache.ComplianceSummary").option(OPTION_CONFIG_FILE,"stoxx-server.xml").load()
18/07/18 10:59:06 ERROR : Failed to resolve default logging config file:
config/java.util.logging.properties
Console logging handler is not configured.
[10:59:07]__  
[10:59:07]   /  _/ ___/ |/ /  _/_  __/ __/
[10:59:07]  _/ // (7 7// /  / / / _/
[10:59:07] /___/\___/_/|_/___/ /_/ /___/
[10:59:07]
[10:59:07] ver. 2.5.0#20180523-sha1:86e110c7
[10:59:07] 2018 Copyright(C) Apache Software Foundation
[10:59:07]
[10:59:07] Ignite documentation: http://ignite.apache.org
[10:59:07]
[10:59:07] Quiet mode.
[10:59:07]   ^-- Logging by 'JavaLogger [quiet=true, config=null]'
[10:59:07]   ^-- To see **FULL** console log here add -DIGNITE_QUIET=false
or "-v" to ignite.{sh|bat}
[10:59:07]
[10:59:07] OS: Windows 7 6.1 amd64
[10:59:07] VM information: Java(TM) SE Runtime Environment 1.8.0_151-b12
Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.151-b12
18/07/18 10:59:07 WARN : Possible too long JVM pause: 510 milliseconds.
18/07/18 10:59:07 WARN GridDiagnostic: Initial heap size is 128MB (should be
no less than 512MB, use -Xms512m -Xmx512m).
[10:59:07] Initial heap size is 128MB (should be no less than 512MB, use
-Xms512m -Xmx512m).
[10:59:07] Configured plugins:
[10:59:07]   ^-- None
[10:59:07]
[10:59:07] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler
[tryStop=false, timeout=0]]
18/07/18 10:59:08 WARN TcpCommunicationSpi: Message queue limit is set to 0
which may lead to potential OOMEs when running cache operations in
FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and
receiver sides.
[10:59:08] Message queue limit is set to 0 which may lead to potential OOMEs
when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to
message queues growth on sender and receiver sides.
18/07/18 10:59:08 WARN NoopCheckpointSpi: Checkpoints are disabled (to
enable configure any GridCheckpointSpi implementation)
18/07/18 10:59:08 WARN GridCollisionManager: Collision resolution is
disabled (all jobs will be activated upon arrival).
[10:59:08] Security status [authentication=off, tls/ssl=off]
[10:59:08] REST protocols do not start on client node. To start the
protocols on client node set '-DIGNITE_REST_START_ON_CLIENT=true' system
property.
18/07/18 10:59:13 ERROR GridDhtPartitionsExchangeFuture: Failed to
reinitialize local partitions (preloading will be stopped):
GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=2,
minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=0c59ff6b-7c95-4072-9866-844f87a6f12d, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1,
172.19.112.175], sockAddrs=[PC50439.oa.pnrad.net/172.19.112.175:0,
/0:0:0:0:0:0:0:1:0, /127.0.0.1:0], discPort=0, order=2, intOrder=0,
lastExchangeTime=1531907
49159, loc=true, ver=2.5.0#20180523-sha1:86e110c7, isClient=true], topVer=2,
nodeId8=0c59ff6b, msg=null, type=NODE_JOINED, tstamp=1531907953545],
nodeId=0c59ff6b, evt=NODE_JOINED]
class org.apache.ignite.IgniteException: Spring application context resource
is not injected.
at
org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory.create(CacheJdbcPojoStoreFactory.java:171)
at
org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory.create(CacheJdbcPojoStoreFactory.java:100)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.createCache(GridCacheProcessor.java:1437)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheStart(GridCacheProcessor.java:1945)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCachesOnLocalJoin(GridCacheProcessor.java:1830)
at

Re: what version of spark is supported by ignite 2.5

2018-07-18 Thread aealexsandrov 
Sorry, I have a typo. ignite contains "spark-core_2.11" inside
. Not spark-core_2.10.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/