Re: Error in running wordcount hadoop example in ignite

[Mehdi Seydali]

thanks that works with 2.4

On Tue, Apr 2, 2019 at 7:32 AM Denis Mekhanikov 
wrote:

> Hi!
>
> As far as I can see, you tried running examples from Hadoop 2.7.7
> But Ignite uses Hadoop version 2.4.1 internally.
> So, I would start from checking the same examples for the matching version
> of Hadoop (2.4.1).
>
> Denis
>
> чт, 28 февр. 2019 г. в 08:58, mehdi sey :
>
>> hi
>> i want to execute wordcount example of hadoop in apache ignite. i have
>> used
>> apache-ignite-hadoop-2.6.0-bin for execute map reduce tasks. my
>> default-config.xml in apache-ignite-hadoop-2.6.0-bin/config folder is just
>> as bellow:
>> http://www.springframework.org/schema/beans";
>>xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
>> xmlns:util="http://www.springframework.org/schema/util";
>>xsi:schemaLocation="http://www.springframework.org/schema/beans
>>http://www.springframework.org/schema/beans/spring-beans.xsd
>>http://www.springframework.org/schema/util
>>http://www.springframework.org/schema/util/spring-util.xsd";>
>>
>>
>> 
>> Spring file for Ignite node configuration with IGFS and Apache
>> Hadoop map-reduce support enabled.
>> Ignite node will start with this configuration by default.
>> 
>>
>>
>> >
>> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>> > value="SYSTEM_PROPERTIES_MODE_FALLBACK"/>
>> 
>> 
>>
>>
>> > class="org.apache.ignite.configuration.IgniteConfiguration">
>>
>> 
>> > class="org.apache.ignite.configuration.ConnectorConfiguration">
>> 
>> 
>> 
>>
>> 
>> 
>>
>> > static-field="org.apache.ignite.events.EventType.EVT_TASK_STARTED"/>
>> > static-field="org.apache.ignite.events.EventType.EVT_TASK_FINISHED"/>
>> > static-field="org.apache.ignite.events.EventType.EVT_JOB_MAPPED"/>
>> > static-field="org.apache.ignite.events.EventType.EVT_TASK_FAILED"/>
>> > static-field="org.apache.ignite.events.EventType.EVT_TASK_TIMEDOUT"/>
>> >
>> static-field="org.apache.ignite.events.EventType.EVT_TASK_SESSION_ATTR_SET"/>
>> > static-field="org.apache.ignite.events.EventType.EVT_TASK_REDUCED"/>
>>
>>
>> > static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_PUT"/>
>> > static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_READ"/>
>> >
>> static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_REMOVED"/>
>> 
>> 
>>
>> i have run an ignite node with bellow command in command line in linux
>> ubuntu:
>> */usr/local/apache-ignite-hadoop-2.6.0-bin/bin/ignite.sh
>> /usr/local/apache-ignite-hadoop-2.6.0-bin/config/default-config.xml*
>>
>> after starting ignite node i execute a wordcount example in hadoop for
>> ruuning in ignite with bellow command:
>>
>> *./hadoop jar
>>
>> /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.7.jar
>> wordcount /input/hadoop output2*
>>
>> but after executing above command i have encounter an error just as
>> attached
>> image. please help about solving problem. i have seen also below link but
>> it
>> could not help.
>>
>> http://apache-ignite-users.70518.x6.nabble.com/NPE-issue-with-trying-to-submit-Hadoop-MapReduce-tc2146.html#a2183
>> 
>> 
>>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Re: Failed to start grid: com/google/common/base/Preconditions

[Mehdi Seydali]

i dont really know where "30s" comes from. please verify my config that are
them correct?
i SecondaryFileSystem my configPath just as below:








/usr/local/hadoop/etc/hadoop/core-site.xml







where /usr/local/hadoop/etc/hadoop/core-site.xml file is just as below:


  fs.defaultFS
  hdfs://localhost:9000
   

   
  hadoop.tmp.dir
  /app/hadoop/tmp
   
but when i want to execute worcount example in hadoop i have use a config
in ignite_config folder for hadoop just as below. ignite_config folder
consist of two files name as core-site.xml and mapred-site.xml.
hint: command for executing wordcount example in hadoop
time hadoop --config
/home/mehdi/ignite-conf/ignite-configs-master/igfs-hadoop-fs-cache/ignite_conf
jar
/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar
wordcount /user/input/ /output

i have attached core-site.xml and mapred-site.xml. please give me an
aknowledge that is it correct this configuration for using igfs as cache
for hdfs?



On Tue, Mar 12, 2019 at 6:46 PM Ilya Kasnacheev 
wrote:

> Hello!
>
> Where does this "30s" setting comes from? I guess it should be nominated
> in ms, as a number. However, unless this value comes from Ignite in some
> way, we're not related to that.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> вт, 12 мар. 2019 г. в 13:14, Mehdi Seydali :
>
>> i added java.lang.ClassNotFoundException:
>> com.ctc.wstx.io.InputBootstrapper from previous email and added to ignite
>> lib. after adding this library ignite node started but i have encounter
>> another error just like below:
>>
>> [13:03:28]__  
>> [13:03:28]   /  _/ ___/ |/ /  _/_  __/ __/
>> [13:03:28]  _/ // (7 7// /  / / / _/
>> [13:03:28] /___/\___/_/|_/___/ /_/ /___/
>> [13:03:28]
>> [13:03:28] ver. 2.6.0#20180710-sha1:669feacc
>> [13:03:28] 2018 Copyright(C) Apache Software Foundation
>> [13:03:28]
>> [13:03:28] Ignite documentation: http://ignite.apache.org
>> [13:03:28]
>> [13:03:28] Quiet mode.
>> [13:03:28]   ^-- Logging to file
>> '/usr/local/apache-ignite-fabric-2.6.0-bin/work/log/ignite-41a0490a.log'
>> [13:03:28]   ^-- Logging by 'Log4JLogger [quiet=true,
>> config=/usr/local/apache-ignite-fabric-2.6.0-bin/config/ignite-log4j.xml]'
>> [13:03:28]   ^-- To see **FULL** console log here add
>> -DIGNITE_QUIET=false or "-v" to ignite.{sh|bat}
>> [13:03:28]
>> [13:03:28] OS: Linux 4.15.0-46-generic amd64
>> [13:03:28] VM information: Java(TM) SE Runtime Environment
>> 1.8.0_192-ea-b04 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM
>> 25.192-b04
>> [13:03:28] Configured plugins:
>> [13:03:28]   ^-- Ignite Native I/O Plugin [Direct I/O]
>> [13:03:28]   ^-- Copyright(C) Apache Software Foundation
>> [13:03:28]
>> [13:03:28] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler
>> [tryStop=false, timeout=0]]
>> [13:03:28] Message queue limit is set to 0 which may lead to potential
>> OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due
>> to message queues growth on sender and receiver sides.
>> [13:03:29] Security status [authentication=off, tls/ssl=off]
>> SLF4J: Class path contains multiple SLF4J bindings.
>> SLF4J: Found binding in
>> [jar:file:/usr/local/apache-ignite-fabric-2.6.0-bin/libs/slf4j-nop-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> SLF4J: Found binding in
>> [jar:file:/usr/local/apache-ignite-fabric-2.6.0-bin/libs/htrace%20dependency/slf4j-nop-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> SLF4J: Found binding in
>> [jar:file:/usr/local/apache-ignite-fabric-2.6.0-bin/libs/ignite-rest-http/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> SLF4J: Found binding in
>> [jar:file:/usr/local/apache-ignite-fabric-2.6.0-bin/libs/ignite-yarn/ignite-yarn-2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> SLF4J: Found binding in
>> [jar:file:/usr/local/apache-ignite-fabric-2.6.0-bin/libs/ignite-zookeeper/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>> explanation.
>> SLF4J: Actual binding is of type [org.slf4j.helpers.NOPLoggerFactory]
>> [13:03:31] HADOOP_HOME is set to /usr/local/hadoop
>&g

Re: Failed to start grid: com/google/common/base/Preconditions

[Mehdi Seydali]

i added java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper
from previous email and added to ignite lib. after adding this library
ignite node started but i have encounter another error just like below:

[13:03:28]__  
[13:03:28]   /  _/ ___/ |/ /  _/_  __/ __/
[13:03:28]  _/ // (7 7// /  / / / _/
[13:03:28] /___/\___/_/|_/___/ /_/ /___/
[13:03:28]
[13:03:28] ver. 2.6.0#20180710-sha1:669feacc
[13:03:28] 2018 Copyright(C) Apache Software Foundation
[13:03:28]
[13:03:28] Ignite documentation: http://ignite.apache.org
[13:03:28]
[13:03:28] Quiet mode.
[13:03:28]   ^-- Logging to file
'/usr/local/apache-ignite-fabric-2.6.0-bin/work/log/ignite-41a0490a.log'
[13:03:28]   ^-- Logging by 'Log4JLogger [quiet=true,
config=/usr/local/apache-ignite-fabric-2.6.0-bin/config/ignite-log4j.xml]'
[13:03:28]   ^-- To see **FULL** console log here add -DIGNITE_QUIET=false
or "-v" to ignite.{sh|bat}
[13:03:28]
[13:03:28] OS: Linux 4.15.0-46-generic amd64
[13:03:28] VM information: Java(TM) SE Runtime Environment 1.8.0_192-ea-b04
Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.192-b04
[13:03:28] Configured plugins:
[13:03:28]   ^-- Ignite Native I/O Plugin [Direct I/O]
[13:03:28]   ^-- Copyright(C) Apache Software Foundation
[13:03:28]
[13:03:28] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler
[tryStop=false, timeout=0]]
[13:03:28] Message queue limit is set to 0 which may lead to potential
OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due
to message queues growth on sender and receiver sides.
[13:03:29] Security status [authentication=off, tls/ssl=off]
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/local/apache-ignite-fabric-2.6.0-bin/libs/slf4j-nop-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/apache-ignite-fabric-2.6.0-bin/libs/htrace%20dependency/slf4j-nop-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/apache-ignite-fabric-2.6.0-bin/libs/ignite-rest-http/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/apache-ignite-fabric-2.6.0-bin/libs/ignite-yarn/ignite-yarn-2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/apache-ignite-fabric-2.6.0-bin/libs/ignite-zookeeper/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.helpers.NOPLoggerFactory]
[13:03:31] HADOOP_HOME is set to /usr/local/hadoop
[13:03:31] Resolved Hadoop classpath locations:
/usr/local/hadoop/share/hadoop/common, /usr/local/hadoop/share/hadoop/hdfs,
/usr/local/hadoop/share/hadoop/mapreduce
[13:03:32] Nodes started on local machine require more than 20% of physical
RAM what can lead to significant slowdown due to swapping (please decrease
JVM heap size, data region size or checkpoint buffer size)
[required=5344MB, available=7953MB]
[13:03:34] Performance suggestions for grid  (fix if possible)
[13:03:34] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
[13:03:34]   ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM
options)
[13:03:34]   ^-- Set max direct memory size if getting 'OOME: Direct buffer
memory' (add '-XX:MaxDirectMemorySize=[g|G|m|M|k|K]' to JVM options)
[13:03:34]   ^-- Disable processing of calls to System.gc() (add
'-XX:+DisableExplicitGC' to JVM options)
[13:03:34] Refer to this page for more performance suggestions:
https://apacheignite.readme.io/docs/jvm-and-system-tuning
[13:03:34]
[13:03:34] To start Console Management & Monitoring run
ignitevisorcmd.{sh|bat}
[13:03:34]
[13:03:34] Ignite node started OK (id=41a0490a)
[13:03:34] Topology snapshot [ver=4, servers=2, clients=0, CPUs=8,
offheap=3.1GB, heap=2.0GB]
[13:03:34]   ^-- Node [id=41A0490A-8A7B-4977-A48B-B5D98D49CF1B,
clusterState=ACTIVE]
[13:03:34] Data Regions Configured:
[13:03:34]   ^-- default [initSize=256.0 MiB, maxSize=1.6 GiB,
persistenceEnabled=false]
[13:03:42] New version is available at ignite.apache.org: 2.7.0
[13:03:56] Topology snapshot [ver=5, servers=1, clients=0, CPUs=8,
offheap=1.6GB, heap=1.0GB]
[13:03:56]   ^-- Node [id=41A0490A-8A7B-4977-A48B-B5D98D49CF1B,
clusterState=ACTIVE]
[13:03:56] Data Regions Configured:
[13:03:56]   ^-- default [initSize=256.0 MiB, maxSize=1.6 GiB,
persistenceEnabled=false]
[2019-03-12 13:04:08,878][ERROR][igfs-igfs-ipc-#64][IgfsImpl] File info
operation in DUAL mode failed [path=/output]
class org.apache.ignite.IgniteException: For input string: "30s"
at
org.apache.ignite.internal.processors.hadoop.impl.fs.HadoopLazyConcurrentMap.getOrCreate(HadoopLazyConcurrentMap.java:100)
at
org.apache.ignite.internal.processors.hadoop.impl.delegate.HadoopCachingFileSystemFactoryDelegate.getWithMappedName(HadoopCachingFileSystemFactoryDelegate.ja

Re: Failed to start grid: com/google/common/base/Preconditions

[Mehdi Seydali]

ignite-hadoop was in lib path of ignite but i aslo copied other optional
folder module to lib folder and i have encounter another error just like
below:
[12:34:14]__  
[12:34:14]   /  _/ ___/ |/ /  _/_  __/ __/
[12:34:14]  _/ // (7 7// /  / / / _/
[12:34:14] /___/\___/_/|_/___/ /_/ /___/
[12:34:14]
[12:34:14] ver. 2.6.0#20180710-sha1:669feacc
[12:34:14] 2018 Copyright(C) Apache Software Foundation
[12:34:14]
[12:34:14] Ignite documentation: http://ignite.apache.org
[12:34:14]
[12:34:14] Quiet mode.
[12:34:14]   ^-- Logging to file
'/usr/local/apache-ignite-fabric-2.6.0-bin/work/log/ignite-96498dcf.log'
[12:34:14]   ^-- Logging by 'Log4JLogger [quiet=true,
config=/usr/local/apache-ignite-fabric-2.6.0-bin/config/ignite-log4j.xml]'
[12:34:14]   ^-- To see **FULL** console log here add -DIGNITE_QUIET=false
or "-v" to ignite.{sh|bat}
[12:34:14]
[12:34:14] OS: Linux 4.15.0-46-generic amd64
[12:34:14] VM information: Java(TM) SE Runtime Environment 1.8.0_192-ea-b04
Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.192-b04
[12:34:14] Configured plugins:
[12:34:14]   ^-- Ignite Native I/O Plugin [Direct I/O]
[12:34:14]   ^-- Copyright(C) Apache Software Foundation
[12:34:14]
[12:34:14] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler
[tryStop=false, timeout=0]]
[12:34:14] Message queue limit is set to 0 which may lead to potential
OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due
to message queues growth on sender and receiver sides.
[12:34:14] Security status [authentication=off, tls/ssl=off]
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/local/apache-ignite-fabric-2.6.0-bin/libs/ignite-rest-http/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/apache-ignite-fabric-2.6.0-bin/libs/ignite-yarn/ignite-yarn-2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/apache-ignite-fabric-2.6.0-bin/libs/ignite-zookeeper/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[2019-03-11 12:34:15,475][ERROR][main][IgniteKernal] Exception during start
processors, node will be stopped and close connections
java.lang.NoClassDefFoundError: com/ctc/wstx/io/InputBootstrapper
at
org.apache.ignite.internal.processors.hadoop.impl.HadoopUtils.safeCreateConfiguration(HadoopUtils.java:334)
at
org.apache.ignite.internal.processors.hadoop.impl.delegate.HadoopBasicFileSystemFactoryDelegate.start(HadoopBasicFileSystemFactoryDelegate.java:129)
at
org.apache.ignite.internal.processors.hadoop.impl.delegate.HadoopCachingFileSystemFactoryDelegate.start(HadoopCachingFileSystemFactoryDelegate.java:58)
at
org.apache.ignite.internal.processors.hadoop.impl.delegate.HadoopIgfsSecondaryFileSystemDelegateImpl.start(HadoopIgfsSecondaryFileSystemDelegateImpl.java:413)
at
org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem.start(IgniteHadoopIgfsSecondaryFileSystem.java:276)
at
org.apache.ignite.internal.processors.igfs.IgfsImpl.(IgfsImpl.java:185)
at
org.apache.ignite.internal.processors.igfs.IgfsContext.(IgfsContext.java:102)
at
org.apache.ignite.internal.processors.igfs.IgfsProcessor.start(IgfsProcessor.java:116)
at
org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1739)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:990)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2014)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1723)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1151)
at
org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1069)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:955)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:854)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:724)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:693)
at org.apache.ignite.Ignition.start(Ignition.java:352)
at
org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301)
Caused by: java.lang.ClassNotFoundException:
com.ctc.wstx.io.InputBootstrapper
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 20 more
[2019-03-11 12:34:15,478][ERROR][main][IgniteKernal] Got exception while
starting (will rollback startup routine).
java.lang.NoClassDefFoundError: com/ctc/wstx/io/InputBootstrapper
at
org.apache.ignite.internal.processors.hadoop.impl.HadoopUtil

Re: DatasetFactory.createSimpleDataset with a cache

[Mehdi Seydali]

i think classification and clustering is most important

On Wed, Jan 23, 2019 at 2:09 PM zaleslaw  wrote:

> Hi, what kind of ML algorithms are you going to use?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Distributed Training in tensorflow

[Mehdi Seydali]

i read your documentation. i have another question in my mind. you said
that tensorflow itself support distributed learning and we can  pure
TensorFlow you'll have to start TensorFlow workers manually and distribute
data manually as well. in you project you have considered tensorflow as
distributed or pure tensorflow?

On Wed, Jan 9, 2019 at 12:37 PM dmitrievanthony 
wrote:

> Let me also add that it depends on what you want to achieve. TensorFlow
> supports distributed training and it does it on it's own. But if you use
> pure TensorFlow you'll have to start TensorFlow workers manually and
> distribute data manually as well. And you can do it, I mean start workers
> manually on the nodes Ignite cluster occupies or even some other nodes. It
> will work and perhaps work well in some cases and work very well in case of
> accurate manual setup.
>
> At the same time, Apache Ignite provides a cluster management functionality
> for TensorFlow that allows to start workers automatically on the same nodes
> Apache Ignite keeps the data. From our perspective it's the most efficient
> way to setup TensorFlow cluster on top of Apache Ignite cluster because it
> allows to reduce data transfers. You can find more details about this in
> readme: https://apacheignite.readme.io/docs/ignite-dataset and
> https://apacheignite.readme.io/docs/tf-command-line-tool.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Distributed Training in tensorflow

[Mehdi Seydali]

i have another question. is it possible to implement neural network
algorithm on apache ignite directly? i think for example we implement RNN
on ignite node and execute them? do you have seen this subject in ignite
ML?

On Fri, Jan 11, 2019 at 2:29 PM zaleslaw  wrote:

> Yes, I agree with your conclusion. I have no benchmarks, of course, but it
> seems that no speedup in DL4j on Spark on Ignite RDD
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite and spark for deep learning

[Mehdi Seydali]

I have in doubt. In last mail you said that dl4j over ignite have no speed
up and you say now maybe develope in future.is it worthwhile for doing it?

On Friday, January 11, 2019, zaleslaw  wrote:

> Currently we have no integration with DL4j in ignite ML and have no such
> plans in roadmap.
> The TF is the best platform for the Deep Learning now and we spend time on
> integration with TF first of all.
>
> Maybe, in future, direct integration between DL4j and ignite will be added.
> But it costst a hundred hours of development, of course.
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Distributed Training in tensorflow

[Mehdi Seydali]

Yes you are write. I have many debate about this. I have an idea that if we
have dl4j ( running over spark)  what is the matter of doing run dl4j over
ignite.   previously i have this idea  but after googling and share with
you i think this is a waste time. Spark itself is in memory computing
platform also ignite is. In distributed deep learning with are going to
speed up learning via distribute model learning. Dl4j is a distributed deep
learning data model and i think with integrating it with ignite we have no
more speed up. It was in my opinion we can use igniterdd for speed up but i
underestand that in deep learning we rarely shared data for using
igniterdd. Do you agree with my interpretation?do you have any comment?

On Wednesday, January 9, 2019, dmitrievanthony 
wrote:

> Let me also add that it depends on what you want to achieve. TensorFlow
> supports distributed training and it does it on it's own. But if you use
> pure TensorFlow you'll have to start TensorFlow workers manually and
> distribute data manually as well. And you can do it, I mean start workers
> manually on the nodes Ignite cluster occupies or even some other nodes. It
> will work and perhaps work well in some cases and work very well in case of
> accurate manual setup.
>
> At the same time, Apache Ignite provides a cluster management functionality
> for TensorFlow that allows to start workers automatically on the same nodes
> Apache Ignite keeps the data. From our perspective it's the most efficient
> way to setup TensorFlow cluster on top of Apache Ignite cluster because it
> allows to reduce data transfers. You can find more details about this in
> readme: https://apacheignite.readme.io/docs/ignite-dataset and
> https://apacheignite.readme.io/docs/tf-command-line-tool.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite and spark for deep learning

[Mehdi Seydali]

I completely agree with you. I discuss many time with our team that this
integration have not any gain about speed up and i have your idea about
caching in ignite because in deep learning we have nothing to share in job
because every job independently works on it's portion of data. In your
opinion one idea can be ignite+ignite ml+ dl4j +spark? What benefit we can
achive in this integration. ?ignite ml cant used for deep independently?

On Wednesday, January 9, 2019, zaleslaw  wrote:

> Dear Mehdi Sey
>
> Yes, both platforms are used for in-memory computing, but they have
> different APIs and history of feature creation and different ways of
> integration with famous DL frameworks (like DL4j and TensorFlow).
>
> From my point of view, you have no speed up in Ignite + Spark + DL4j
> integration.
>
> Caching data in Ignite as a backend for RDD and dataframes first of all is
> acceleration of business logic based on SQL queries. Not the same for ML
> frameworks.
>
> We have no proof, that usage Ignite as a backend could speed up DL4j or
> MLlib algorithms.
>
> Moreover, to avoid this, we wrote own ML library which is more better than
> MLlib and runs natively on Ignite.
>
> In my opinon, you should choose Ignite + Ignite ML + TF integration or
> Spark
> + DL4j to solve your Data Science task (where you need neural networks).
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: JDK 11 support

[Mehdi Seydali]

No. I have run it on jdk 8 also

On Sunday, January 6, 2019, Yakov Zhdanov  wrote:

> Ignite 2.7 should run on JDK 11. Vladimir, Denis, correct me if I'm wrong.
>
> Please try this link for information - https://apacheignite.readme.
> io/v2.7/docs/getting-started#section-running-ignite-with-java-9
>
> --Yakov
>

Re: deep learning over apache ignite

[Mehdi Seydali]

i have a question in my mind. we know that spark and ignite are in memory
computing platform and both of them cache data. if we want to consider
caching data in ignite as a benefit, spark also cache data. in this idea if
we want to develope dl4j over ignite What will we gain except for ignite
computation?

On Tue, Dec 18, 2018 at 7:38 PM dmitrievanthony 
wrote:

> First of all, when you are talking about "acceleration of running deep
> neural
> network" what do you mean, training or inference?
> If you are talking about inference, it actually doesn't matter what cluster
> management system we use, we need only to run parallel/distributed
> inference.
> If you are talking about training, how are you going to speed it up? Use
> distributed training on multiple machines? Or GPU's? Or maybe you just want
> to speed up data preprocessing?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: JDK 11 support

[Mehdi Seydali]

i think you must run ignite on JDK 8.

On Sat, Jan 5, 2019 at 7:03 PM ignite_user2016 
wrote:

> what version of Ignite run on JDK 11 ? Does any one tried running Ignite on
> open JDK 11 ?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: diferrences between IgniteRdd and SparkRdd

[Mehdi Seydali]

I have another question. Could we create Rdd in spark separately frim
ignite or we must create rdd from ignite and share with spark. In other
words, is it possible create spark and ignite Rdd independently?

On Wednesday, December 26, 2018, aealexsandrov 
wrote:

> Hi,
>
> The main difference between native Spark RDD and IgniteRDD is that Ignite
> RDD provides a shared in-memory view on data across different Spark jobs,
> workers, or applications, while native Spark RDD cannot be seen by other
> Spark jobs or applications.
>
> You can see the attached image that describes it.
>
> spark-ignite-rdd.png
>  t1704/spark-ignite-rdd.png>
>
> According to your second question:
>
> The provided code do next:
>
> 1)Create Spark context
> 2)Creates Ignite context with provided configuration.
> 3)Creates an Ignite Shared RDD of Type (Int,Int)
> 4)Fill the Ignite Shared RDD in with Int pairs.
>
> To get the full example please take a look at ScalarSharedRDDExample.java
> that located in source files:
>
> https://github.com/apache/ignite/blob/master/examples/
> src/main/scala/org/apache/ignite/scalar/examples/spark/
> ScalarSharedRDDExample.scala
>
> BR,
> Andrei
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: deep learning over apache ignite

[Mehdi Seydali]

my dear friend i have another question in my mine. two platforms (spark and
ignite) use in memory for computing. instead of loading data into ignite
catch we also can loading data to spark memory and catch it. if we could
do  this why we load data to ignite catch?. loading to ignite catch have
benefit only for sharing rdd between spark jobs ad indexing query index. as
you said for only sharing ignite rdd you said that we must load data to
ignite?

On Sat, Dec 22, 2018 at 6:28 PM zaleslaw  wrote:

> I will try to answer these questions
> 1) We can using ignite as cache storage and preparing data for processing
> on
> dl4j?
> - yes you can, it's main case, to load data to ignite caches correctly
> 2) Job on spark can spawn hirarchy job for accelerate execution.
> - Ignite can compute piece pf code in parallel and etc, but you need to
> implement correct infrastrucure about code tasks spreading and so on over
> ignite.
> Look at our TF integration and distributing training
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: error in running shared rdd example with intellij

[Mehdi Seydali]

how i could change client mode to server mode?

On Sat, Dec 29, 2018 at 9:49 AM Mehdi Seydali 
wrote:

> i have run it in 4 nodes. but the same error has remained.
>
> On Fri, Dec 28, 2018 at 5:44 PM Ilya Kasnacheev 
> wrote:
>
>> Hello!
>>
>> Looks like this node is started in client node, but you don't have server
>> nodes available for it to connect to.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> пт, 28 дек. 2018 г. в 15:59, mehdi sey :
>>
>>> hi i want to run SharedRddExample in intellij IDE. but i have encounter
>>> error
>>> just as bellow . why?
>>> /usr/lib/jvm/java-8-oracle/bin/java
>>> -DIGNITE_HOME=/usr/local/apache-ignite-fabric-2.6.0-bin/
>>>
>>> -javaagent:/snap/intellij-idea-community/109/lib/idea_rt.jar=35933:/snap/intellij-idea-community/109/bin
>>> -Dfile.encoding=UTF-8 -classpath
>>>
>>> /usr/lib/jvm/java-8-oracle/jre/lib/charsets.jar:/usr/lib/jvm/java-8-oracle/jre/lib/deploy.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/cldrdata.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/dnsns.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/jaccess.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/jfxrt.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/localedata.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/nashorn.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/sunec.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/sunjce_provider.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/sunpkcs11.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/zipfs.jar:/usr/lib/jvm/java-8-oracle/jre/lib/javaws.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jce.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jfr.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jfxswt.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jsse.jar:/usr/lib/jvm/java-8-oracle/jre/lib/management-agent.jar:/usr/lib/jvm/java-8-oracle/jre/lib/plugin.jar:/usr/lib/jvm/java-8-oracle/jre/lib/resources.jar:/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar:/home/mehdi/SparkIgniteRDD/out/production/SparkIgniteRDD:/home/mehdi/.m2/repository/org/apache/spark/spark-core_2.11/2.4.0/spark-core_2.11-2.4.0.jar:/home/mehdi/.m2/repository/org/apache/avro/avro/1.8.2/avro-1.8.2.jar:/home/mehdi/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar:/home/mehdi/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar:/home/mehdi/.m2/repository/com/thoughtworks/paranamer/paranamer/2.7/paranamer-2.7.jar:/home/mehdi/.m2/repository/org/apache/commons/commons-compress/1.8.1/commons-compress-1.8.1.jar:/home/mehdi/.m2/repository/org/tukaani/xz/1.5/xz-1.5.jar:/home/mehdi/.m2/repository/org/apache/avro/avro-mapred/1.8.2/avro-mapred-1.8.2.jar:/home/mehdi/.m2/repository/org/apache/avro/avro-ipc/1.8.2/avro-ipc-1.8.2.jar:/home/mehdi/.m2/repository/commons-codec/commons-codec/1.9/commons-codec-1.9.jar:/home/mehdi/.m2/repository/com/twitter/chill_2.11/0.9.3/chill_2.11-0.9.3.jar:/home/mehdi/.m2/repository/com/esotericsoftware/kryo-shaded/4.0.2/kryo-shaded-4.0.2.jar:/home/mehdi/.m2/repository/com/esotericsoftware/minlog/1.3.0/minlog-1.3.0.jar:/home/mehdi/.m2/repository/org/objenesis/objenesis/2.5.1/objenesis-2.5.1.jar:/home/mehdi/.m2/repository/com/twitter/chill-java/0.9.3/chill-java-0.9.3.jar:/home/mehdi/.m2/repository/org/apache/xbean/xbean-asm6-shaded/4.8/xbean-asm6-shaded-4.8.jar:/home/mehdi/.m2/repository/org/apache/hadoop/hadoop-client/2.6.5/hadoop-client-2.6.5.jar:/home/mehdi/.m2/repository/org/apache/hadoop/hadoop-common/2.6.5/hadoop-common-2.6.5.jar:/home/mehdi/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/home/mehdi/.m2/repository/xmlenc/xmlenc/0.52/xmlenc-0.52.jar:/home/mehdi/.m2/repository/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar:/home/mehdi/.m2/repository/commons-io/commons-io/2.4/commons-io-2.4.jar:/home/mehdi/.m2/repository/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar:/home/mehdi/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/home/mehdi/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/home/mehdi/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar:/home/mehdi/.m2/repository/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar:/home/mehdi/.m2/repository/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar:/home/mehdi/.m2/repository/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar:/home/mehdi/.m2/repository/com/google/code/gson/gson/2.2.4/gson-2.2.4.jar:/home/mehdi/.m2/repository/org/apache/hadoop/hadoop-auth/2.6.5/hadoop-auth-2.6.5.jar:/home/mehdi/.m2/repository/org/apache/httpcomponents/httpclient/4.2.5/httpclient-4.2.5.jar:/home/mehdi/.m2/repository/org/apache/httpcomponents/httpcore/4.2.4/httpcore-4.2.4.jar:/home/mehdi/.m2/repository/org/apache/directory/server/apacheds-kerberos-codec/2.0

Re: error in running shared rdd example with intellij

[Mehdi Seydali]

i have run it in 4 nodes. but the same error has remained.

On Fri, Dec 28, 2018 at 5:44 PM Ilya Kasnacheev 
wrote:

> Hello!
>
> Looks like this node is started in client node, but you don't have server
> nodes available for it to connect to.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 28 дек. 2018 г. в 15:59, mehdi sey :
>
>> hi i want to run SharedRddExample in intellij IDE. but i have encounter
>> error
>> just as bellow . why?
>> /usr/lib/jvm/java-8-oracle/bin/java
>> -DIGNITE_HOME=/usr/local/apache-ignite-fabric-2.6.0-bin/
>>
>> -javaagent:/snap/intellij-idea-community/109/lib/idea_rt.jar=35933:/snap/intellij-idea-community/109/bin
>> -Dfile.encoding=UTF-8 -classpath
>>
>> /usr/lib/jvm/java-8-oracle/jre/lib/charsets.jar:/usr/lib/jvm/java-8-oracle/jre/lib/deploy.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/cldrdata.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/dnsns.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/jaccess.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/jfxrt.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/localedata.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/nashorn.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/sunec.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/sunjce_provider.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/sunpkcs11.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/zipfs.jar:/usr/lib/jvm/java-8-oracle/jre/lib/javaws.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jce.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jfr.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jfxswt.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jsse.jar:/usr/lib/jvm/java-8-oracle/jre/lib/management-agent.jar:/usr/lib/jvm/java-8-oracle/jre/lib/plugin.jar:/usr/lib/jvm/java-8-oracle/jre/lib/resources.jar:/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar:/home/mehdi/SparkIgniteRDD/out/production/SparkIgniteRDD:/home/mehdi/.m2/repository/org/apache/spark/spark-core_2.11/2.4.0/spark-core_2.11-2.4.0.jar:/home/mehdi/.m2/repository/org/apache/avro/avro/1.8.2/avro-1.8.2.jar:/home/mehdi/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar:/home/mehdi/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar:/home/mehdi/.m2/repository/com/thoughtworks/paranamer/paranamer/2.7/paranamer-2.7.jar:/home/mehdi/.m2/repository/org/apache/commons/commons-compress/1.8.1/commons-compress-1.8.1.jar:/home/mehdi/.m2/repository/org/tukaani/xz/1.5/xz-1.5.jar:/home/mehdi/.m2/repository/org/apache/avro/avro-mapred/1.8.2/avro-mapred-1.8.2.jar:/home/mehdi/.m2/repository/org/apache/avro/avro-ipc/1.8.2/avro-ipc-1.8.2.jar:/home/mehdi/.m2/repository/commons-codec/commons-codec/1.9/commons-codec-1.9.jar:/home/mehdi/.m2/repository/com/twitter/chill_2.11/0.9.3/chill_2.11-0.9.3.jar:/home/mehdi/.m2/repository/com/esotericsoftware/kryo-shaded/4.0.2/kryo-shaded-4.0.2.jar:/home/mehdi/.m2/repository/com/esotericsoftware/minlog/1.3.0/minlog-1.3.0.jar:/home/mehdi/.m2/repository/org/objenesis/objenesis/2.5.1/objenesis-2.5.1.jar:/home/mehdi/.m2/repository/com/twitter/chill-java/0.9.3/chill-java-0.9.3.jar:/home/mehdi/.m2/repository/org/apache/xbean/xbean-asm6-shaded/4.8/xbean-asm6-shaded-4.8.jar:/home/mehdi/.m2/repository/org/apache/hadoop/hadoop-client/2.6.5/hadoop-client-2.6.5.jar:/home/mehdi/.m2/repository/org/apache/hadoop/hadoop-common/2.6.5/hadoop-common-2.6.5.jar:/home/mehdi/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/home/mehdi/.m2/repository/xmlenc/xmlenc/0.52/xmlenc-0.52.jar:/home/mehdi/.m2/repository/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar:/home/mehdi/.m2/repository/commons-io/commons-io/2.4/commons-io-2.4.jar:/home/mehdi/.m2/repository/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar:/home/mehdi/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/home/mehdi/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/home/mehdi/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar:/home/mehdi/.m2/repository/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar:/home/mehdi/.m2/repository/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar:/home/mehdi/.m2/repository/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar:/home/mehdi/.m2/repository/com/google/code/gson/gson/2.2.4/gson-2.2.4.jar:/home/mehdi/.m2/repository/org/apache/hadoop/hadoop-auth/2.6.5/hadoop-auth-2.6.5.jar:/home/mehdi/.m2/repository/org/apache/httpcomponents/httpclient/4.2.5/httpclient-4.2.5.jar:/home/mehdi/.m2/repository/org/apache/httpcomponents/httpcore/4.2.4/httpcore-4.2.4.jar:/home/mehdi/.m2/repository/org/apache/directory/server/apacheds-kerberos-codec/2.0.0-M15/apacheds-kerberos-codec-2.0.0-M15.jar:/home/mehdi/.m2/repository/org/apache/directory/server/apacheds-i18n/2.0.0-M15/apacheds-i18n-2.0.0-M15.jar:/home/mehdi/.m2/repository/org/apache/directory/api/api-asn1-api/1.0.0-M20/api-asn1-api-1.0.0-M20.jar:/home/mehdi/.m2/repository/org/apache/directory/api/api-util/1.0.0

Re: error in ignite-spark

[Mehdi Seydali]

yes  i have considered intellij. how i could use it?

On Fri, Dec 28, 2018 at 2:27 PM Ilya Kasnacheev 
wrote:

> Hello!
>
> Have you considered installing IDEA? Developing Ignite under Netbeans or
> Eclipse is an expert option which is clearly not your case.
>
> Otherwise you can start with
> https://maven.apache.org/guides/introduction/introduction-to-profiles.html
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 28 дек. 2018 г. в 10:09, Mehdi Seydali :
>
>> i have searched alot  but i couldn't find enable scala profile i maven.
>> do you have any link?
>>
>> On Wed, Dec 26, 2018 at 8:34 PM Ilya Kasnacheev <
>> ilya.kasnach...@gmail.com> wrote:
>>
>>> Hello!
>>>
>>> With -Pscala command or via netbeans dialogs.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> ср, 26 дек. 2018 г. в 19:54, Mehdi Seydali :
>>>
>>>> How could i enable scala profile??
>>>>
>>>> On Wed, 26 Dec 2018, 13:44 Ilya Kasnacheev, 
>>>> wrote:
>>>>
>>>>> Hello!
>>>>>
>>>>> You have to enable 'scala' profile in Maven to be able to build Spark
>>>>> examples. However, even after I rebuild with scala profile, netbeans will
>>>>> not show them propetly. You will still be able to run them all right, I
>>>>> think.
>>>>>
>>>>> Regards,
>>>>> --
>>>>> Ilya Kasnacheev
>>>>>
>>>>>
>>>>> ср, 26 дек. 2018 г. в 12:27, mehdi sey :
>>>>>
>>>>>> hi. i want to execute a RDD example in spark from example folder of
>>>>>> ignite
>>>>>> 2.7, but i have encounter and error just like an attached picture. in
>>>>>> import
>>>>>> section you see an underlined line. i have added dependency but still
>>>>>> is
>>>>>> remained why?
>>>>>>
>>>>>> <
>>>>>> http://apache-ignite-users.70518.x6.nabble.com/file/t2160/Screenshot_from_2018-12-26_10-31-16.png>
>>>>>>
>>>>>> <
>>>>>> http://apache-ignite-users.70518.x6.nabble.com/file/t2160/Screenshot_from_2018-12-26_10-34-19.png>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>>>>
>>>>>

Re: error in ignite-spark

[Mehdi Seydali]

with those command that you said i coud run in command line. but how i
could run it in intellij or in netbeans? how i could enabled scala profile
in netbeans or intellij?

On Wed, Dec 26, 2018 at 4:04 PM Ilya Kasnacheev 
wrote:

> Hello!
>
> No you don't.
>
> You can use the following maven commands from examples directory to run
> example:
>
> mvn clean install -Pscala -DskipTests
> mvn -Pscala exec:java
> -Dexec.mainClass=org.apache.ignite.examples.spark.SharedRDDExample
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 26 дек. 2018 г. в 13:42, Mehdi Seydali :
>
>> spark example have been written with java not scala. i think this error
>> relate to spark version. i have another question. for running this example
>> is it  necessary to intsall spark before running?.
>>
>> On Wed, Dec 26, 2018 at 1:44 PM Ilya Kasnacheev <
>> ilya.kasnach...@gmail.com> wrote:
>>
>>> Hello!
>>>
>>> You have to enable 'scala' profile in Maven to be able to build Spark
>>> examples. However, even after I rebuild with scala profile, netbeans will
>>> not show them propetly. You will still be able to run them all right, I
>>> think.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> ср, 26 дек. 2018 г. в 12:27, mehdi sey :
>>>
>>>> hi. i want to execute a RDD example in spark from example folder of
>>>> ignite
>>>> 2.7, but i have encounter and error just like an attached picture. in
>>>> import
>>>> section you see an underlined line. i have added dependency but still is
>>>> remained why?
>>>>
>>>> <
>>>> http://apache-ignite-users.70518.x6.nabble.com/file/t2160/Screenshot_from_2018-12-26_10-31-16.png>
>>>>
>>>> <
>>>> http://apache-ignite-users.70518.x6.nabble.com/file/t2160/Screenshot_from_2018-12-26_10-34-19.png>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>>
>>>

Re: error in ignite-spark

[Mehdi Seydali]

i have searched alot  but i couldn't find enable scala profile i maven. do
you have any link?

On Wed, Dec 26, 2018 at 8:34 PM Ilya Kasnacheev 
wrote:

> Hello!
>
> With -Pscala command or via netbeans dialogs.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 26 дек. 2018 г. в 19:54, Mehdi Seydali :
>
>> How could i enable scala profile??
>>
>> On Wed, 26 Dec 2018, 13:44 Ilya Kasnacheev, 
>> wrote:
>>
>>> Hello!
>>>
>>> You have to enable 'scala' profile in Maven to be able to build Spark
>>> examples. However, even after I rebuild with scala profile, netbeans will
>>> not show them propetly. You will still be able to run them all right, I
>>> think.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> ср, 26 дек. 2018 г. в 12:27, mehdi sey :
>>>
>>>> hi. i want to execute a RDD example in spark from example folder of
>>>> ignite
>>>> 2.7, but i have encounter and error just like an attached picture. in
>>>> import
>>>> section you see an underlined line. i have added dependency but still is
>>>> remained why?
>>>>
>>>> <
>>>> http://apache-ignite-users.70518.x6.nabble.com/file/t2160/Screenshot_from_2018-12-26_10-31-16.png>
>>>>
>>>> <
>>>> http://apache-ignite-users.70518.x6.nabble.com/file/t2160/Screenshot_from_2018-12-26_10-34-19.png>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>>
>>>

Re: error in importing ignite 2.7 to netbeans

[Mehdi Seydali]

   
true





is it correct?

On Fri, Dec 28, 2018 at 8:41 AM Mehdi Seydali 
wrote:

> In which file i can see the profile?pom in the root folder of ignite?
>
>
> On Wednesday, December 26, 2018, aealexsandrov 
> wrote:
>
>> Please check that you set the scale profile because of without it the
>> spark
>> dependencies will not be added:
>>
>> ...
>> 
>> 
>> scala
>>
>> 
>> src/main/spark
>> 
>>
>> 
>> 
>> org.apache.ignite
>> ignite-scalar
>> 2.7.0
>> 
>>
>> 
>> org.apache.ignite
>> ignite-spark
>> 2.7.0
>> 
>> 
>> ...
>>
>> BR,
>> Andrei
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Re: error in importing ignite 2.7 to netbeans

[Mehdi Seydali]

In which file i can see the profile?pom in the root folder of ignite?


On Wednesday, December 26, 2018, aealexsandrov 
wrote:

> Please check that you set the scale profile because of without it the spark
> dependencies will not be added:
>
> ...
> 
> 
> scala
>
> 
> src/main/spark
> 
>
> 
> 
> org.apache.ignite
> ignite-scalar
> 2.7.0
> 
>
> 
> org.apache.ignite
> ignite-spark
> 2.7.0
> 
> 
> ...
>
> BR,
> Andrei
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: error in ignite-spark

[Mehdi Seydali]

How could i enable scala profile??

On Wed, 26 Dec 2018, 13:44 Ilya Kasnacheev, 
wrote:

> Hello!
>
> You have to enable 'scala' profile in Maven to be able to build Spark
> examples. However, even after I rebuild with scala profile, netbeans will
> not show them propetly. You will still be able to run them all right, I
> think.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 26 дек. 2018 г. в 12:27, mehdi sey :
>
>> hi. i want to execute a RDD example in spark from example folder of ignite
>> 2.7, but i have encounter and error just like an attached picture. in
>> import
>> section you see an underlined line. i have added dependency but still is
>> remained why?
>>
>> <
>> http://apache-ignite-users.70518.x6.nabble.com/file/t2160/Screenshot_from_2018-12-26_10-31-16.png>
>>
>> <
>> http://apache-ignite-users.70518.x6.nabble.com/file/t2160/Screenshot_from_2018-12-26_10-34-19.png>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Re: error in importing ignite 2.7 to netbeans

[Mehdi Seydali]

this is pom file in folder of example.




http://maven.apache.org/POM/4.0.0"; xmlns:xsi="
http://www.w3.org/2001/XMLSchema-instance";
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
4.0.0


org.apache.ignite
ignite-parent
1
../parent


ignite-examples
2.7.0



javax.cache
cache-api
${javax.cache.version}



org.apache.ignite
ignite-core
${project.version}



org.apache.ignite
ignite-spring
${project.version}



org.springframework
spring-core


org.springframework
spring-beans





org.apache.ignite
ignite-log4j
${project.version}



org.apache.ignite
ignite-indexing
${project.version}



org.apache.ignite
ignite-spring-data_2.0
${project.version}



org.apache.ignite
ignite-ml
${project.version}



commons-cli
commons-cli
1.2



com.google.code.simple-spring-memcached
spymemcached
2.8.4



org.apache.ignite
ignite-core
${project.version}
test-jar
test



org.gridgain
ignite-shmem
1.0.0
test



org.javassist
javassist
${javassist.version}
test





src/main/java
src/test/java
src/main/java
src/test/java




scala


src/main/spark
src/test/spark




org.apache.ignite
ignite-scalar
${project.version}



org.scalatest
scalatest_2.11
2.2.4
test


org.scala-lang
scala-library





org.apache.ignite
ignite-spark
${project.version}






net.alchim31.maven
scala-maven-plugin






lgpl


src/main/java-lgpl
src/test/java-lgpl




org.apache.ignite
ignite-hibernate_5.1
${project.version}



org.apache.ignite
ignite-schedule
${project.version}



org.apache.ignite
ignite-geospatial
${project.version}








src/main/java

**/*.java



config





org.codehaus.mojo
build-helper-maven-plugin


add-sources
generate-sources

add-source



${lgpl.folder}
${spark.folder}





add-tests
generate-test-sources

add-test-source



${lgpl.test.folder}
${spark.test.folder}







org.apache.maven.plugins
maven-deploy-plugin

true






 is anything 

Re: error in ignite-spark

[Mehdi Seydali]

spark example have been written with java not scala. i think this error
relate to spark version. i have another question. for running this example
is it  necessary to intsall spark before running?.

On Wed, Dec 26, 2018 at 1:44 PM Ilya Kasnacheev 
wrote:

> Hello!
>
> You have to enable 'scala' profile in Maven to be able to build Spark
> examples. However, even after I rebuild with scala profile, netbeans will
> not show them propetly. You will still be able to run them all right, I
> think.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 26 дек. 2018 г. в 12:27, mehdi sey :
>
>> hi. i want to execute a RDD example in spark from example folder of ignite
>> 2.7, but i have encounter and error just like an attached picture. in
>> import
>> section you see an underlined line. i have added dependency but still is
>> remained why?
>>
>> <
>> http://apache-ignite-users.70518.x6.nabble.com/file/t2160/Screenshot_from_2018-12-26_10-31-16.png>
>>
>> <
>> http://apache-ignite-users.70518.x6.nabble.com/file/t2160/Screenshot_from_2018-12-26_10-34-19.png>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Re: deep learning over apache ignite

[Mehdi Seydali]

as you said we can use hierarchical spawning job in spark. there is
remained another issue. for example if we have load data from ignite cache
and
a job on spark has been spawned. this job spawn some other jobs. this jobs
can be execute on ignite too? if this job fail how about the other job
spawned? spark must handle them or ignite?

On Sat, Dec 22, 2018 at 6:28 PM zaleslaw  wrote:

> I will try to answer these questions
> 1) We can using ignite as cache storage and preparing data for processing
> on
> dl4j?
> - yes you can, it's main case, to load data to ignite caches correctly
> 2) Job on spark can spawn hirarchy job for accelerate execution.
> - Ignite can compute piece pf code in parallel and etc, but you need to
> implement correct infrastrucure about code tasks spreading and so on over
> ignite.
> Look at our TF integration and distributing training
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: deep learning over apache ignite

[Mehdi Seydali]

My dear friend. I have a glance on tensorflow on ignite. I have the same
idea as you but only one thing is added. In addition distributing job to
local data, i will use gpu in addition to cpu core. Are you agree with
total idea?  Is there anything for debating?

On Wednesday, December 19, 2018, dmitrievanthony 
wrote:

> Yes, in TensorFlow on Apache Ignite we support distributed learning as you
> described it (please see details in  this documentation
>   ).
>
> Speaking about performance, TensorFlow supports distributed learning itself
> (please see details  here
>  distribute>
> ). But to start distributed learning in pure TensorFlow you need to setup
> cluster manually, manually distribute training data between cluster nodes
> and handle node failures.
>
> In TensorFlow on Apache Ignite we do it for you automatically. Apache
> Ignite
> plays cluster manager role, it starts and maintains TensorFlow cluster with
> optimal configuration and handles node failures. At the same time, the
> training is fully performed by TensorFlow anyway. So, the training
> performance is absolutely equal to the case when you use pure TensorFlow
> with proper manually configured and started TensorFlow cluster because we
> don't participate in the training process when the cluster is running
> properly.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: deep learning over apache ignite

[Mehdi Seydali]

I have another question? As you know dl4j execute over spark. When we want
to integrate dl4j with ignite is it meaningful? For acceleration execution
we can do below idea
1. We can using ignite as cache storage and preparing data for processing
on dl4j.
2. Job on spark can spawn hirarchy job for accelerate execution.
Do you have any comment?

On Wednesday, December 19, 2018, dmitrievanthony 
wrote:

> Yes, in TensorFlow on Apache Ignite we support distributed learning as you
> described it (please see details in  this documentation
>   ).
>
> Speaking about performance, TensorFlow supports distributed learning itself
> (please see details  here
>  distribute>
> ). But to start distributed learning in pure TensorFlow you need to setup
> cluster manually, manually distribute training data between cluster nodes
> and handle node failures.
>
> In TensorFlow on Apache Ignite we do it for you automatically. Apache
> Ignite
> plays cluster manager role, it starts and maintains TensorFlow cluster with
> optimal configuration and handles node failures. At the same time, the
> training is fully performed by TensorFlow anyway. So, the training
> performance is absolutely equal to the case when you use pure TensorFlow
> with proper manually configured and started TensorFlow cluster because we
> don't participate in the training process when the cluster is running
> properly.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: deep learning over apache ignite

[Mehdi Seydali]

My dear friend. Our essential goals are just as below:
1. Distributed training for training model
2. Utilize Gpu in addition to Cpu for accelerate execution.
 In tensorflow do you implement two above mentioned? How much speed up do
you gain relative to pure tensorflow?

On Tuesday, December 18, 2018, dmitrievanthony 
wrote:

> First of all, when you are talking about "acceleration of running deep
> neural
> network" what do you mean, training or inference?
> If you are talking about inference, it actually doesn't matter what cluster
> management system we use, we need only to run parallel/distributed
> inference.
> If you are talking about training, how are you going to speed it up? Use
> distributed training on multiple machines? Or GPU's? Or maybe you just want
> to speed up data preprocessing?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: deep learning over apache ignite

[Mehdi Seydali]

Do you have any recommendation about this idea?is it possible develope this
idea and compare with tensoflow over ignite ?

On Tuesday, December 18, 2018, dmitrievanthony 
wrote:

> Hello,
>
> It's a good idea, machine learning in Apache Ignite is actively developing
> right now and new features are always appreciated.
>
> At the same time, let me point that deep learning is already supported in
> Apache Ignite via integration with TensorFlow as the most popular deep
> learning framework. This integrations allows to: use data stored in Apache
> Ignite for training, use Apache Ignite as a model storage for training and
> use Apache Ignite as a cluster manager for TensorFlow (all these features
> together allow to perform efficient distributed training where it's
> required).
>
> For more information please see the following resources:
> Apache Ignite [Deep Learning With TensorFlow]
> 
> TensorFlow [Apache Ignite Contrib Readme]
>  ignite>
>
> If you are interested in integration with DL4J anyway, feel free to share
> your ideas and we'll try to help you to implement it.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite client node connect to visor

[Mehdi Seydali]

No,the visor is not work just as server or client. It is only cluster
monitoring.

On Tuesday, December 18, 2018, ashishb008 
wrote:

> Hello,
>
> Suppose we have a active cluster of 5 nodes, and some partitioned caches on
> the cluster. And if I started the visor, will it add an extra node (server)
> to existing cluster?
>
> Thanks
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: deep learning over apache ignite

[Mehdi Seydali]

Our,s goal are two things:
1. Acceleration of running deep neural network on two framework
simultaneously ( ignite and spark)
2. Study that developing deep model with dl4j over ignite has better
performance and speed up relared to dl4j?



On Tuesday, December 18, 2018, zaleslaw  wrote:

> No, currently we doesn't work on integration bw DL4j and Ignite
>
> The first step can be getting data from IgniteCache and wrapping to tensor
> and putting to local instance of DL4j. But currently I could help with code
> review only not with the example building.
>
> First of all, we should understand what is the goal of integration? Should
> it be simple channel to get data from Ignite without saving to file or it
> should be more closest integration
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: deep learning over apache ignite

[Mehdi Seydali]

I am in first step and i have not succeeded to integrate dl4j with apache
ignite. Is it possible for you how could you integrate them?

On Tuesday, December 18, 2018, Mehdi Seydali 
wrote:

> In a nutshell you want to say that this is a not good idea and may not
> reach a good performancs? You have integrated dl4j with apache ignite too?
>
> On Tuesday, December 18, 2018, zaleslaw  wrote:
>
>> Hi, I'm a contributor of Ignite ML too and I'd like dl4j library too and
>> have
>> experience with that.
>> I have a few experiments with integration but the main problem is tensor
>> algebra implementation and effective Ignite cluster/caches usage.
>>
>> Could you provide a simple example of your draft ideas on github and I'll
>> be
>> happy to help with review
>>
>> Feel free to post the link here in this thread
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Re: deep learning over apache ignite

[Mehdi Seydali]

In a nutshell you want to say that this is a not good idea and may not
reach a good performancs? You have integrated dl4j with apache ignite too?

On Tuesday, December 18, 2018, zaleslaw  wrote:

> Hi, I'm a contributor of Ignite ML too and I'd like dl4j library too and
> have
> experience with that.
> I have a few experiments with integration but the main problem is tensor
> algebra implementation and effective Ignite cluster/caches usage.
>
> Could you provide a simple example of your draft ideas on github and I'll
> be
> happy to help with review
>
> Feel free to post the link here in this thread
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>