Creating Dataset classNotFound exception

[mahesh76private]

Hi 

I have a very simple program to create a DataSet to see if we can migrate to
Ignite ML.
Strangely on the server logs, why would I get a CreateMLDataSet Class Not
found exception?


public class CreateMLDataSet {

  public static void main(String[] args) throws Exception {

CacheConfiguration cacheCfg = new CacheConfiguration(tableName);
IgniteCache tableCache =
ignite.getOrCreateCache(cacheCfg).withKeepBinary();

IgniteBiFunction featureExtractor =
new IgniteBiFunction() {
  @Override
  public Vector apply(BinaryObject key, BinaryObject value) {
Integer val  = (Integer)value.field("quantity");
Vector vector = VectorUtils.of(val);
return vector;
  }
};

SimpleDataset dataSet = null;
try {
  dataSet = DatasetFactory.createSimpleDataset(ignite, tableCache,
featureExtractor);
} catch (Exception e) {
  e.printStackTrace();
}


// Calculation of the mean value.
double[] mean = dataSet.mean();
System.out.println("Mean \n\t" + Arrays.toString(mean));

// Calculation of the standard deviation.
double[] std = dataSet.std();
System.out.println("Standard deviation \n\t" + Arrays.toString(std));

// Calculation of the covariance matrix.
double[][] cov = dataSet.cov();
System.out.println("Covariance matrix ");
for (double[] row : cov)
  System.out.println("\t" + Arrays.toString(row));

// Calculation of the correlation matrix.
double[][] corr = dataSet.corr();
System.out.println("Correlation matrix ");
for (double[] row : corr) System.out.println("\t" +
Arrays.toString(row));
  }
}




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Consistency

[Cindy Xing]

In the ignite doc
(https://apacheignite.readme.io/docs/primary-and-backup-copies), it
mentioned that the cache will be immediate consistent independent of the
sync mode chosen. Can someone help explain how?/Cache Data ConsistencyNote
that regardless of write synchronization mode, cache data will always remain
fully consistent across all participating nodes when using
transactions./ThanksCindy



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: MVCC and continuous query

[Cindy Xing]

My goal is to enable my app taking action upon changes in the ignite data
store. 
New to ignite and not exactly know the ignite API available. 

Cindy



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

RE: building and running from source

[Scott Cote]

Success maybe?

I forgot to tell maven to skip tests , so I threw in that switch to get a 
snapshot.

Probing deeper – it seems this is what I needed -the source matches the 
binaries and now I understand how to build the web console  (? Lets see if I 
get through this without crying for yelp again ??    )

Thanks for looking and telepathically sending me in the right direction (hope 
my humor is appreciated).   Still – if you have any guidance on doing the 
snapshot builds – all ears.

Cheers,

SCott

[INFO] Executing tasks

main:
  [zip] Building zip: 
C:\cygwin64\home\scote\ignite\target\bin\apache-ignite-2.8.0-SNAPSHOT-bin.zip
[INFO] Executed tasks
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ apache-ignite 
---
[INFO]
[INFO] --- flatten-maven-plugin:1.0.1:flatten (flatten) @ apache-ignite ---
[INFO] Generating flattened POM of project 
org.apache.ignite:apache-ignite:pom:2.8.0-SNAPSHOT...
[WARNING] FlattenMode minimum is deprecated!
[WARNING] Ignoring multiple XML header comment!
[INFO]
[INFO] --- maven-dependency-plugin:2.8:copy-dependencies (copy-libs) @ 
apache-ignite ---
[INFO]
[INFO] --- maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @ 
apache-ignite ---
[INFO]
[INFO] --- maven-source-plugin:2.2.1:jar-no-fork (source) @ apache-ignite ---
[INFO]
[INFO] --- maven-javadoc-plugin:2.10.1:jar (module-javadoc) @ apache-ignite ---
[INFO] Not executing Javadoc as the project is not a Java classpath-capable 
package
[INFO] 
[INFO] Reactor Summary for apache-ignite 2.8.0-SNAPSHOT:
[INFO]
[INFO] ignite-apache-license-gen .. SUCCESS [  1.467 s]
[INFO] ignite-tools ... SUCCESS [  4.726 s]
[INFO] ignite-core  SUCCESS [03:08 min]
[INFO] ignite-compress  SUCCESS [  2.163 s]
[INFO] ignite-indexing  SUCCESS [ 21.963 s]
[INFO] ignite-dev-utils ... SUCCESS [  5.840 s]
[INFO] ignite-direct-io ... SUCCESS [  2.395 s]
[INFO] ignite-log4j ... SUCCESS [  2.527 s]
[INFO] ignite-urideploy ... SUCCESS [  4.088 s]
[INFO] ignite-spring .. SUCCESS [  4.534 s]
[INFO] ignite-hadoop .. SUCCESS [ 20.785 s]
[INFO] ignite-extdata-p2p . SUCCESS [  5.059 s]
[INFO] ignite-extdata-uri-dep . SUCCESS [  1.289 s]
[INFO] ignite-extdata-uri . SUCCESS [  7.147 s]
[INFO] ignite-extdata-platform  SUCCESS [  0.393 s]
[INFO] ignite-rest-http ... SUCCESS [  2.956 s]
[INFO] ignite-clients . SUCCESS [  4.582 s]
[INFO] ignite-spring-data . SUCCESS [  3.083 s]
[INFO] ignite-spring-data_2.0 . SUCCESS [  2.902 s]
[INFO] ignite-web . SUCCESS [  2.269 s]
[INFO] ignite-aop . SUCCESS [  2.937 s]
[INFO] ignite-ssh . SUCCESS [  1.659 s]
[INFO] ignite-jta . SUCCESS [  3.922 s]
[INFO] ignite-aws . SUCCESS [  4.271 s]
[INFO] ignite-log4j2 .. SUCCESS [  2.574 s]
[INFO] ignite-slf4j ... SUCCESS [  1.987 s]
[INFO] ignite-jcl . SUCCESS [  2.063 s]
[INFO] ignite-codegen . SUCCESS [  1.682 s]
[INFO] ignite-gce . SUCCESS [  2.239 s]
[INFO] ignite-cloud ... SUCCESS [  4.905 s]
[INFO] ignite-mesos ... SUCCESS [  3.837 s]
[INFO] ignite-kafka ... SUCCESS [  4.915 s]
[INFO] ignite-flume ... SUCCESS [  2.703 s]
[INFO] ignite-yarn  SUCCESS [  6.842 s]
[INFO] ignite-jms11 ... SUCCESS [  2.244 s]
[INFO] ignite-twitter . SUCCESS [  2.425 s]
[INFO] ignite-mqtt  SUCCESS [  3.896 s]
[INFO] ignite-zookeeper ... SUCCESS [  5.786 s]
[INFO] ignite-hibernate-core .. SUCCESS [  2.811 s]
[INFO] ignite-camel ... SUCCESS [  2.742 s]
[INFO] ignite-storm ... SUCCESS [  2.348 s]
[INFO] ignite-osgi-paxlogging . SUCCESS [  0.181 s]
[INFO] 

RE: building and running from source

[Scott Cote]

Am trying out :
mvn package -P release

this builds a snapshot?

SCott

From: Scott Cote 
Sent: Thursday, January 17, 2019 4:32 PM
To: user@ignite.apache.org
Subject: building and running from source

Hello,

I want to build a copy of ignite with the web console.  Cannot - as far as I 
can tell - download a binary for the web console (don't want to use the 
gridgain cloud version).  The instructions for the web console state that one 
must build it from source.   To that point, the binary distribution of 2.7 does 
NOT have a modules folder.


I have downloaded the source (git clone https://github.com/apache/ignite) so 
that I can build the web console, but the outcome of executing:

Mvn clean package -dskipTests

Does not leave me with a distribution anything like as if I downloaded the 
"bin" .   For example, there are modules, but no lib folders.

If I issue the command ignite.bat, I observe errors on my pc such as

c:\cygwin64\home\scote\ignite>bin\ignite.bat
The input line is too long.
"C:\cygwin64\home\scote\ignite\modules\kafka\target\classes" was unexpected at 
this time.

What is the maven command to build a copy of ignite from the command line using 
maven and then be able to run the ignite.sh command  ?


Thanks in advance.

SCott

building and running from source

[Scott Cote]

Hello,

I want to build a copy of ignite with the web console.  Cannot - as far as I 
can tell - download a binary for the web console (don't want to use the 
gridgain cloud version).  The instructions for the web console state that one 
must build it from source.   To that point, the binary distribution of 2.7 does 
NOT have a modules folder.


I have downloaded the source (git clone https://github.com/apache/ignite) so 
that I can build the web console, but the outcome of executing:

Mvn clean package -dskipTests

Does not leave me with a distribution anything like as if I downloaded the 
"bin" .   For example, there are modules, but no lib folders.

If I issue the command ignite.bat, I observe errors on my pc such as

c:\cygwin64\home\scote\ignite>bin\ignite.bat
The input line is too long.
"C:\cygwin64\home\scote\ignite\modules\kafka\target\classes" was unexpected at 
this time.

What is the maven command to build a copy of ignite from the command line using 
maven and then be able to run the ignite.sh command  ?


Thanks in advance.

SCott

Re: ignite git repo

[Dmitriy Govorukhin]

This link work for me, https://gitbox.apache.org/repos/asf/ignite

чт, 17 янв. 2019 г., 20:03 Scott Cote sc...@etcc.com:

> Petr,
>
>
>
> Thank you for your quick reply.
>
>
>
> git clone https://gitbox.apache.org/repos/ignite.git
>
>
>
> 404 on that one ….
>
>
>
> “[image: git] projects
>  /
>
>
>
> 404 - No projects found
>
> OPML  TXT
> 
>
> “
>
>
>
> I just realized that this is a security outcome ….   Looking at
> gitbox.apache.com, I need an account there.   Can I just clone (for now)
> from a github repo?
>
>
>
> Issuing the command:
>
>
>
> git clone https://github.com/apache/ignite
>
>
>
> yielded what seems like what I want ….
>
>
>
> If I want to give back ….  How do I go about getting (gitting  ) an
> apache id?   I might already have one from some work with Apache Metron
> (not sure). …
>
>
>
> For Apache Metron - was investigating the use of ignite to replace the
> lucene decendents of hot data repos …
>
>
>
> For now – I’m building off of the mirror apache github site ….
>
>
>
> Cheers
>
>
>
> SCott
>
>
>
> *From:* Petr Ivanov 
> *Sent:* Thursday, January 17, 2019 10:18 AM
> *To:* user@ignite.apache.org
> *Subject:* Re: ignite git repo
>
>
>
> Repository is move to GitBox
>
> Clone command: git clone http://gitbox.apache.org/repos/ignite.git
>
>
>
>
>
>
>
> On 17 Jan 2019, at 19:17, Scott Cote  wrote:
>
>
>
> Hello Igniters:
>
>
>
> I’m having trouble cloning the git repo.  Want to build from source and
> maybe make an add on ….
>
>
>
> Went to the link:  https://ignite.apache.org/download.cgi#git-repository
> to get the “git” url/command
>
> Am issuing the command:   git clone
> https://git-wip-us.apache.org/repos/asf/ignite
>
>
>
> But am getting (gitting???):
>
>
>
> $ git clone https://git-wip-us.apache.org/repos/asf/ignite
>
> Cloning into 'ignite'...
>
> fatal: repository 'https://git-wip-us.apache.org/repos/asf/ignite/' not
> found
>
>
>
> Investigating the url and I’m redirected to
>
>
>
> https://gitbox.apache.org/repos/asf?p=ignite.git
>
>
>
> So how do I clone?  The instructions are off ….
>
>
>
> Thanks in advance.
>
>
>
> SCott
>
>
>

Re: Concurrent merge into operations cause critical system error on ignite 2.7 node.

[Ilya Kasnacheev]

Hello!

Thank you for the reproducer!

Unfortunately, it seems that this behavior is expected. Much like with
putAll, you risk deadlock if you do MERGE INTO with unsorted sets of keys.
Meaning you should always pre-sort keys that you pass to MERGE INTO.

Moreover, I have discussed it with people behind Ignite SQL support, they
say this is an implementation detail that you should not rely upon. MERGE
has no guarantees to not reorder its rows (however it does seem to not
reorder them currently) so the best course is to avoid doing more than one
multi value MERGE at the moment.

Better use JDBC batching for that.

Regards,
-- 
Ilya Kasnacheev


чт, 17 янв. 2019 г. в 04:50, yangjiajun <1371549...@qq.com>:

> Hello.
>
> Here is my test code:
>
>
> import java.sql.Connection;
> import java.sql.DriverManager;
> import java.sql.PreparedStatement;
> import java.sql.SQLException;
> import java.sql.Statement;
> import java.util.Properties;
>
> /**
>  * concurrent merge into
>  * */
> public class MergeInto {
>
> private final static String mergeSql1 = "merge INTO
> city2(id,name,name1)
> VALUES(1,'1','1'),(2,'1','1'),(3,'1','1')";
> private final static String mergeSql2 = "merge INTO
> city2(id,name,name1)
> VALUES(2,'1','1'),(1,'1','1')";
>
> private static Connection conn;
> private static Connection conn1;
>
> public static void main(String[] args) throws Exception {
>
> initialize(false);
>
> testQuery();
>
> while(true){
>
> }
> }
>
> public static void close() throws Exception {
> conn.close();
> }
>
> public static void initialize(boolean initData) throws Exception {
> Class.forName("org.apache.ignite.IgniteJdbcThinDriver");
> final String dbUrl =
>
> "jdbc:ignite:thin://ip:port;lazy=true;skipReducerOnUpdate=true;replicatedOnly=true";
> final Properties props = new Properties();
> conn = DriverManager.getConnection(dbUrl, props);
> conn1= DriverManager.getConnection(dbUrl, props);
> if(initData){
> initData();
> }
> }
>
> private static void initData() throws SQLException{
>
> long start=System.currentTimeMillis();
> conn.prepareStatement("set streaming on").execute();
>
> String sql="insert INTO  city2(id,name,name1)
> VALUES(?,?,?)";
> PreparedStatement ps=conn.prepareStatement(sql);
> for(int i=0;i<160;i++){
> String s1=String.valueOf(Math.random());
> String s2=String.valueOf(Math.random());
> ps.setInt(1, i);
> ps.setString(2, s1);
> ps.setString(3, s2);
> ps.execute();
> }
> conn.prepareStatement("set streaming off").execute();
> long end=System.currentTimeMillis();
> System.out.println(end-start);
> }
>
> public static void testQuery() throws Exception {
>
> new Thread(new Runnable() {
>
> @Override
> public void run() {
>
> while(true){
>
> long
> startTime=System.currentTimeMillis();
> try (Statement stmt =
> conn.createStatement()) {
>
> stmt.executeUpdate(mergeSql1);
> } catch (SQLException e1) {
> e1.printStackTrace();
> }
>
>
> System.out.println("conn:"+(System.currentTimeMillis()-startTime));
> }
>
> }
> }).start();
>
> new Thread(new Runnable() {
>
> @Override
> public void run() {
>
> while(true){
> long
> startTime=System.currentTimeMillis();
>
> try (Statement stmt =
> conn1.createStatement()) {
>
> stmt.executeUpdate(mergeSql2);
> } catch (SQLException e1) {
> e1.printStackTrace();
> }
>
>
> System.out.println("conn1:"+(System.currentTimeMillis()-startTime));
> }
> }
> }).start();
>
> }
> }
>
>
> ilya.kasnacheev wrote
> > Hello!
> >
> > Can you provide a reproducer project which would reliably show this
> > behavior?
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > ср, 16 янв. 2019 г. в 15:37, yangjiajun <
>
> > 1371549332@
>
> >>:
> >
> 

RE: ignite git repo

[Scott Cote]

Petr,

Thank you for your quick reply.

git clone https://gitbox.apache.org/repos/ignite.git

404 on that one ….

“[git]projects 
/


404 - No projects found
OPML 
TXT
“

I just realized that this is a security outcome ….   Looking at 
gitbox.apache.com, I need an account there.   Can I just clone (for now) from a 
github repo?

Issuing the command:

git clone https://github.com/apache/ignite

yielded what seems like what I want ….

If I want to give back ….  How do I go about getting (gitting  ) an apache id? 
  I might already have one from some work with Apache Metron (not sure). …

For Apache Metron - was investigating the use of ignite to replace the lucene 
decendents of hot data repos …

For now – I’m building off of the mirror apache github site ….

Cheers

SCott

From: Petr Ivanov 
Sent: Thursday, January 17, 2019 10:18 AM
To: user@ignite.apache.org
Subject: Re: ignite git repo

Repository is move to GitBox
Clone command: git clone http://gitbox.apache.org/repos/ignite.git




On 17 Jan 2019, at 19:17, Scott Cote mailto:sc...@etcc.com>> 
wrote:

Hello Igniters:

I’m having trouble cloning the git repo.  Want to build from source and maybe 
make an add on ….

Went to the link:  https://ignite.apache.org/download.cgi#git-repository to 
get the “git” url/command
Am issuing the command:   git clone 
https://git-wip-us.apache.org/repos/asf/ignite

But am getting (gitting???):

$ git clone https://git-wip-us.apache.org/repos/asf/ignite
Cloning into 'ignite'...
fatal: repository 'https://git-wip-us.apache.org/repos/asf/ignite/' not found

Investigating the url and I’m redirected to

https://gitbox.apache.org/repos/asf?p=ignite.git

So how do I clone?  The instructions are off ….

Thanks in advance.

SCott

Re: ignite git repo

[Petr Ivanov]

Repository is move to GitBox
Clone command: git clone http://gitbox.apache.org/repos/ignite.git



> On 17 Jan 2019, at 19:17, Scott Cote  wrote:
> 
> Hello Igniters:
>  
> I’m having trouble cloning the git repo.  Want to build from source and maybe 
> make an add on ….
>  
> Went to the link:  https://ignite.apache.org/download.cgi#git-repository 
>  to get the “git” 
> url/command
> Am issuing the command:   git clone 
> https://git-wip-us.apache.org/repos/asf/ignite 
> 
>  
> But am getting (gitting???):
>  
> $ git clone https://git-wip-us.apache.org/repos/asf/ignite 
> 
> Cloning into 'ignite'...
> fatal: repository 'https://git-wip-us.apache.org/repos/asf/ignite/ 
> ' not found
>  
> Investigating the url and I’m redirected to 
>  
> https://gitbox.apache.org/repos/asf?p=ignite.git 
> 
>  
> So how do I clone?  The instructions are off ….
>  
> Thanks in advance.
>  
> SCott

ignite git repo

[Scott Cote]

Hello Igniters:

I'm having trouble cloning the git repo.  Want to build from source and maybe 
make an add on 

Went to the link:  https://ignite.apache.org/download.cgi#git-repository to 
get the "git" url/command
Am issuing the command:   git clone 
https://git-wip-us.apache.org/repos/asf/ignite

But am getting (gitting???):

$ git clone https://git-wip-us.apache.org/repos/asf/ignite
Cloning into 'ignite'...
fatal: repository 'https://git-wip-us.apache.org/repos/asf/ignite/' not found

Investigating the url and I'm redirected to

https://gitbox.apache.org/repos/asf?p=ignite.git

So how do I clone?  The instructions are off 

Thanks in advance.

SCott

Ignite in GCP cloud

[nitin dharrao]

Hi 
I am new to Apache Ignite. Need some guidance so that I can proceed with the
right approach. 
We want to use Ignite Data Grid (memory cache) and want to deploy Apache
Ignite in Google Cloud Platform within Kubernetes setting. 

1. We are not allowed to access the image hosted on the public domain. I
will need to create it and host it on our internal private domain. I don't
have an option to download and rehost it on our side. How do I create the
apache ignite node image? 

2. I would set up two instances of Ignite, I understood that Cloud storage
based IP discovery
(https://apacheignite.readme.io/v1.3/docs/gce-configuration) is the only
option for dynamic IPs. Please let me know if I am on the right track.

3. In order to connect to kubernetes ignite, my thin java client can use
Ignition.setClientMode(true). This should be sufficient to manage the cache
in the Kubernetes nodes. 

Please let me know if you need any additional information. Thanks!

Bests,




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: NearCache

[Mikhail]

Hi Greg,

you can try to use OptimizedMarshaller:

igniteConfig.setMarshaller(new OptimizedMarshaller())

however, this will lead to some limitations, which you can find in java doc
of setMarshaller method.

Thanks,
Mike.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Data streamer has been closed.

[yangjiajun]

Hello.

Please try this loop:
for(int i=0;i<10;i++){
System.out.println(i);
initData(); 
overwriteData();
}

And here is my config file:
example-default12.xml

  

It is harder to reproduce this issue in 2.7 than 2.6.But the exception does
appear.


ilya.kasnacheev wrote
> Hello!
> 
> I have ran your code several times and not encountered this problem. Can
> you make a new reproducer which will show it (semi)reliably?
> 
> Regards,
> -- 
> Ilya Kasnacheev
> 
> 
> чт, 17 янв. 2019 г. в 05:18, yangjiajun <

> 1371549332@

>>:
> 
>> Hello.
>>
>> Thanks for reply.Unfortunately,I still get the exception after running my
>> test on 2.7  for several times.
>>
>>
>> ilya.kasnacheev wrote
>> > Hello!
>> >
>> > I can reproduce this problem, but then again, it does not seem to
>> > reproduce
>> > on 2.7. Have you considered upgrading?
>> >
>> > Regards,
>> > --
>> > Ilya Kasnacheev
>> >
>> >
>> > ср, 16 янв. 2019 г. в 14:14, yangjiajun <
>>
>> > 1371549332@
>>
>> >>:
>> >
>> >> Hello.
>> >>
>> >> I do  test on a ignite 2.6 node with persistence enabled and get an
>> >> exception:
>> >>
>> >>  Exception in thread "main" java.sql.BatchUpdateException: class
>> >> org.apache.ignite.IgniteCheckedException: Data streamer has been
>> closed.
>> >> at
>> >>
>> >>
>> org.apache.ignite.internal.jdbc.thin.JdbcThinConnection$StreamState.readResponses(JdbcThinConnection.java:1017)
>> >> at java.lang.Thread.run(Unknown Source)
>> >>
>> >> Here is my test code:
>> >>
>> >> import java.sql.Connection;
>> >> import java.sql.DriverManager;
>> >> import java.sql.PreparedStatement;
>> >> import java.sql.SQLException;
>> >> import java.util.Properties;
>> >>
>> >> /**
>> >>  * test insert data in streaming mode
>> >>  * */
>> >> public class InsertStreamingMode {
>> >>
>> >> private static Connection conn;
>> >>
>> >> public static void main(String[] args) throws Exception {
>> >>
>> >> initialize();
>> >>
>> >> close();
>> >> }
>> >>
>> >> public static void close() throws Exception {
>> >> conn.close();
>> >> }
>> >>
>> >> public static void initialize() throws Exception {
>> >>
>> Class.forName("org.apache.ignite.IgniteJdbcThinDriver");
>> >> final String dbUrl =
>> >>
>> >>
>> "jdbc:ignite:thin://ip:port;lazy=true;skipReducerOnUpdate=true;replicatedOnly=true";
>> >> final Properties props = new Properties();
>> >> conn = DriverManager.getConnection(dbUrl, props);
>> >> initData();
>> >> overwriteData();
>> >> }
>> >>
>> >> private static void initData() throws SQLException{
>> >>
>> >> long start=System.currentTimeMillis();
>> >> conn.prepareStatement("SET STREAMING ON
>> ALLOW_OVERWRITE
>> >> ON").execute();
>> >>
>> >> String sql="insert INTO  city1(id,name,name1)
>> >> VALUES(?,?,?)";
>> >> PreparedStatement ps=conn.prepareStatement(sql);
>> >> for(int i=0;i<160;i++){
>> >> String s1=String.valueOf(Math.random());
>> >> String s2=String.valueOf(Math.random());
>> >> ps.setInt(1, i);
>> >> ps.setString(2, s1);
>> >> ps.setString(3, s2);
>> >> ps.execute();
>> >> }
>> >> conn.prepareStatement("set streaming off").execute();
>> >> long end=System.currentTimeMillis();
>> >> System.out.println(end-start);
>> >> }
>> >>
>> >> private static void overwriteData() throws SQLException{
>> >>
>> >> long start=System.currentTimeMillis();
>> >> conn.prepareStatement("SET STREAMING ON
>> ALLOW_OVERWRITE
>> >> ON").execute();
>> >>
>> >> String sql="insert INTO  city1(id,name,name1)
>> >> VALUES(?,?,?)";
>> >> PreparedStatement ps=conn.prepareStatement(sql);
>> >> for(int i=0;i<160;i++){
>> >> String s1="test";
>> >> String s2="test";
>> >> ps.setInt(1, i);
>> >> ps.setString(2, s1);
>> >> ps.setString(3, s2);
>> >> ps.execute();
>> >> }
>> >> conn.prepareStatement("set streaming off").execute();
>> >> long end=System.currentTimeMillis();
>> >> System.out.println(end-start);
>> >> }
>> >> }
>> >>
>> >> Here is the table:
>> >> CREATE TABLE city1(id LONG PRIMARY KEY, name VARCHAR,name1 VARCHAR)
>> WITH
>> >> "template=replicated"
>> >>
>> >> The exception occurs on overwriteData 

Re: MVCC and continuous query

[Ilya Kasnacheev]

Hello!

It's hard to say, maybe you need IgniteQueue if you want to have a pure
pub/sub?

Otherwise it's not obvious why you aren't content with Continuous Query
here.

Please clarify what do you mean by 'client API'. Ignite has a lot of APIs
and it's not obvious which is the one you mean here.

Regards,
-- 
Ilya Kasnacheev


чт, 17 янв. 2019 г. в 03:13, Cindy Xing :

> Thanks Ilya.
>
> Do we have example show case the pub/sub programming model with MVCC
> enabled?
> Basically instead of pulling but react in case of update on specific key.
>
> Also, does ignite client API expose the version info.?
>
>
> Cindy
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Iterating through native persistence entries before joining the cluster

[Ilya Kasnacheev]

Hello!

I guess you could be starting a service instance on every node, doing
warmup in init().

Then service requests will be only routed on nodes where warmup is complete.

Regards,
-- 
Ilya Kasnacheev


чт, 17 янв. 2019 г. в 13:26, Lukas Polacek :

> Technically, I could delay joining the cluster by adding Thread.sleep()
> into the BEFORE_NODE_START lifecycle bean (or my newly
> created BEFORE_CLUSTER_JOIN). I'd like to iterate through the local cache
> in one of those lifecycle beans - but that doesn't seem possible.
>
> Regarding cluster groups: I can't use IgniteCluster::forAttributes, since
> attributes are set at the node start. The only remaining option seems
> IgniteCluster::forPredicate, but that's also not very helpful, since I
> don't see how to leverage ClusterNode's properties, in particular its
> metrics.
>
> We have been using IgniteSet as mentioned above to produce an "active
> cluster group" but I'm looking for something simpler. Today I found a very
> problematic case which will require yet another special logic to handle it.
>
> On Fri, Jan 11, 2019 at 4:14 PM Ilya Kasnacheev 
> wrote:
>
>> Hello!
>>
>> Actually, delaying joining the cluster is complex enough since it will
>> add a lot of unexpected concerns.
>>
>> There are ClusterGroup's precisely for that purpose.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> пт, 11 янв. 2019 г. в 15:58, Lukas Polacek > >:
>>
>>> Hi,
>>> we already do that by using an IgniteSet which contains consistent IDs
>>> of the nodes that are ready, but it's very error-prone and unnecessarily
>>> complex. Delaying joining the cluster would make everything simpler.
>>>
>>> On Thu, Jan 10, 2019 at 1:38 PM Ilya Kasnacheev <
>>> ilya.kasnach...@gmail.com> wrote:
>>>
 Hello!

 I think you should decouple workload readiness from joining the cluster.

 You should let node join cluster first, then iterate entries, and only
 then allow workload to this node.

 Regards,
 --
 Ilya Kasnacheev


 чт, 3 янв. 2019 г. в 13:18, Lukas Polacek <
 lukas.pola...@innovatrics.com>:

> Hi,
> in our use case we need to run some C++ code (via JNI) whenever
> something is pushed into the local Ignite cache. In other words, we need 
> to
> have Ignite in sync with C++ memory. We have a local listener that listens
> to EVT_CACHE_OBJECT_PUT events and executes the C++ code, so everything is
> fine while the node is running. However, we use native persistence, so
> after a node restart, the local cache is read from the disk but the C++
> code hasn't been run for any cache entries, which means that Ignite and 
> C++
> memory are out of sync.
>
> Iterating through the local cache entries is only possible once the
> node has already joined the cluster, but that's too late for us - it needs
> to be done before joining the cluster.
>
> I've managed to add a lifecycle bean event BEFORE_CLUSTER_JOIN (see
> http://apache-ignite-users.70518.x6.nabble.com/Register-listeners-before-joining-the-cluster-tc25944.html,
> a PR is hopefully coming soon), which is triggered before joining the
> cluster but at that point we cannot access the cache via
> ignite.cache(...). Is there a way to access all entries in the native
> persistence at that point or earlier? I'm also fine with modifying the
> Ignite source code if that's necessary (and simple enough), since we are
> just prototyping.
>

Re: NearCache

[Ilya Kasnacheev]

Hello!

You can have onHeap cache
https://apacheignite.readme.io/docs/memory-configuration#section-on-heap-caching

setting copyOnRead to false will let you keep unmarshalled values in it. Be
sure to avoid modifying them in your code since that's not tracked.

Regards,
-- 
Ilya Kasnacheev


ср, 16 янв. 2019 г. в 18:03, Grégory Jevardat de Fombelle <
gregory.jevar...@unige.ch>:

> Hello
>
> Is there any option to have  a nearCache in Ignite that store
> unMarschalled values instead of serialized ones. I ask this for performance
> reasons.
> I noticed that for big cached objects, default Java unserialization is
> quite expensive, like ~ 2 secs for a complex > 100MB object.  So in the end
> caching this kind of objects in a near cache is not really interesting
> given the penalty of the serialisation.
> On the opposite caching a reference of this object in a custom applicative
> cache like ThreadLocal map of even just a Hashmap, the retrieval is almost
> instantaneous (order of 10-100 nano secs)
>
> Note that I did not tested binary serialisation or storing Json serialized
> objects or other custom serialisation libraries like protobuf or kryo. Also
> I'am not sure if it is possible to use custom libraries for serialisation
> of huge and complex graph of objects.
> Are there any options or architectural recommandations and some benchmarks
> on this topic ?

RE: server always shutdown

[Stanislav Lukyanov]

Hi,

You have to provide the details – configs, usage scenarios, error messages, 
logs.
Without that no one will be able to tell what’s wrong with your cluster.

Stan

From: hulitao198758
Sent: 17 января 2019 г. 8:42
To: user@ignite.apache.org
Subject: server always shutdown

Ignite. Sh start multi-node server side is very unstable, often appear process 
termination situation, I now is running on a server three cluster nodes, three 
servers make up three cluster environment, often a server node will hang, I do 
not know what situation 

Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Unsubscribe

[Dolinkov, Sergey]

Unsubscribe

[Viraj Rathod]

-- 
Regards,
Viraj Rathod

Re: Iterating through native persistence entries before joining the cluster

[Lukas Polacek]

Technically, I could delay joining the cluster by adding Thread.sleep()
into the BEFORE_NODE_START lifecycle bean (or my newly
created BEFORE_CLUSTER_JOIN). I'd like to iterate through the local cache
in one of those lifecycle beans - but that doesn't seem possible.

Regarding cluster groups: I can't use IgniteCluster::forAttributes, since
attributes are set at the node start. The only remaining option seems
IgniteCluster::forPredicate, but that's also not very helpful, since I
don't see how to leverage ClusterNode's properties, in particular its
metrics.

We have been using IgniteSet as mentioned above to produce an "active
cluster group" but I'm looking for something simpler. Today I found a very
problematic case which will require yet another special logic to handle it.

On Fri, Jan 11, 2019 at 4:14 PM Ilya Kasnacheev 
wrote:

> Hello!
>
> Actually, delaying joining the cluster is complex enough since it will add
> a lot of unexpected concerns.
>
> There are ClusterGroup's precisely for that purpose.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 11 янв. 2019 г. в 15:58, Lukas Polacek  >:
>
>> Hi,
>> we already do that by using an IgniteSet which contains consistent IDs of
>> the nodes that are ready, but it's very error-prone and unnecessarily
>> complex. Delaying joining the cluster would make everything simpler.
>>
>> On Thu, Jan 10, 2019 at 1:38 PM Ilya Kasnacheev <
>> ilya.kasnach...@gmail.com> wrote:
>>
>>> Hello!
>>>
>>> I think you should decouple workload readiness from joining the cluster.
>>>
>>> You should let node join cluster first, then iterate entries, and only
>>> then allow workload to this node.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> чт, 3 янв. 2019 г. в 13:18, Lukas Polacek >> >:
>>>
 Hi,
 in our use case we need to run some C++ code (via JNI) whenever
 something is pushed into the local Ignite cache. In other words, we need to
 have Ignite in sync with C++ memory. We have a local listener that listens
 to EVT_CACHE_OBJECT_PUT events and executes the C++ code, so everything is
 fine while the node is running. However, we use native persistence, so
 after a node restart, the local cache is read from the disk but the C++
 code hasn't been run for any cache entries, which means that Ignite and C++
 memory are out of sync.

 Iterating through the local cache entries is only possible once the
 node has already joined the cluster, but that's too late for us - it needs
 to be done before joining the cluster.

 I've managed to add a lifecycle bean event BEFORE_CLUSTER_JOIN (see
 http://apache-ignite-users.70518.x6.nabble.com/Register-listeners-before-joining-the-cluster-tc25944.html,
 a PR is hopefully coming soon), which is triggered before joining the
 cluster but at that point we cannot access the cache via
 ignite.cache(...). Is there a way to access all entries in the native
 persistence at that point or earlier? I'm also fine with modifying the
 Ignite source code if that's necessary (and simple enough), since we are
 just prototyping.

>>>

Re: Data streamer has been closed.

[Ilya Kasnacheev]

Hello!

I have ran your code several times and not encountered this problem. Can
you make a new reproducer which will show it (semi)reliably?

Regards,
-- 
Ilya Kasnacheev


чт, 17 янв. 2019 г. в 05:18, yangjiajun <1371549...@qq.com>:

> Hello.
>
> Thanks for reply.Unfortunately,I still get the exception after running my
> test on 2.7  for several times.
>
>
> ilya.kasnacheev wrote
> > Hello!
> >
> > I can reproduce this problem, but then again, it does not seem to
> > reproduce
> > on 2.7. Have you considered upgrading?
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > ср, 16 янв. 2019 г. в 14:14, yangjiajun <
>
> > 1371549332@
>
> >>:
> >
> >> Hello.
> >>
> >> I do  test on a ignite 2.6 node with persistence enabled and get an
> >> exception:
> >>
> >>  Exception in thread "main" java.sql.BatchUpdateException: class
> >> org.apache.ignite.IgniteCheckedException: Data streamer has been closed.
> >> at
> >>
> >>
> org.apache.ignite.internal.jdbc.thin.JdbcThinConnection$StreamState.readResponses(JdbcThinConnection.java:1017)
> >> at java.lang.Thread.run(Unknown Source)
> >>
> >> Here is my test code:
> >>
> >> import java.sql.Connection;
> >> import java.sql.DriverManager;
> >> import java.sql.PreparedStatement;
> >> import java.sql.SQLException;
> >> import java.util.Properties;
> >>
> >> /**
> >>  * test insert data in streaming mode
> >>  * */
> >> public class InsertStreamingMode {
> >>
> >> private static Connection conn;
> >>
> >> public static void main(String[] args) throws Exception {
> >>
> >> initialize();
> >>
> >> close();
> >> }
> >>
> >> public static void close() throws Exception {
> >> conn.close();
> >> }
> >>
> >> public static void initialize() throws Exception {
> >> Class.forName("org.apache.ignite.IgniteJdbcThinDriver");
> >> final String dbUrl =
> >>
> >>
> "jdbc:ignite:thin://ip:port;lazy=true;skipReducerOnUpdate=true;replicatedOnly=true";
> >> final Properties props = new Properties();
> >> conn = DriverManager.getConnection(dbUrl, props);
> >> initData();
> >> overwriteData();
> >> }
> >>
> >> private static void initData() throws SQLException{
> >>
> >> long start=System.currentTimeMillis();
> >> conn.prepareStatement("SET STREAMING ON ALLOW_OVERWRITE
> >> ON").execute();
> >>
> >> String sql="insert INTO  city1(id,name,name1)
> >> VALUES(?,?,?)";
> >> PreparedStatement ps=conn.prepareStatement(sql);
> >> for(int i=0;i<160;i++){
> >> String s1=String.valueOf(Math.random());
> >> String s2=String.valueOf(Math.random());
> >> ps.setInt(1, i);
> >> ps.setString(2, s1);
> >> ps.setString(3, s2);
> >> ps.execute();
> >> }
> >> conn.prepareStatement("set streaming off").execute();
> >> long end=System.currentTimeMillis();
> >> System.out.println(end-start);
> >> }
> >>
> >> private static void overwriteData() throws SQLException{
> >>
> >> long start=System.currentTimeMillis();
> >> conn.prepareStatement("SET STREAMING ON ALLOW_OVERWRITE
> >> ON").execute();
> >>
> >> String sql="insert INTO  city1(id,name,name1)
> >> VALUES(?,?,?)";
> >> PreparedStatement ps=conn.prepareStatement(sql);
> >> for(int i=0;i<160;i++){
> >> String s1="test";
> >> String s2="test";
> >> ps.setInt(1, i);
> >> ps.setString(2, s1);
> >> ps.setString(3, s2);
> >> ps.execute();
> >> }
> >> conn.prepareStatement("set streaming off").execute();
> >> long end=System.currentTimeMillis();
> >> System.out.println(end-start);
> >> }
> >> }
> >>
> >> Here is the table:
> >> CREATE TABLE city1(id LONG PRIMARY KEY, name VARCHAR,name1 VARCHAR) WITH
> >> "template=replicated"
> >>
> >> The exception occurs on overwriteData method.
> >>
> >>
> >>
> >> --
> >> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
> >>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Does Ignite message support C++?

[Igor Sapego]

No you should not have the same configs, but you need to have the same
configuration of the protocol, which is different for Java and C++ by
default
(I believe, you use Java server).

You may find details here [1].

[1] -
https://apacheignite-cpp.readme.io/docs/cross-platform-interoperability

Best Regards,
Igor


On Thu, Jan 17, 2019 at 1:00 PM summasumma  wrote:

> Thanks Igor.
>
> Yes it is trying to start in client mode now with correct file. My bad
> filename misspelled by a character.
>
> Now im getting following error:
> [09:55:28] REST protocols do not start on client node. To start the
> protocols on client node set '-DIGNITE_REST_START_ON_CLIENT=true' system
> property.
> [09:55:29,110][SEVERE][main][IgniteKernal] Failed to start manager:
> GridManagerAdapter [enabled=true,
> name=o.a.i.i.managers.discovery.GridDiscoveryManager]
> class org.apache.ignite.IgniteCheckedException: Failed to start SPI:
> TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000,
> marsh=JdkMarshaller
> [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@6c0d9d86],
> reconCnt=10, reconDelay=2000, maxAckTimeout=60, forceSrvMode=false,
> clientReconnectDisabled=false, internalLsnr=null]
> at
>
> org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:300)
> at
>
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:915)
> at
>
> org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1721)
> at
> org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1028)
> at
>
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2014)
> at
>
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1723)
> at
> org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1151)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:649)
> at
>
> org.apache.ignite.internal.processors.platform.PlatformAbstractBootstrap.start(PlatformAbstractBootstrap.java:43)
> at
>
> org.apache.ignite.internal.processors.platform.PlatformIgnition.start(PlatformIgnition.java:75)
> Caused by: class org.apache.ignite.spi.IgniteSpiException: Local node's
> binary configuration is not equal to remote node's binary configuration
> [locNodeId=3a0f06e7-b430-4e70-83cd-69d5067fb06f,
> rmtNodeId=de5d439d-bbfa-4446-aeaf-4beedb6c4953,
> locBinaryCfg={globIdMapper=org.apache.ignite.binary.BinaryBasicIdMapper,
> compactFooter=false, globSerializer=null}, rmtBinaryCfg=null]
> at
>
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.checkFailedError(TcpDiscoverySpi.java:1874)
> at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1771)
> at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>
>
> IS IT LIKE WE SHOULD HAVE EXACT SAME XML FILES with DISCOVERYSPI
> configuration having IPs of all the server and client nodes in it?
> Mine is:
>
> 
>  class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> 
>
>
> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> 
> 
> 10.42.27.121:47500..47509  --->
> this is server
> 10.42.27.131:47500..47509  --->
> this is another server
> 10.42.27.125:47500..47509
>  >
> this is c++ client im starting
> 
> 
>   
> 
> 
> 
>
>
> If this is the case what if i need to scale with another client in new IP
> to
> the cluster?
> Please clarify.
>
> Thanks,
> ...summa
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Does Ignite message support C++?

[summasumma]

Thanks Igor.

Yes it is trying to start in client mode now with correct file. My bad
filename misspelled by a character.

Now im getting following error:
[09:55:28] REST protocols do not start on client node. To start the
protocols on client node set '-DIGNITE_REST_START_ON_CLIENT=true' system
property.
[09:55:29,110][SEVERE][main][IgniteKernal] Failed to start manager:
GridManagerAdapter [enabled=true,
name=o.a.i.i.managers.discovery.GridDiscoveryManager]
class org.apache.ignite.IgniteCheckedException: Failed to start SPI:
TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000,
marsh=JdkMarshaller
[clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@6c0d9d86],
reconCnt=10, reconDelay=2000, maxAckTimeout=60, forceSrvMode=false,
clientReconnectDisabled=false, internalLsnr=null]
at
org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:300)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:915)
at
org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1721)
at
org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1028)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2014)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1723)
at
org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1151)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:649)
at
org.apache.ignite.internal.processors.platform.PlatformAbstractBootstrap.start(PlatformAbstractBootstrap.java:43)
at
org.apache.ignite.internal.processors.platform.PlatformIgnition.start(PlatformIgnition.java:75)
Caused by: class org.apache.ignite.spi.IgniteSpiException: Local node's
binary configuration is not equal to remote node's binary configuration
[locNodeId=3a0f06e7-b430-4e70-83cd-69d5067fb06f,
rmtNodeId=de5d439d-bbfa-4446-aeaf-4beedb6c4953,
locBinaryCfg={globIdMapper=org.apache.ignite.binary.BinaryBasicIdMapper,
compactFooter=false, globSerializer=null}, rmtBinaryCfg=null]
at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.checkFailedError(TcpDiscoverySpi.java:1874)
at
org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1771)
at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)


IS IT LIKE WE SHOULD HAVE EXACT SAME XML FILES with DISCOVERYSPI
configuration having IPs of all the server and client nodes in it?
Mine is:





   


10.42.27.121:47500..47509  --->
this is server
10.42.27.131:47500..47509  --->
this is another server
10.42.27.125:47500..47509   >
this is c++ client im starting


  





If this is the case what if i need to scale with another client in new IP to
the cluster?
Please clarify.

Thanks,
...summa




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Does Ignite message support C++?

[Igor Sapego]

It should start in client mode. Make sure that the right configuration file
is used.

Best Regards,
Igor


On Thu, Jan 17, 2019 at 12:40 PM summasumma  wrote:

> Thanks Igor.
>
> I am trying to do an example putandget operation using the Ignite-c++
> client
> code from the following link:
> https://github.com/apache/ignite/tree/master/modules/platforms/cpp/examples
>
> ==
> Source code:
>
> https://github.com/apache/ignite/blob/master/modules/platforms/cpp/examples/put-get-example/src/put_get_example.cpp
>
> Snippet below:
> {
>IgniteConfiguration cfg;
>cfg.springCfgPath =
> "platforms/cpp/examples/put-get-example/config/example-cache.xml";
> try
> {
> // Start a node.
> Ignite grid = Ignition::Start(cfg);
>  
> }
> ===
> But when i run the above example, the code is always starting Ignite in
> server-mode eventhough the 'example-cache.xml' contains the following
> settings:
>
> 
>
> How can i start a Ignite c++ node and perform putget operation here?
>
> Please clarify.
>
> Thanks,
> ...summa
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Does Ignite message support C++?

[summasumma]

Thanks Igor.

I am trying to do an example putandget operation using the Ignite-c++ client
code from the following link:
https://github.com/apache/ignite/tree/master/modules/platforms/cpp/examples

==
Source code: 
https://github.com/apache/ignite/blob/master/modules/platforms/cpp/examples/put-get-example/src/put_get_example.cpp

Snippet below:
{
   IgniteConfiguration cfg;
   cfg.springCfgPath =
"platforms/cpp/examples/put-get-example/config/example-cache.xml";
try
{
// Start a node.
Ignite grid = Ignition::Start(cfg);
 
}
===
But when i run the above example, the code is always starting Ignite in
server-mode eventhough the 'example-cache.xml' contains the following
settings:
 


How can i start a Ignite c++ node and perform putget operation here?

Please clarify.

Thanks,
...summa





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/