Hive (IGFS + IgniteMR) vs Hive (Tez)

[theena]

Hi
I am doing a POC on HDP 2.5 Cluster with Ignite as Hadoop Accelerator.

we have 3 node cluster each with 8 core and 60G RAM.

I was able to run hive on Tez query on a sample data set and finished in 32
sec.
The same query took 94 sec in Hive + IGFS + Ignite-MR.

I followed most of the instructions from this forum and Ignite Website. Just
want to check if I am missing any important parameters that could improve
the performance.

Below are more details:

core-site properties:


  fs.igfs.impl
  org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem


  fs.AbstractFileSystem.igfs.impl
  org.apache.ignite.hadoop.fs.v2.IgniteHadoopFileSystem


  fs.defaultFS
igfs://igfs@
  true


IG_MR_JOB_TRACKER=ip-10-0-0-200:11211

1. I can run IGFS and use HDFS as the secondary FS

hadoop fs -ls  igfs://igfs@//tmp/orders/


2. I can run Ignite-MR


hadoop  --config /usr/etc/ignite_conf jar
/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples-2.*.jar
wordcount -Dignite.job.shared.classloader=false
-Dmapreduce.jobtracker.address=$IG_MR_JOB_TRACKER
-Dmapreduce.framework.name=ignite igfs://igfs@//tmp/orders/
igfs://igfs@//tmp/6

3. I can run hive query on IGFS+IgniteMR and see the console log.

beeline -n hdfs -u "$HIVE_JDBC"  --hiveconf fs.defaultFS=igfs://igfs@/ 
--hiveconf mapreduce.framework.name=ignite --hiveconf
mapreduce.jobtracker.address=$IG_MR_JOB_TRACKER  

set ignite.job.shared.classloader=false ;
set hive.rpc.query.plan = true; 
set hive.execution.engine = mr; 
set hive.auto.convert.join = false; -- Added this to avoid mapRed task
failure while running the query.

4. I could not run Hive+Tez+IGFS

beeline -n hdfs -u "$HIVE_JDBC" 
set fs.defaultFS=igfs://igfs@/  ; 
set hive.execution.engine = tez; 
set tez.use.cluster.hadoop-libs = true;  
set ignite.job.shared.classloader=false ;
set hive.rpc.query.plan = true; 

INFO  : Tez session hasn't been created yet. Opening session
ERROR : Failed to execute tez graph.
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown.
Application application_1530330378726_0007 failed 2 times due to AM
Container for appattempt_1530330378726_0007_02 exited with  exitCode:
-1000
For more detailed output, check the application tracking page:
http://ip-10.ec2.internal:8088/cluster/app/application_1530330378726_0007
Then click on links to logs of each attempt.
Diagnostics: java.io.IOException: Failed to parse endpoint: null
Failing this attempt. Failing the application.
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:779)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:217)
at
org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:279)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:159)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1745)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1491)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1151)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197)
at
org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76)
at
org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:253)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1865)
at
org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:264)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Error: Error while processing statement: FAILED: Execution Error, return
code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask (state=08S01,code=1)


Attached the log file and default config.
default-config.xml
  
master.log
  
worker.log
  
compute.log

Ignite File System (igfs) spillover to disk

[matt]

Is it possible to have the IGFS component write to disk once heap/off-heap
consumption hits a certain threshold? I have a custom cache store for one
component of the app, but a different component requires temporary storage
of raw data; we're using igfs for this, but what happens if the file size is
much larger than the available RAM? Can igfs be configured to use the
(relatively new) native "DataStorage" feature to spillover from RAM to disk,
but then also have a specific cache use a custom store?

Thanks
- Matt



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Deadlock during cache loading

[smovva]

I have a fairly similar setup. What type of EC2 instances are you using? Just
for compare my setup.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: OutOfMemoryError while streaming

[smovva]

Thanks again.

I have one more question. After pushing some data, I see only two nodes
having entries out of 8. All the others have 0 entries. 
My keys are 9 char long strings with a common prefix (all have the same
first 3 chars). Wondering why the keys are not evenly distributed. 




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: OutOfMemoryError while streaming

[breischl]

Non-heap memory is different than off-heap memory. Non-heap is (roughly
speaking) memory that the JVM itself uses. Off-heap is what Ignite is using
for storage off the heap. So you're probably not looking at what you think
you're looking at. 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: OutOfMemoryError while streaming

[smovva]

breischl, Thanks for the quick response. 
I'm not sure why the node stats show

Non-heap memory maximum | 744mb

Each server node has 32GB and I've assigned 3g for heap. Shouldn't this be
16GB since that's what I set the data region size to? 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: OutOfMemoryError while streaming

[breischl]

You're probably just running out of memory, though if you examine the
stacktrace it may tell you if you're running out of heap or off-heap memory.
If there's a call to Unsafe.something() in there, it's probably off-heap.
Otherwise it's probably on-heap. 

You do seem to be configuring only a 3 GB heap, and 16 GB off off-heap, so
you're likely not utilizing all the available memory. 

I believe you have to explicitly enable off-heap metrics collection, which
is probably why it's reporting zero usage. You can do that through code,
config, or JMX. See https://apacheignite.readme.io/docs/memory-metrics.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

OutOfMemoryError while streaming

[smovva]

I'm seeing OutOfMemoryError when I have multiple data streamers pushing data
into the cluster.
My cluster consists of 8 servers running one node each. Each server has 32GB
RAM and 4 Cores.
All the nodes are started like this
=
bin/ignite.sh -J-Xmx3g config/dev-cluster-config.xml
=

My cache consists of  
  - key: strings of length 9 chars 
  - Value: POJOs that have some metadata and one large array of integers
(8800 values)

I have 4 data streamers and as they start pushing data, I see some server
nodes throwing OutOfMemoryErrors and the data streamers hangup. 

The streamer code is here.

==
CacheConfiguration cacheCfg = new
CacheConfiguration<>(DATA_CACHE_NAME);
cacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
cacheCfg.setCacheMode(CacheMode.PARTITIONED);
cacheCfg.setBackups(2);
   
cacheCfg.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);
cacheCfg.setIndexedTypes(String.class, Tile.class);

IgniteCache viewpointCache =
ignite.getOrCreateCache(cacheCfg);

try (IgniteDataStreamer streamer =
ignite.dataStreamer(viewpointCache.getName())) {
Path sourceFile = Paths.get(hashIndexFile);
try(BufferedReader reader = Files.newBufferedReader(sourceFile,
Charset.forName("UTF-8")))
{
String currentLine = null;
while((currentLine = reader.readLine()) != null){
String hash = currentLine.trim();
streamer.addData(hash, getTileFor(hash));
}
}
}
==

Cluster-config.xml

 

   













 










=

Also, I noticed that there is nothing in off-heap memory. Here is the cache
stats for one node.
Total: 513278 
  Heap: 5  
  Off-Heap: 513273
  Off-Heap Memory: 0
and config also shows 
| Non-heap memory initialized | 2mb 
| Non-heap memory used| 86mb  
| Non-heap memory committed   | 88mb  
| Non-heap memory maximum | 744mb
How do I make it use and store more in Non-heap memory?

Thanks!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Deadlock during cache loading

[breischl]

StreamTransformer does an invoke() pretty much exactly like what I'm doing,
so that would not seem to change anything. 

https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/stream/StreamTransformer.java#L50-L53



I may try using a put(), but since I need to compare the existing cache
value, I'd need to get(), compare, then put(). I thought that may open up a
potential race condition, if two different updates happen close to each
other they could each do get(), then each do put(), but potentially in the
wrong order. Unless there's some locking I don't understand there? 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Deadlock during cache loading

[Denis Mekhanikov]

Entries, that are provided to the *receive()* method are immutable.
But you can either do *cache.put() *inside the *receive() *method, just
like *DataStreamerCacheUpdaters#Individual

*does,
or use *StreamTransformer *just like in StreamTransformerExample


Denis

пт, 29 июн. 2018 г. в 19:17, breischl :

> Hi Denis,
>   It was not clear to me that we could do the update from within the
> StreamReceiver without some sort of cache operation. Would we just use the
> CacheEntry.setValue() method to do that? Something roughly like the
> following?
>
> Thanks!
>
>
> public void receive(IgniteCache cache,
> Collection> newEntries) throws IgniteException
> {
>
> for (val newEntry : newEntries) {
> //Do our custom logic
> newEntry.setValue(someNewValue);
> }
> }
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Deadlock during cache loading

[breischl]

Hi Denis,
  It was not clear to me that we could do the update from within the
StreamReceiver without some sort of cache operation. Would we just use the
CacheEntry.setValue() method to do that? Something roughly like the
following?

Thanks!


public void receive(IgniteCache cache,
Collection> newEntries) throws IgniteException {

for (val newEntry : newEntries) {
//Do our custom logic
newEntry.setValue(someNewValue);
}
}





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Deadlock during cache loading

[breischl]

Hi Denis,
  It was not clear to me that we could do the update from within the
StreamReceiver without some sort of cache operation. Would we just use the
CacheEntry.setValue() method to do that? Something roughly like the
following?

Thanks!


public void receive(IgniteCache cache,
Collection> newEntries) throws IgniteException {

for (val newEntry : newEntries) {
//Do our custom logic
newEntry.setValue(someNewValue);
}
}





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: What is the difference between PRIMARY_SYNC and FULL_ASYNC

[Denis Mekhanikov]

Yep, you understood the difference correctly.
I'm not sure, if we are waiting for local updates in FULL_ASYNC mode,
though.
But anyways, updating local entries is much faster than the remote ones.

I'm not aware of any available public benchmark results with comparison of
different write synchronization modes.
But you can find a number of benchmarks in Ignite's repository:
https://github.com/apache/ignite/tree/master/modules/yardstick
You can run them with different configurations and compare yourself.

Consider sharing your results with the community when you get any.

Denis

пт, 29 июн. 2018 г. в 18:17, Cong Guo :

> Hi,
>
>
>
> Does PRIMARY_SYNC means waiting for only primary copies even if the
> primary copies are on remote nodes, while FULL_ASYNC means waiting for only
> local copies no matter the local copies are primary or not?
>
>
>
> Could you please give me an example case to show different performance
> results with the two CacheWriteSynchronizationModes?
>
>
>
> Thanks,
>
> Cong
>

What is the difference between PRIMARY_SYNC and FULL_ASYNC

[Cong Guo]

Hi,

Does PRIMARY_SYNC means waiting for only primary copies even if the primary 
copies are on remote nodes, while FULL_ASYNC means waiting for only local 
copies no matter the local copies are primary or not?

Could you please give me an example case to show different performance results 
with the two CacheWriteSynchronizationModes?

Thanks,
Cong

Re: Deadlock during cache loading

[Denis Mekhanikov]

Hi!

Why do you do this inside an invoke()?
All of this can be done just inside a receiver.
Can you get rid of the invoke and check, that deadlocks disappear?

Denis

пт, 29 июн. 2018 г. в 17:24, breischl :

> That does seem to be what's happening, but we're only invoke()'ing on keys
> that were passed into receive(), so that should not require going off-box.
> Right?
>
> Here's the relevant code...
>
>
> @Override
> public void receive(IgniteCache cache,
> Collection> newEntries) throws IgniteException
> {
>
> for (val newEntry : newEntries) {
> val entryKey = newEntry.getKey();
>
> cache.invoke(entryKey, ((CacheEntryProcessor Object>)
> (entry, args) -> {
> val key =
> (TKey) args[0]; //passed this in to make the lambda
> non-capturing, which is a slight perf optimization (fewer memory allocs)
> val newVal = (TEntity) args[1];
> val oldVal = entry.getValue();
>
> if (oldVal == null) {
> //Didn't already exist, we can just set the new values and
> be done
> log.info("event=receiverCreatingNewInstance key={}
> newValue={}", key, newVal);
> entry.setValue(newVal);
> } else if (isNewer(oldVal, newVal)) {
> log.info("event=newEntryHasHigherVersion key={}
> oldVersion={} newVersion={}", key, getVersionForLogging(oldVal),
> getVersionForLogging(newVal));
> entry.setValue(newVal);
> } else {
> log.info("event=newEntryHasLowerVersion key={}
> oldVersion={}
> newVersion={}", key, getVersionForLogging(oldVal),
> getVersionForLogging(newVal));
> }
> return null;
> }), entryKey, newEntry.getValue());
> }
> }
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Information regarding Ignite Web Console

[Denis Mekhanikov]

Sriveena,

You should configure corresponding query entities to be able to query data
in cache.
Annotation driven configuration is also available.

See more:
https://apacheignite.readme.io/docs/cache-queries#section-query-configuration-by-annotations

Denis

пт, 29 июн. 2018 г. в 12:43, Sriveena Mattaparthi <
sriveena.mattapar...@ekaplus.com>:

> Hi Denis,
>
>
>
> I am trying to use the below code to query the binary object
>
>
>
> IgniteCache cache =
> start.getOrCreateCache(cfg).withKeepBinary();
>
> BinaryObjectBuilder builder = start.binary().builder("BinaryTest");
>
> builder.setField("name", "Test");
>
> cache.put(1, builder.build());
>
>
>
> QueryCursor> query = cache.query(new SqlFieldsQuery("select
> name from BinaryTest"));
>
>
>
> But it is failing in latest 2.5 version saying BinaryTest Table does not
> exist.
>
>
>
> How do we query the binary objects in the above example?
>
>
>
> Please help.
>
>
>
> Thanks & Regards,
>
> Sriveena
>
>
>
> *From:* Denis Mekhanikov [mailto:dmekhani...@gmail.com]
> *Sent:* Wednesday, June 27, 2018 6:37 PM
>
>
> *To:* user@ignite.apache.org
> *Subject:* Re: Information regarding Ignite Web Console
>
>
>
> Sriveena,
>
>
>
> You can have objects of different types in one cache, but querying it will
> be tricky.
>
> You will have to configure QueryEntities
> 
>  for
> your data, that will describe, which fields are available for querying.
>
> Annotation based configuration
> 
> is also available.
>
> Querying nested object is also possible, if you configure the query
> entities properly:
> https://apacheignite-sql.readme.io/docs/schema-and-indexes#section-indexing-nested-objects
> 
>
>
>
> So, if you want to run SQL queries over your data, it should have some
> concrete schema.
>
>
>
> Denis
>
>
>
> ср, 27 июн. 2018 г. в 14:08, Sriveena Mattaparthi <
> sriveena.mattapar...@ekaplus.com>:
>
> Thank you so much for the quicker responses unlike any other forums..I
> really appreciate that.
>
>
>
> One last question Denis, we have plan to load all the mongodb collections
> to ignite cache and perform complex aggregations and join in memory.
>
>
>
> But Unlike any RDBMS data stores we cannot have fixed model objects for
> each collections as each document in the collection may have its own
> columns and datatypes.
>
>
>
> Could you please suggest, if ignite is the choice for this kind of
> scenario where same mongo collection have different type of data.
>
>
>
> Please note that we have tried using BinaryObject, but we are stuck that
> ignite doesn’t support querying on the inner binaryobject.( binaryobject
> inside a binaryobject – sub documents, array inside a mongo document)
>
>
>
> Thanks & Regards,
>
> Sriveena
>
>
>
> *From:* Denis Mekhanikov [mailto:dmekhani...@gmail.com]
> *Sent:* Wednesday, June 27, 2018 4:02 PM
>
>
> *To:* user@ignite.apache.org
> *Subject:* Re: Information regarding Ignite Web Console
>
>
>
> Sriveena,
>
>
>
> CacheStore
> 
>  extends
> the CacheWriter
> 
> interface, which has delete
> 

RE: Deadlock during cache loading

[breischl]

That does seem to be what's happening, but we're only invoke()'ing on keys
that were passed into receive(), so that should not require going off-box.
Right? 

Here's the relevant code...


@Override
public void receive(IgniteCache cache,
Collection> newEntries) throws IgniteException {

for (val newEntry : newEntries) {
val entryKey = newEntry.getKey();

cache.invoke(entryKey, ((CacheEntryProcessor)
(entry, args) -> {
val key =
(TKey) args[0]; //passed this in to make the lambda
non-capturing, which is a slight perf optimization (fewer memory allocs)
val newVal = (TEntity) args[1];
val oldVal = entry.getValue();

if (oldVal == null) {
//Didn't already exist, we can just set the new values and
be done
log.info("event=receiverCreatingNewInstance key={}
newValue={}", key, newVal);
entry.setValue(newVal);
} else if (isNewer(oldVal, newVal)) {
log.info("event=newEntryHasHigherVersion key={}
oldVersion={} newVersion={}", key, getVersionForLogging(oldVal),
getVersionForLogging(newVal));
entry.setValue(newVal);
} else {
log.info("event=newEntryHasLowerVersion key={} oldVersion={}
newVersion={}", key, getVersionForLogging(oldVal),
getVersionForLogging(newVal));
}
return null;
}), entryKey, newEntry.getValue());
}
}





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Questions on setting up firewall for multicast cluster discovery.

[vkulichenko]

Hi Jon,

First of all, you don't have to use multicast for discovery. Using static IP
configuration or one other shared IP finder might simplify the setup:
https://apacheignite.readme.io/docs/tcpip-discovery

Second of all, I'm not sure I fully understand what you're trying to
achieve. Are both nodes in the same network? If yes, why is there a firewall
between them and why do you need to restrict internal traffic?

-Val



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Questions on setting up firewall for multicast cluster discovery.

[Jon Tricker]

Am trying to set up a couple of 2.5.0  nodes on CentOS boxes. I have opened the 
recommended ports:

firewall-cmd --add-port=47500-47502/tcp
firewall-cmd --add-port=47100-47200/tcp
firewall-cmd --add-port=47400/udp

I see an initial UDP packet, to the ignite multicast group address, received 
correctly on destination port 47400. However then the remote node (x.y.2.84 in 
the following trace) sends a second UDP packet from 47400 to a random port on 
the local machine (x.y.2.99). Giving the following firewall trace and failure 
to join the cluster.

Jun 29 11:00:21 localhost kernel: FINAL_REJECT: IN=enp0s3 OUT= 
MAC=08:00:27:6c:dd:8f:08:00:27:96:51:2f:08:00 SRC=x.y.2.84 DST=x.y.2.99 LEN=543 
TOS=0x00 PREC=0x00 TTL=64 ID=30905 DF PROTO=UDP SPT=47400 DPT=35072 LEN=523
Jun 29 11:01:22 localhost kernel: FINAL_REJECT: IN=enp0s3 OUT= 
MAC=08:00:27:6c:dd:8f:08:00:27:96:51:2f:08:00 SRC=x.y.2.84 DST=x.y.2.99 LEN=543 
TOS=0x00 PREC=0x00 TTL=64 ID=65234 DF PROTO=UDP SPT=47400 DPT=47668 LEN=523
Jun 29 11:01:22 localhost kernel: FINAL_REJECT: IN=enp0s3 OUT= 
MAC=08:00:27:6c:dd:8f:08:00:27:96:51:2f:08:00 SRC=x.y.2.84 DST=x.y.2.99 LEN=543 
TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=47400 DPT=40812 LEN=523

Obviously I don't know the address of the remote machine in advance. Or the 
incoming port number. The only option seems to be opening the entire random 
port range to UDP traffic:

firewall-cmd --add-port=1024-65535/udp

This works and the cluster is joined. However, even if this could also be 
limited to source port 47400, it is dangerous. Remote malware could use that 
port to access other services.

Is there a better way to do this?


The information in this e-mail and any attachments is confidential and may be 
legally privileged. It is intended solely for the addressee or addressees. Any 
use or disclosure of the contents of this e-mail/attachments by a not intended 
recipient is unauthorized and may be unlawful. If you have received this e-mail 
in error please notify the sender. Please note that any views or opinions 
presented in this e-mail are solely those of the author and do not necessarily 
represent those of TEMENOS. We recommend that you check this e-mail and any 
attachments against viruses. TEMENOS accepts no liability for any damage caused 
by any malicious code or virus transmitted by this e-mail.

RE: Information regarding Ignite Web Console

[Sriveena Mattaparthi]

Hi Denis,

I am trying to use the below code to query the binary object

IgniteCache cache = 
start.getOrCreateCache(cfg).withKeepBinary();
BinaryObjectBuilder builder = start.binary().builder("BinaryTest");
builder.setField("name", "Test");
cache.put(1, builder.build());

QueryCursor> query = cache.query(new SqlFieldsQuery("select name 
from BinaryTest"));

But it is failing in latest 2.5 version saying BinaryTest Table does not exist.

How do we query the binary objects in the above example?

Please help.

Thanks & Regards,
Sriveena

From: Denis Mekhanikov [mailto:dmekhani...@gmail.com]
Sent: Wednesday, June 27, 2018 6:37 PM
To: user@ignite.apache.org
Subject: Re: Information regarding Ignite Web Console

Sriveena,

You can have objects of different types in one cache, but querying it will be 
tricky.
You will have to configure 
QueryEntities
 for your data, that will describe, which fields are available for querying.
Annotation based 
configuration
 is also available.
Querying nested object is also possible, if you configure the query entities 
properly: 
https://apacheignite-sql.readme.io/docs/schema-and-indexes#section-indexing-nested-objects

So, if you want to run SQL queries over your data, it should have some concrete 
schema.

Denis

ср, 27 июн. 2018 г. в 14:08, Sriveena Mattaparthi 
mailto:sriveena.mattapar...@ekaplus.com>>:
Thank you so much for the quicker responses unlike any other forums..I really 
appreciate that.

One last question Denis, we have plan to load all the mongodb collections to 
ignite cache and perform complex aggregations and join in memory.

But Unlike any RDBMS data stores we cannot have fixed model objects for each 
collections as each document in the collection may have its own columns and 
datatypes.

Could you please suggest, if ignite is the choice for this kind of scenario 
where same mongo collection have different type of data.

Please note that we have tried using BinaryObject, but we are stuck that ignite 
doesn’t support querying on the inner binaryobject.( binaryobject inside a 
binaryobject – sub documents, array inside a mongo document)

Thanks & Regards,
Sriveena

From: Denis Mekhanikov 
[mailto:dmekhani...@gmail.com]
Sent: Wednesday, June 27, 2018 4:02 PM

To: user@ignite.apache.org
Subject: Re: Information regarding Ignite Web Console

Sriveena,

CacheStore
 extends the 
CacheWriter
 interface, which has 
delete
 and 
deleteAll

Re: And again... Failed to get page IO instance (page content is corrupted)

[Andrey Mashenkov]

Hi Oleg,

Yes, page corruption issues shouldn't happened when persistence is disabled.
Please, let us know it you will face one.


On Fri, Jun 29, 2018 at 1:56 AM Olexandr K 
wrote:

> Hi Andrey,
>
> Thanks for clarifying this.
> We have just a single persistent cache and I reworked the code to get rid
> of expiration policy.
> All our non-persistent caches have expiration policy but this should not
> be a problem, right?
>
> BR, Oleksandr
>
> On Thu, Jun 28, 2018 at 8:37 PM, Andrey Mashenkov <
> andrey.mashen...@gmail.com> wrote:
>
>> Hi Oleg,
>>
>> The issue you mentioned IGNITE-8659 [1] is caused by IGNITE-5874 [2] that
>> will not a part of ignite-2.6 release.
>> For now, 'ExpiryPolicy with persistence' is totally broken and all it's
>> fixes are planned to the next 2.7 release.
>>
>>
>> [1] https://issues.apache.org/jira/browse/IGNITE-8659
>> [2] https://issues.apache.org/jira/browse/IGNITE-5874
>>
>> On Tue, Jun 26, 2018 at 11:26 PM Olexandr K <
>> olexandr.kundire...@gmail.com> wrote:
>>
>>> Hi Andrey,
>>>
>>> I see Fix version 2.7 in Jira:
>>> https://issues.apache.org/jira/browse/IGNITE-8659
>>> This is a critical bug.. bouncing of server node in not-a-right-time
>>> causes a catastrophe.
>>> This mean no availability in fact - I had to clean data folders to start
>>> my cluster after that
>>>
>>> BR, Oleksandr
>>>
>>>
>>> On Fri, Jun 22, 2018 at 4:06 PM, Andrey Mashenkov <
>>> andrey.mashen...@gmail.com> wrote:
>>>
 Hi,

 We've found and fixed few issues related to ExpiryPolicy usage.
 Most likely, your issue is [1] and it is planned to ignite 2.6 release.

 [1] https://issues.apache.org/jira/browse/IGNITE-8659


 On Fri, Jun 22, 2018 at 8:43 AM Olexandr K <
 olexandr.kundire...@gmail.com> wrote:

> Hi Team,
>
> Issue is still there in 2.5.0
>
> Steps to reproduce:
> 1) start 2 servers + 2 clients topology
> 2) start load testing on client nodes
> 3) stop server 1
> 4) start server 1
> 5) stop server 1 again when rebalancing is in progress
> => and we got data corrupted here, see error below
> => we were not able to restart Ignite cluster after that and need to
> perform data folders cleanup...
>
> 2018-06-21 11:28:01.684 [ttl-cleanup-worker-#43] ERROR  - Critical
> system error detected. Will be handled accordingly to configured handler
> [hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler,
> failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class
> o.a.i.IgniteException: Runtime failure on bounds: [lower=null,
> upper=PendingRow [
> org.apache.ignite.IgniteException: Runtime failure on bounds:
> [lower=null, upper=PendingRow []]
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:971)
> ~[ignite-core-2.5.0.jar:2.5.0]
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:950)
> ~[ignite-core-2.5.0.jar:2.5.0]
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:1024)
> ~[ignite-core-2.5.0.jar:2.5.0]
> at
> org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197)
> ~[ignite-core-2.5.0.jar:2.5.0]
> at
> org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:137)
> [ignite-core-2.5.0.jar:2.5.0]
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> [ignite-core-2.5.0.jar:2.5.0]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
> Caused by: java.lang.IllegalStateException: Item not found: 2
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341)
> ~[ignite-core-2.5.0.jar:2.5.0]
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450)
> ~[ignite-core-2.5.0.jar:2.5.0]
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492)
> ~[ignite-core-2.5.0.jar:2.5.0]
> at
> org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150)
> ~[ignite-core-2.5.0.jar:2.5.0]
> at
> org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
> ~[ignite-core-2.5.0.j
>
> BR, Oleksandr
>
> On Thu, Jun 14, 2018 at 2:51 PM, Olexandr K <
> olexandr.kundire...@gmail.com> wrote:
>
>> Upgraded to 2.5.0 and didn't get such error so far..
>> Thanks!
>>
>> On Wed, Jun 13, 2018 at 4:58 PM, dkarachentsev <

RE: Best practice for class versioning: marshaller error

[Calvin KL Wong, CLSA]

Then I am confused with the exception I got.  Please bear with me.

I believe what happened was a client (with a new version of user class) trying 
to deserialize an old object (an old version of user class) from Continuous 
Query.
This was what happened base on the log:
1. GridCacheIoManager unmarshalled GridCacheQueryResponse
2. GridCacheQueryResponse unmarshalled one element in its bytes collection
3. BinaryMarshaller (internally GridBinaryMarshaller) was being used
1. It then found out that it needed to unmarshal an Optimized object
   1. The Optimized object turned out to be a GridCacheQueryResponseEntry 
and tried to unmarshal GridCacheQueryResponseEntry.val
1. Looks like we tried to unmarshall serveral 'Serializable'  
objects along the way

OptimizedMarshaller is always used for all system type (unless it is 
Binarylizable).  I looked at the OptimizedObjectInputStream.java, I cannot find 
any way to 'read object using BinaryMarshaller or BinaryReaderExImpl ' once we 
are inside OptimizedObjectInputStream.readObject0().

Can I assume that BinaryMarshaller won't be used for any object embedded inside 
GridCacheQueryResponse?
If I am correct, do you have any suggestion on how I can avoid this type of 
issue?  Hopefully I am wrong.

Thanks,
Calvin

Caused by: java.io.IOException: Unexpected error occurred during unmarshalling 
of an instance of the class: java.time.Ser. Check that all nodes are running 
the same version of Ignite and that all nodes have GridOptimizedMarshaller 
configured with identical optimized classes lists, if any (see setClassNames 
and setClassNamesPath methods). If your serialized classes implement 
java.io.Externalizable interface, verify that serialization logic is correct.
at 
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:359)
 ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at 
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:199)
 ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:365) 
~[?:1.8.0_74]
at 
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:513)
 ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at 
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:601)
 ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] 
< Step 3.1.1.1
at 
org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:927)
 ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at 
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at 
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:199)
 ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:365) 
~[?:1.8.0_74]
at 
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:513)
 ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at 
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:601)
 ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] 
< Step 3.1.1.1
at 
org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:927)
 ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at 
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
 ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at 
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:199)
 ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:365) 
~[?:1.8.0_74]
at 
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:513)
 ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at 
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:601)
 ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] 
< Step 3.1.1.1
at 
org.apache.ignite.internal