Hi,
Currently we are using Apache Ignite for caching huge amount of data (> 500 GB)
and performing some computations over it. One of our computations required
returning big chunk of data (> 4GB after serialization) from the compute node
to the requester, which caused integer overflow while serialization.
Exception:
NFO | jvm 3 | 2016/07/07 11:29:55 | Caused by:
java.lang.NegativeArraySizeException: null
INFO | jvm 3 | 2016/07/07 11:29:55 | at
org.gridgain.grid.util.io.GridUnsafeDataOutput.requestFreeSize(GridUnsafeDataOutput.java:153)
INFO | jvm 3 | 2016/07/07 11:29:55 | at
org.gridgain.grid.util.io.GridUnsafeDataOutput.writeInt(GridUnsafeDataOutput.java:352)
INFO | jvm 3 | 2016/07/07 11:29:55 | at
org.gridgain.grid.marshaller.optimized.GridOptimizedObjectOutputStream.writeInt(GridOptimizedObjectOutputStream.java:602)
INFO | jvm 3 | 2016/07/07 11:29:55 | at
org.gridgain.grid.marshaller.optimized.GridOptimizedObjectOutputStream.writeArrayList(GridOptimizedObjectOutputStream.java:320)
INFO | jvm 3 | 2016/07/07 11:29:55 | at
org.gridgain.grid.marshaller.optimized.GridOptimizedClassDescriptor.write(GridOptimizedClassDescriptor.java:779)
INFO | jvm 3 | 2016/07/07 11:29:55 | at
org.gridgain.grid.marshaller.optimized.GridOptimizedObjectOutputStream.writeObject0(GridOptimizedObjectOutputStream.java:201)
INFO | jvm 3 | 2016/07/07 11:29:55 | at
org.gridgain.grid.marshaller.optimized.GridOptimizedObjectOutputStream.writeFields(GridOptimizedObjectOutputStream.java:485)
INFO | jvm 3 | 2016/07/07 11:29:55 | at
org.gridgain.grid.marshaller.optimized.GridOptimizedObjectOutputStream.writeSerializable(GridOptimizedObjectOutputStream.java:306)
INFO | jvm 3 | 2016/07/07 11:29:55 | at
org.gridgain.grid.marshaller.optimized.GridOptimizedClassDescriptor.write(GridOptimizedClassDescriptor.java:829)
INFO | jvm 3 | 2016/07/07 11:29:55 | at
org.gridgain.grid.marshaller.optimized.GridOptimizedObjectOutputStream.writeObject0(GridOptimizedObjectOutputStream.java:201)
INFO | jvm 3 | 2016/07/07 11:29:55 | at
org.gridgain.grid.marshaller.optimized.GridOptimizedObjectOutputStream.writeArrayList(GridOptimizedObjectOutputStream.java:323)
INFO | jvm 3 | 2016/07/07 11:29:55 | at
org.gridgain.grid.marshaller.optimized.GridOptimizedClassDescriptor.write(GridOptimizedClassDescriptor.java:779)
INFO | jvm 3 | 2016/07/07 11:29:55 | at
org.gridgain.grid.marshaller.optimized.GridOptimizedObjectOutputStream.writeObject0(GridOptimizedObjectOutputStream.java:201)
INFO | jvm 3 | 2016/07/07 11:29:55 | at
org.gridgain.grid.marshaller.optimized.GridOptimizedObjectOutputStream.writeFields(GridOptimizedObjectOutputStream.java:485)
INFO | jvm 3 | 2016/07/07 11:29:55 | at
org.gridgain.grid.marshaller.optimized.GridOptimizedObjectOutputStream.writeSerializable(GridOptimizedObjectOutputStream.java:306)
INFO | jvm 3 | 2016/07/07 11:29:55 | at
org.gridgain.grid.marshaller.optimized.GridOptimizedClassDescriptor.write(GridOptimizedClassDescriptor.java:829)
INFO | jvm 3 | 2016/07/07 11:29:55 | at
org.gridgain.grid.marshaller.optimized.GridOptimizedObjectOutputStream.writeObject0(GridOptimizedObjectOutputStream.java:201)
INFO | jvm 3 | 2016/07/07 11:29:55 | at
org.gridgain.grid.marshaller.optimized.GridOptimizedObjectOutputStream.writeArrayList(GridOptimizedObjectOutputStream.java:323)
INFO | jvm 3 | 2016/07/07 11:29:55 | at
org.gridgain.grid.marshaller.optimized.GridOptimizedClassDescriptor.write(GridOptimizedClassDescriptor.java:779)
INFO | jvm 3 | 2016/07/07 11:29:55 | at
org.gridgain.grid.marshaller.optimized.GridOptimizedObjectOutputStream.writeObject0(GridOptimizedObjectOutputStream.java:201)
INFO | jvm 3 | 2016/07/07 11:29:55 | at
org.gridgain.grid.marshaller.optimized.GridOptimizedObjectOutputStream.writeFields(GridOptimizedObjectOutputStream.java:485)
INFO | jvm 3 | 2016/07/07 11:29:55 | at
org.gridgain.grid.marshaller.optimized.GridOptimizedObjectOutputStream.defaultWriteObject(GridOptimizedObjectOutputStream.java:655)
INFO | jvm 3 | 2016/07/07 11:29:55 | at
java.util.Collections$SynchronizedCollection.writeObject(Collections.java:2081)
INFO | jvm 3 | 2016/07/07 11:29:55 | at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO | jvm 3 | 2016/07/07 11:29:55 | at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
INFO | jvm 3 | 2016/07/07 11:29:55 | at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
INFO | jvm 3 | 2016/07/07 11:29:55 | at
java.lang.reflect.Method.invoke(Method.java:498)
INFO | jvm 3 | 2016/07/07 11:29:55 | at
org.gridgain.grid.marshaller.optimized.GridOptimizedObjectOutputStream.writeSerializable(GridOptimizedObjectOutputStream.java:296)
INFO | jvm 3 | 2016/07/07 11:29:55 | ... 22 common frames omitted
We were using GridGain 6.2 and currently moved to Apache Ignite 1.5. I checked
the code and found that part of the code responsible for doing the
serialization part is the same, so the problem will not be solved in Apache
Ignite.
Class OptimizedMarshaller.java
/** {@inheritDoc} */
@Override public byte[] marshal(@Nullable Object obj) throws
IgniteCheckedException {
OptimizedObjectOutputStream objOut = null;
try {
objOut = OptimizedObjectStreamRegistry.out();
objOut.context(clsMap, ctx, mapper, requireSer);
objOut.writeObject(obj);
return objOut.out().array();
}
catch (IOException e) {
throw new IgniteCheckedException("Failed to serialize object: " +
obj, e);
}
finally {
OptimizedObjectStreamRegistry.closeOut(objOut);
}
}
The problem with that code that the APIs force using byte[] for serialization
which means we can't return objects that can be serialized with more than
byte[INT_MAX].
The question is: Does Apache Ignite provide out of the box streaming
serialization between nodes?
Thanks,
Mohamed Sahmoud
**********************************************************************
The clear choice for graduates – No.1 in the 2015 Top 100 Australian Graduate
Employers Survey
Prepared by the Australian Financial Review in association with GradConnection
Proud winner of the 2014 Human Rights Business Award for our commitment to
Reconciliation
*******************************************************************************************************************
The information in this e-mail is confidential and may be legally privileged.
It is intended solely for the addressee. Access to this e-mail by anyone else
is unauthorised. If you have received this communication in error, please
notify us immediately by return e-mail with the subject heading "Received in
error" or telephone +61 2 93357000, then delete the email and destroy any
copies of it. If you are not the intended recipient, any disclosure, copying,
distribution or any action taken or omitted to be taken in reliance on it, is
prohibited and may be unlawful. Any opinions or advice contained in this e-mail
are subject to the terms and conditions expressed in the governing KPMG client
engagement letter. Opinions, conclusions and other information in this e-mail
and any attachments that do not relate to the official business of the firm are
neither given nor endorsed by it.
KPMG cannot guarantee that e-mail communications are secure or error-free, as
information could be intercepted, corrupted, amended, lost, destroyed, arrive
late or incomplete, or contain viruses.
KPMG, an Australian partnership and a member firm of the KPMG network of
independent member firms affiliated with KPMG International, a Swiss
cooperative. KPMG International provides no services to clients.
Liability limited by a scheme approved under Professional Standards Legislation.
*******************************************************************************************************************
