Re: BinaryObjectImpl.deserializeValue with specific ClassLoader

[Dmitriy Karachentsev]

Crosspost to dev list.

Igniters,

This proposal looks quite reasonable. Do we have any restrictions that
could prevent implementing such feature?
I think it's nice to have in Ignite.

Thanks!
-Dmitry.

On Sun, Apr 23, 2017 at 1:37 AM, npordash  wrote:

> Thanks!
>
> That would definitely help address the hack I've implemented where I have
> to
> reference classes in Ignite's internal package.
>
> However, I still have to work with the caches in binary which is less than
> ideal. It's pretty common in use-cases like this to first try to use
> Thread.currentThread().getContextClassLoader() and if that's not set then
> fallback to something else. In general, I think ignite should try to
> resolve
> a class based on the caller's context first instead of only relying on
> ignite's classloader or what it was configured with.
>
> In the spirit of the webinar that Denis just had, I think this kind of
> behavior will become mandatory for the service grid to get to the point
> where it can do deployments of services without requiring class files to be
> on ignite's classpath. I've heard that is something that's still
> tentatively
> planned and the use-case I outlined is an attempt to get around the current
> service grid limitations. :)
>
> WDYT?
>
> -Nick
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/BinaryObjectImpl-deserializeValue-with-
> specific-ClassLoader-tp12055p12171.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Binarylizable interface in apache ignite

[Dmitriy Karachentsev]

Hi.

Actually to make your data just a bit more compact you may
use org.apache.ignite.binary.BinaryRawWriter
and org.apache.ignite.binary.BinaryRawReader (writer.rawWriter() and
reader.rawReader()), but in this case you'll not be able to use SQL queries.

Binarylizable interface gives you a way to customize
marshalling/unmarshalling process, for example, do not store fields that
could be calculated, etc., just like Externalizable does. And of course if
you haven't serialized fields, you cannot query for them.

On Fri, Nov 18, 2016 at 2:56 PM, rishi007bansod 
wrote:

> I am trying to use Binarylizable interface in apache ignite for reducing
> memory utilization(for compact representation of data). Following is data
> class that I have made binarylizable.
>
> class order_line implements Binarylizable{
> @QuerySqlField(index = true)
> int ol_o_id;
> @QuerySqlField(index = true)
> int ol_d_id;
> @QuerySqlField(index = true)
> int ol_w_id;
> @QuerySqlField(index = true)
> int ol_number;
> @QuerySqlField
> int ol_i_id;
> @QuerySqlField
> int ol_supply_w_id;
> @QuerySqlField
> String ol_delivery_d;
> @QuerySqlField
> int ol_quantity;
> @QuerySqlField
> double ol_amount;
> @QuerySqlField
> String ol_dist_info;
>
>
> private order_lineKey key;
> public order_lineKey key()
> {
> if(key == null)
> key = new order_lineKey(ol_w_id, ol_d_id, ol_o_id,
> ol_number);
>
> return key;
>
> }
> @Override
> public void readBinary(BinaryReader reader) throws
> BinaryObjectException {
> // TODO Auto-generated method stub
> ol_o_id = reader.readInt("ol_o_id");
> ol_d_id = reader.readInt("ol_d_id");
> ol_w_id = reader.readInt("ol_w_id");
> ol_number = reader.readInt("ol_number");
> ol_i_id = reader.readInt("ol_i_id");
> ol_supply_w_id = reader.readInt("ol_supply_w_id");
> ol_delivery_d = reader.readString("ol_delivery_d");
> ol_quantity = reader.readInt("ol_quantity");
> ol_amount = reader.readDouble("ol_amount");
> ol_dist_info = reader.readString("ol_dist_info");
>
>
> }
> @Override
> public void writeBinary(BinaryWriter writer) throws
> BinaryObjectException {
> // TODO Auto-generated method stub
>
> writer.writeInt("ol_o_id",ol_o_id);
> writer.writeInt("ol_d_id",ol_d_id);
> writer.writeInt("ol_w_id",ol_w_id);
> writer.writeInt("ol_number",ol_number);
> writer.writeInt("ol_i_id",ol_i_id);
> writer.writeInt("ol_supply_w_id",ol_supply_w_id);
> writer.writeString("ol_delivery_d",ol_delivery_d);
> writer.writeInt("ol_quantity",ol_quantity);
> writer.writeDouble("ol_amount", ol_amount);
> writer.writeString("ol_dist_info",ol_dist_info);
>
> }
>
> }
>
>
> Is this the correct way for compact binary representation of data? Because
> I
> am not getting any improvement in memory consumption after using this
> interface. Also, in readBinary() and writeBinary() methods do we have to
> write writer.writeInt/String/Double and reader.readInt/String/Double
> methods
> for every field or only fields participating in SQL queries(Does this have
> any effect on memory consumption)?
>
>
>
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Binarylizable-interface-in-apache-ignite-tp9078.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Memory consumption in apache ignite

[Dmitriy Karachentsev]

You don't have to deploy your data classes on server, because it may work
with binary types. Just configure cache as usual but use QueryEntity[1] for
that. On client side you may put and query for data as you did before.

[1]:
http://apacheignite.gridgain.org/docs/sql-queries#configuring-sql-indexes-using-queryentity

On Fri, Nov 18, 2016 at 1:23 PM, rishi007bansod 
wrote:

> I have measured database size on disk 370MB
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Memory-consumption-in-apache-ignite-tp9035p9073.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Memory consumption in apache ignite

[Dmitriy Karachentsev]

For sure, if you disable indexing SQL query will not be available. It was
just recommendation if it's suitable for you.
You may try as Vlad suggested: use server node without data classes, so it
will not be able to cache deserialized values, and run your application on
client node.

On Fri, Nov 18, 2016 at 7:21 AM, rishi007bansod 
wrote:

> But now with indexing types disabled I am getting following error,
>
> Exception in thread "main" javax.cache.CacheException: Indexing is disabled
> for cache: order_line_cache. Use setIndexedTypes or setTypeMetadata methods
> on CacheConfiguration to enable.
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxy.validate(
> IgniteCacheProxy.java:732)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxy.query(
> IgniteCacheProxy.java:664)
> at data_test.order(data_test.java:2903)
> at data_test.main(data_test.java:2839)
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Memory-consumption-in-apache-ignite-tp9035p9064.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Memory consumption in apache ignite

[Dmitriy Karachentsev]

Yes, in your case Ignite stores deserialized values as well. You may try to
avoid it by switching copy on read to true, remove indexed types and
disable peer class loading if it's set.

On Thu, Nov 17, 2016 at 4:05 PM, rishi007bansod 
wrote:

> I have used single node. My cache configuration is,
> Cache configuration I have used is,
>
> CacheConfiguration ccfg_order_line = new
> CacheConfiguration<>();
> ccfg_order_line.setIndexedTypes(order_lineKey.class,
> order_line.class);
> ccfg_order_line.setName("order_line_cache");
> ccfg_order_line.setCopyOnRead(false);
> ccfg_order_line.setMemoryMode(CacheMemoryMode.ONHEAP_TIERED);
> ccfg_order_line.setSwapEnabled(false);
> ccfg_order_line.setBackups(0);
> IgniteCache cache_order_line =
> ignite.createCache(ccfg_order_line);
>
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Memory-consumption-in-apache-ignite-tp9035p9044.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Memory consumption in apache ignite

[Dmitriy Karachentsev]

Hi.

How many nodes do you use? Could you provide configuration, please?

On Thu, Nov 17, 2016 at 11:50 AM, rishi007bansod 
wrote:

> For 370MB of data ignite is consuming about 3GB space in memory(ON HEAP
> MODE). Following is heap dump I got. What memory optimizations can be
> applied? Does ignite stores object by default in both serialized and
> deserialized formats?
>
> 
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Memory-consumption-in-apache-ignite-tp9035.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Restarting Tomcat container and Rebalancing

[Dmitriy Karachentsev]

Hi.

You can use events to listen for rebalance completeness:

ignite.events().localListen(new IgnitePredicate() {
@Override public boolean apply(Event evt) {
CacheRebalancingEvent evt = (CacheRebalancingEvent)e;

// do some useful stuff

return true;
}
}, EventType.EVT_CACHE_REBALANCE_STOPPED);

On Thu, Nov 17, 2016 at 8:03 AM, styriver  wrote:

> We are running ignite inside a tomcat container. We have two tomcat
> instances
> on two different servers each tomcat instance is a server node. In our
> corporate turnover workflow we stop and restart each tomcat instance in
> sequence. Example:
> Deploy new war
> restart tomcat instance on server A (ignite server node)
> Deploy new war
> restart tomcat instance on server B (ignite server node)
>
> There is no manual step this sequence is all automated. What I am seeing is
> on node shutdown and restart we lose cache entries. I am assuming this is
> happening because a cache rebalance is taking place and because this is
> automated we are not allowing any time for the rebalance to finish. So
>
> Server A restarts and starts to rebalance it's cache from Server B. But
> before the rebalance finishes we have moved onto restarting Server B.
>
> This is how I am shutting down currently:
> Ignition.stop(ignite.name(), false);
>
> Is there anything I can do to ensure that before Server B restarts the
> rebalance is complete?
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Restarting-Tomcat-container-
> and-Rebalancing-tp9033.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Connection Cleanup when IgniteCallable Killed

[Dmitriy Karachentsev]

Hi Jaime.

Actually tasks are interrupted, you can find how it works in following
code: (for Ignite 1.7) in GridJobProcessor.onEvent() calls
GridJobProcessor.cancelJob(GridJobWorker, boolean) for each task that leads
to interruption.

If there down on your code was caught InterruptedException but not called
Thread.currentThread().interrupt() the interrupted flag will be false.
Please check if you can get exceptions that have InterruptedException in
their cause and close your resources properly.

If it's not possible, you may use a workaround, for example, pass to
callable client's ClusterNode and check periodically if it's alive:

// Call on client side before broadcasting
final ClusterNode locNode = ignite.cluster().localNode();

// Check in callable if it's not null
if (Ignition.localIgnite().cluster().forNode(locNode).node() == null)
  // Close resources

But it's better to handle interruption correctly, because task may be
cancelled due to various reasons.

Hope that helps.

On Fri, Nov 11, 2016 at 6:59 AM, jaime spicciati 
wrote:

> All,
> I currently broadcast an IgniteCallable in my cluster which opens
> connections to various resources, specifically Zookeeper via Apache
> Curator.
>
> If the originating node (the client that launched the IgniteCallable) is
> stopped prematurely I see that Ignite will rightfully cancel the broadcast
> call within the cluster. This is all great but Apache Curator has a thread
> in the background watching Zookeeper. So when Ignite stops the
> IgniteCallable in the cluster the connection to Zookeeper is still open
> which is keeping ephemeral nodes from being deleted.
>
> I tried implementing logic to handle thread interrupts to close the
> zookeeper connection but it doesn't look like IgniteCallable is cancelled
> through interrupts. I looked through the Ignite code base and can't quite
> figure out how it is cancelling my IgniteCallable so that I can hook into
> the IgniteCallable life cycle.
>
> Long story short, how do I do resource/connection cleanup in an
> IgniteCallable when the client disconnects ungracefully, and the connection
> is held by a thread launched from within the IgniteCallable?
>
> Thanks
>

Re: java.lang.ClassNotFoundException: Failed to peer load class

[Dmitriy Karachentsev]

Hi Alex.
It looks like problem on client, because it was disconnected from server
after try to retrieve class. Please, check if there are no other errors on
client.

Could you, please, repeat your test with -DIGNITE_QUIET=false and provide
full client and server logs?

Thanks!

On Wed, Nov 9, 2016 at 4:38 PM, alex  wrote:

> sorry for that. last post not add to mailing list. So I post it again and
> subscribe to mailing list.
> And the new one give more detail information.
> Thank for replying @vdpyatkov
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/java-lang-ClassNotFoundException-Failed-
> to-peer-load-class-tp8811p8831.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>