Low self-descriptiveness of query exceptions.

[Vasiliy Sisko]

Hello Igniters.

When I execute query: SELECT * FROM “PersonCache”.Person2 an exception with
the next text is thrown: “Error: Failed to parse query: ...”.
That message does not give information about cause of exception.
Possible message of exception should be like: “Table Person2 does not
exists”

Is it possible to support such functionality in query engine?

-- 
Vasiliy Sisko
GridGain Systems
www.gridgain.com

Re: Two-phase commit failure recovery

[alexey.goncharuk]

Hi,

Note that the step described in the blog happens when a node that originated
a transaction crashes. In this case:
1. Ignite will reload the relevant data automatically
2. Not sure what you mean here. From a cluster point of view this happens
when a node leaves grid and a corresponding part of recovery protocol is
triggered. There is no need to do it manually
3. At the start of prepare step all the keys participating in the
transaction are known, so Ignite will reload only participating keys

Hope this helps,
AG



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Two-phase-commit-failure-recovery-tp9040p9066.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Memory consumption in apache ignite

[rishi007bansod]

But now with indexing types disabled I am getting following error,

Exception in thread "main" javax.cache.CacheException: Indexing is disabled
for cache: order_line_cache. Use setIndexedTypes or setTypeMetadata methods
on CacheConfiguration to enable.
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.validate(IgniteCacheProxy.java:732)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.query(IgniteCacheProxy.java:664)
at data_test.order(data_test.java:2903)
at data_test.main(data_test.java:2839)




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Memory-consumption-in-apache-ignite-tp9035p9064.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Restarting Tomcat container and Rebalancing

[styriver]

Hi So what I have implemented are two counters. One counter tracks the
rebalance start event and one counter tracks the rebalance stop events. I
start the stop counter after the first rebalance start event. On shutdown I
compare the counters when they are equal I am assuming rebalance is complete
and I complete shutdown. This appears to be working with my load tests. Is
this the best solution? Or is their a more straightforward way to know that
the rebalance has completed between the two server nodes?




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Restarting-Tomcat-container-and-Rebalancing-tp9033p9063.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Apache Spark & Ignite Integration

[vkulichenko]

a. Actually, I'm looking at the code and in doubts now. Did you make any
experiments? What do they show?
b. Correct. sql() method goes directly to Ignite bypassing Spark and this is
basically the same as calling IgniteCache.query().
c. savePairs() is a distributed operation on a Spark cluster, so this will
not work. To use transactions you will have to use Ignite API.

To be honest, I'm not quite sure you should use IgniteRDD because it sounds
like your use case is bigger than that (indexed SQL, transactions, etc.).
Did you consider using pure Ignite API without integrating it with Spark?

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Apache-Spark-Ignite-Integration-tp8556p9062.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Explicit lock whithin a transaction

[vkulichenko]

If you don't know whether you're going to update in advance, how do you know
whether to lock? If you want to read outside of transaction somewhere in the
middle, you can use IgniteCache.getAllOutTx(..) method.

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Explicit-lock-whithin-a-transaction-tp8586p9061.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: How fit Apache Ignite Distributed Cron job currently alike Quartz Scheduler

[vkulichenko]

No, you will have to implement your own JobStore.

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/How-fit-Apache-Ignite-Distributed-Cron-job-currently-alike-Quartz-Scheduler-tp9017p9060.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Explicit lock whithin a transaction

[akaptsan]

Yes I read and don't update some other objects.
It would be very difficult to read objectsoutside of transaction. I need to
execute qute complicated logic within transacttion. During the execution I
can read/update many objects depending on conditions, settings and so on. I
can't know which objects I need to read until I come to certain point within
transaction.

In SQL I have following options:
- read without lock - select ... 
- read and lock - select ... for update





--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Explicit-lock-whithin-a-transaction-tp8586p9059.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Explicit lock whithin a transaction

[vkulichenko]

So what is read-only in this example? You're updating the account balance
after you read it - this is a classic use case for REPEATABLE_READ. If there
is something else that you read, but don't update, I would just do this
outside of transaction. Makes sense?

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Explicit-lock-whithin-a-transaction-tp8586p9058.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Can java code look up cache in server without Ignition.start(cfg)

[vkulichenko]

Saji,

This is weird, I though there are several versions of it. You can use
Ignition.loadSpringBean() to work around this. Load the IgniteConfiguration
bean from the XML and then pass it getOrStart.

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Can-java-code-look-up-cache-in-server-without-Ignition-start-cfg-tp9008p9057.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Order of transaction commit, cache update and event listener notification

[Alexey Goncharuk]

Hi,

Currently SQL queries do not participate in transactions in any way, so you
can see partially committed data from other transactions.

In other words, if a thread 1 updates keys 1, 2, 3, 4 and started
transaction commit, and thread 2 issues an SQL query, this query may see
keys 1, 2 updated and keys 3, 4 - not.

As Vlad mentioned, there is a ticket for supporting transactional SQL [1]
which you can track (and join discussion regarding the design, btw).

Hope this helps,
AG


[1] https://issues.apache.org/jira/browse/IGNITE-3478

2016-11-17 20:34 GMT+03:00 newbie :

> Thanks for the info and link.
>
> Just an add-on question based on the link:
>
> Do the sql queries really see another transaction's dirty data with
> OPTIMISTIC, REPEATABLE_READ ?
>
> From my testing it seems like if we are writing from transaction T1 and
> reading from a different thread in transaction T2, I don't really see Sql
> query returning uncommitted data from T1. Have I missed something here ?
>
>
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Order-of-transaction-commit-cache-
> update-and-event-listener-notification-tp9018p9055.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Order of transaction commit, cache update and event listener notification

[newbie]

Thanks for the info and link.

Just an add-on question based on the link: 

Do the sql queries really see another transaction's dirty data with
OPTIMISTIC, REPEATABLE_READ ? 

>From my testing it seems like if we are writing from transaction T1 and
reading from a different thread in transaction T2, I don't really see Sql
query returning uncommitted data from T1. Have I missed something here ?






--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Order-of-transaction-commit-cache-update-and-event-listener-notification-tp9018p9055.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Order of transaction commit, cache update and event listener notification

[vdpyatkov]

Hi,

Ignite does not have snapshot isolation in transaction. Although community
seriously wants to implement it [1].

If you are want something like, I think, you can add Date field (make index
by it) into object and execute SQL by concrete timestamp.

[1]: https://issues.apache.org/jira/browse/IGNITE-3484



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Order-of-transaction-commit-cache-update-and-event-listener-notification-tp9018p9053.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Explicit lock whithin a transaction

[akaptsan]

PESSIMISTIC-READ_COMMITTED locks ALL objects I read in transaction. It's
overkill 
I need to readAndLock for some objects and readWithoutLock for others



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Explicit-lock-whithin-a-transaction-tp8586p9054.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Can java code look up cache in server without Ignition.start(cfg)

[StartCoding]

Thanks Very much!

That exactly matched my logic and it is perfectly working.

I noticed that getOrStart takes IgniteConfiguration object.Is there a way
to create that from example-ignite.xml file rather than doing it java
setters?

Thanks
Saji

On Wed, Nov 16, 2016 at 3:09 PM, vkulichenko [via Apache Ignite Users] <
ml-node+s70518n9028...@n6.nabble.com> wrote:

> Saji,
>
> Ignition.ignite() is used to get the already started Ignite instance.
> There is also Ignition.getOrStart() method that actually implements the
> logic you described, so you can just use it.
>
> -Val
>
> --
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-ignite-users.70518.x6.nabble.com/Can-java-
> code-look-up-cache-in-server-without-Ignition-start-cfg-tp9008p9028.html
> To unsubscribe from Can java code look up cache in server without
> Ignition.start(cfg), click here
> 
> .
> NAML
> 
>



-- 
Thanks and Regards
Saji




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Can-java-code-look-up-cache-in-server-without-Ignition-start-cfg-tp9008p9052.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Explicit lock whithin a transaction

[vdpyatkov]

Hi,

I recoment to use PESSIMISTIC, READ_COMMITTED (without any locks)
transaction for each account.  And look at the comment on Val[1].

You are doing somtink like this, as I understand from first post.

Why is this method is not suitable?

[1]:
http://apache-ignite-users.70518.x6.nabble.com/Explicit-lock-whithin-a-transaction-tp8586p8593.html



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Explicit-lock-whithin-a-transaction-tp8586p9051.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Memory consumption in apache ignite

[Dmitriy Karachentsev]

Yes, in your case Ignite stores deserialized values as well. You may try to
avoid it by switching copy on read to true, remove indexed types and
disable peer class loading if it's set.

On Thu, Nov 17, 2016 at 4:05 PM, rishi007bansod 
wrote:

> I have used single node. My cache configuration is,
> Cache configuration I have used is,
>
> CacheConfiguration ccfg_order_line = new
> CacheConfiguration<>();
> ccfg_order_line.setIndexedTypes(order_lineKey.class,
> order_line.class);
> ccfg_order_line.setName("order_line_cache");
> ccfg_order_line.setCopyOnRead(false);
> ccfg_order_line.setMemoryMode(CacheMemoryMode.ONHEAP_TIERED);
> ccfg_order_line.setSwapEnabled(false);
> ccfg_order_line.setBackups(0);
> IgniteCache cache_order_line =
> ignite.createCache(ccfg_order_line);
>
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Memory-consumption-in-apache-ignite-tp9035p9044.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Memory consumption in apache ignite

[vdpyatkov]

Hi,

How are you measure size of data (370MB)?
If you use ONHEAP_TIERED mode, entries can to store on in state (serialized
and deserialized), look at issues [1, 2], but you can avoid this behavior,
if server does not have object classes.

[1]: https://issues.apache.org/jira/browse/IGNITE-3347
[2]: https://issues.apache.org/jira/browse/IGNITE-2417



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Memory-consumption-in-apache-ignite-tp9035p9049.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: SELECT with ORDER BY and pagination

[akaptsan]

How pagination works with ORDER BY? 
Shouldn't client node gets results from all nodes, merge them and re-sort? 



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/SELECT-with-ORDER-BY-and-pagination-tp9037p9048.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Apache Spark & Ignite Integration

[pragmaticbigdata]

Appreciate your follow ups.

a. " Data is stored in Ignite and Spark will fetch data for a particular
partition when you execute something." Does IgniteRDD (i.e. Spark) fetch the
data to the closest Spark node that probably resides on the same server? One
of the earlier responses mention that this is done when new entries are
added to the cache.

a1. Could you please detail on how #a is achieved? I looked at the
IgniteRDD.compute() method implementation which creates a ScanQuery and
makes a call to the affinity api but I didn't follow how does the code
search for the closest ignite node?

b. For igniteRDD.sql() query execution, it seems that the behavior and hence
the performance would be same as executing a sql query on the IgniteCache
from an ignite client node. Is my understanding right? I follow the fact
that the performance would be better when compared to a similar spark SQL
query because of the in-memory indexes.

c. How can I take the advantage of Ignite's ACID transaction support when
doing the data processing in spark? Based on one of the earlier points the
code flow would look like
 val sharedRDD1: IgniteRDD[Int,Int] = ic.fromCache("partitioned")
 val sharedRDD2: IgniteRDD[Int,Int] = ic.fromCache("anotherCache")

 Transaction tx = Ignition.ignite().transactions().txStart()
 sharedRDD1.savePairs(...);
 sharedRDD2.savePairs(...);
 tx.commit()

Is my understanding of the flow correct? If so, how do I maintain
transaction isolation when other spark jobs try to read the data from ignite
in parallel?

Thanks.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Apache-Spark-Ignite-Integration-tp8556p9047.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: SELECT with ORDER BY and pagination

[vdpyatkov]

Hi,

Lock at the method (o.a.i.cache.query.SqlFieldsQuery#setPageSize), that
allow to set  sql page size.
If you are going through cursor (result of cache.query()) or executing
"getAll" then you get all result data.

Client gets data into page range, if used cursor (and sent request to the
next part of data if needed).

[1]: https://apacheignite.readme.io/docs/sql-queries



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/SELECT-with-ORDER-BY-and-pagination-tp9037p9046.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: SQL query against cache with read-through CacheStore

[vdpyatkov]

Hi,

Yes, you are right.
You should preload[1] data before SQL will execute.

[1]: https://apacheignite.readme.io/docs/data-loading



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/SQL-query-against-cache-with-read-through-CacheStore-tp9039p9045.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Memory consumption in apache ignite

[rishi007bansod]

I have used single node. My cache configuration is,
Cache configuration I have used is, 

CacheConfiguration ccfg_order_line = new
CacheConfiguration<>(); 
ccfg_order_line.setIndexedTypes(order_lineKey.class, order_line.class); 
ccfg_order_line.setName("order_line_cache"); 
ccfg_order_line.setCopyOnRead(false); 
ccfg_order_line.setMemoryMode(CacheMemoryMode.ONHEAP_TIERED); 
ccfg_order_line.setSwapEnabled(false); 
ccfg_order_line.setBackups(0); 
IgniteCache cache_order_line =
ignite.createCache(ccfg_order_line); 





--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Memory-consumption-in-apache-ignite-tp9035p9044.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Memory consumption in apache ignite

[Dmitriy Karachentsev]

Hi.

How many nodes do you use? Could you provide configuration, please?

On Thu, Nov 17, 2016 at 11:50 AM, rishi007bansod 
wrote:

> For 370MB of data ignite is consuming about 3GB space in memory(ON HEAP
> MODE). Following is heap dump I got. What memory optimizations can be
> applied? Does ignite stores object by default in both serialized and
> deserialized formats?
>
> 
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Memory-consumption-in-apache-ignite-tp9035.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Order of transaction commit, cache update and event listener notification

[newbie]

Hi Val,

Thanks!

>From my understanding and testing, seems like REPEATABLE_READ returns the
same value (for a key) when read multiple times within the read transaction.
But we need snapshot isolation across many keys. 

So say we start the read transaction, we would like to see all entries as
available at the start and *not * reflect changes coming from concurrent
transaction commits. 

Our use case involves reading a different (DB) cursor and use the Ignite
cache to filter/enrich that data. As we read the DB cursor we do a get on
the Ignite cache. Let's say we are interested in K1, K2 in the cache, it is
possible that as we iterate and read K1 (and haven't read K2 yet), K2 was
updated by a different Ignite transaction and we don't want that. We want to
see the state of the cache as it was at the beginning of the read
transaction. Is there anything available in Ignite for the same?

I was trying to debug but am not that clear - do the CacheEvent listeners
get notified before the transaction commit changes are available ? or the
otherway round.





--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Order-of-transaction-commit-cache-update-and-event-listener-notification-tp9018p9042.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Restarting Tomcat container and Rebalancing

[Dmitriy Karachentsev]

Hi.

You can use events to listen for rebalance completeness:

ignite.events().localListen(new IgnitePredicate() {
@Override public boolean apply(Event evt) {
CacheRebalancingEvent evt = (CacheRebalancingEvent)e;

// do some useful stuff

return true;
}
}, EventType.EVT_CACHE_REBALANCE_STOPPED);

On Thu, Nov 17, 2016 at 8:03 AM, styriver  wrote:

> We are running ignite inside a tomcat container. We have two tomcat
> instances
> on two different servers each tomcat instance is a server node. In our
> corporate turnover workflow we stop and restart each tomcat instance in
> sequence. Example:
> Deploy new war
> restart tomcat instance on server A (ignite server node)
> Deploy new war
> restart tomcat instance on server B (ignite server node)
>
> There is no manual step this sequence is all automated. What I am seeing is
> on node shutdown and restart we lose cache entries. I am assuming this is
> happening because a cache rebalance is taking place and because this is
> automated we are not allowing any time for the rebalance to finish. So
>
> Server A restarts and starts to rebalance it's cache from Server B. But
> before the rebalance finishes we have moved onto restarting Server B.
>
> This is how I am shutting down currently:
> Ignition.stop(ignite.name(), false);
>
> Is there anything I can do to ensure that before Server B restarts the
> rebalance is complete?
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Restarting-Tomcat-container-
> and-Rebalancing-tp9033.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Two-phase commit failure recovery

[akaptsan]

I have read the block
http://gridgain.blogspot.co.uk/2014/09/two-phase-commit-for-in-memory-caches.html.
It is great! But I still have questions.

/
Whenever we cannot tell whether the database commit had happened or not, we
can simply reload the relevant data from database into cache upon committing
the transaction.  
/
1. Can Ignate reload the relevant data automaticaly or we must take care of
it? 
2. How can we detect this situation? 
3. Should we reload the whole cache or we can somehow see which objects must
be removed/reloaded?




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Two-phase-commit-failure-recovery-tp9040.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

SQL query against cache with read-through CacheStore

[akaptsan]

Could you please explain what happens when I execute a query against cache
which is not fully loaded into Ignate RAM? 
I suppose result will be not consisent with underlying persistent storage?




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/SQL-query-against-cache-with-read-through-CacheStore-tp9039.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: How fit Apache Ignite Distributed Cron job currently alike Quartz Scheduler

[Navneet Kumar]

Hi 
Current Use Case : Whenever a service wants to start a Cron for a
subscriber, It would give a request to  distributed Cron Framework , The
framework let’s say in our case Quartz framework should use a distributed
in-memory Cache as its JobStore. 

The distributed in-memory cache system should support automatic job
failover. In case of a node crash, jobs are automatically transferred to
other available nodes for re-execution
Before reaching to JobStore (distributed Cache system), If my request
terminated due to any reason, the framework should return the proper message
back to me.
While reading some records from JobStore if in between I got disconnected
from the node, so how the cluster will we take care of those half cooked
data.
The same request should not be processed twice in case JobStore deployed in
active-active mode.
We need to write our own implementation class by overriding Scheduler’s
JobStore interface.

The above mentioned use cases can be fullfilled by using apache Ignite
provided Cron Scheduler?




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/How-fit-Apache-Ignite-Distributed-Cron-job-currently-alike-Quartz-Scheduler-tp9017p9038.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

SELECT with ORDER BY and pagination

[akaptsan]

Let's say I execute 
/
select ... from HUGE_REPLICATED_CACHE order by ...
/
Then I fetch few first record of the result cursor

How does it work with pagination? How much RAM do I need at client?




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/SELECT-with-ORDER-BY-and-pagination-tp9037.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Explicit lock whithin a transaction

[akaptsan]

Example: 
1. Read and lock a financial account
2. Check if account balance is enought
3. Check other conditions
4. Move funds
5. Update balance 
6. Commit

I have to be sure that account balance will not be changed between 1 and 5




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Explicit-lock-whithin-a-transaction-tp8586p9036.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Memory consumption in apache ignite

[rishi007bansod]

For 370MB of data ignite is consuming about 3GB space in memory(ON HEAP
MODE). Following is heap dump I got. What memory optimizations can be
applied? Does ignite stores object by default in both serialized and
deserialized formats?

  



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Memory-consumption-in-apache-ignite-tp9035.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.