Re: Re: BinaryObject Data Can Not Mapping To SQL-Data

[Vladimir Tchernyi]

Hello Huty,
please read my post [1]. The approach in that paper works successfully in
production for more than one year and seems to be correct
[1]
https://www.gridgain.com/resources/blog/how-fast-load-large-datasets-apache-ignite-using-key-value-api

Vladimir
telegram @vtchernyi

PS
hope I named you correct, the name is not widespread here in Russia

вт, 19 апр. 2022 г. в 09:46, y :

> Hi Vladimir,
> Thank you for your answer.  Emmm.Actually, most of my methods are the
> same as yours except for the following two points：
> 1、I didn't use ComputeTask. The data is sent to the server node through
> the* thin client.*
>
> 2、I didn't use the standard POJO. Key-type is the following code and
> value-type is an empty class. That means *All columns are dynamically
> specified through BinaryObjectBuilder. *
>
> public class PubPartionKeys_1_7 {
> @AffinityKeyMapped
> private String TBDATA_DX01;
> private String TBDATA_DX02;
> private String TBDATA_DX03;
> private String TBDATA_DX04;
> private String TBDATA_DX05;
> private String TBDATA_DX06;
> private String TBDATA_DX07;
>
> public PubPartionKeys_1_7() {
> }
>
> // get/set method
> // .
> }
>
> I would be appreciate it very much if you attach your code back！ ：）
>
> Huty，
> 2022/4/19
>
>
> At 2022-04-19 12:40:20, vtcher...@gmail.com wrote:
>
> Hi,
>
> I have had the same experience without sql, using KV API only. My cluster
> consists of several data nodes and self-written jar application that starts
> the client node. When started, client node executes mapreduce tasks for
> data load and processing.
>
> The workaround is as follows:
> 1. create POJO on the client node;
> 2. convert it to the binary object;
> 3. on the data node, get binary object over the network and get its
> builder (obj.toBuilder());
> 4. set some fields, build and put in the cache.
>
> The builder on the step 3 seems to be the same as the one on the cluent
> node.
>
> Hope that helps,
> Vladimir
>
> 13:06, 18 апреля 2022 г., y :
>
> Hi ,
> When using binary to insert data, I need to  get *an
> exist BinaryObject/BinaryObjectBuilder*  from the database, similar to
> the code below.
> 442062c6$3$1803c222cba$Coremail$hty1994712$163.com
>
> If I create a BinaryObjectBuilder directly, inserting binary data does not
> map to table data. The following code will not throw error, but the* data
> is not mapped to sql. *If there is *no data in my table at first*, how
> can I insert data?
> 3ecbd8f9$4$1803c222cba$Coremail$hty1994712$163.com
>
>
>
>
>
>
>
>
> --
> Отправлено из мобильного приложения Яндекс Почты
>
>
>
>
>

rendezvous partitions on the web

[Vladimir Tchernyi]

Thanks,

I'd fount that already in my recent project repo. The question is about the
website, I did not manage to find that info there.
"how to set non default partition number for some cache" - do we have
java/xml example at ignite.apache.org website?

Vladimir

PS: sorry for missing thread name, just changed it

13:49, 23 ноября 2021 г., andrei :

Hi,

You can set it as part of CacheConfiguration:

https://www.javadoc.io/doc/org.apache.ignite/ignite-core/1.6.0/org/apache/ignite/configuration/CacheConfiguration.html#setAffinity(org.apache.ignite.cache.affinity.AffinityFunction)








Regards,
Andei

11/23/2021 1:39 PM, vtcher...@gmail.com пишет:

Hi community,

I try to remember how to set non default partition number for some cache.
Do we have java/xml example at ignite.apache.org website?

Vladimir



-- 
Отправлено из мобильного приложения Яндекс.Почты

Re: Regarding ignite query result set limit

[Vladimir Pligin]

Hi,

It's possible that you are looking for the lazy SQL flag.
More details:
https://ignite.apache.org/docs/latest/perf-and-troubleshooting/sql-tuning#lazy-loading
It can also be set on the JDBC level.
https://ignite.apache.org/docs/latest/SQL/JDBC/jdbc-driver#parameters



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Peer ClassLoading Issue | Apache Ignite 2.10 with Spring Boot 2.3

[Vladimir Tchernyi]

Hi Siva,

I still have no clue what's wrong, your code seems OK. Just one more thing
to check.

>>  cacheConfiguration.setIndexedTypes(String.class, IgniteUser1.class);
what do you need this string for? Does your code work without this string?

It seems to be something about SQL layer. Here's a little post I wrote [1],
it is written in Russian, hope you will be able to translate and read it.
I'd spent a lot of time forging through StackOverflow and learned that user
POJOs are not peer-deployed at all. So if you need  IgniteUser1 class,
probably that class should extend/implement something from Ignite platform.

--
Vladimir

[1] https://habr.com/ru/post/472568/

вт, 11 мая 2021 г. в 06:00, :

> Thanks Vladimir Chernyi for your blog.  That helped a lot in improving
> the performance. I am converting the object to Binary Object before
> storing. This doesn’t seem to be issue just when loading the data alone.
>
>
>
> I have a scan query which also fails with same exception.
>
>
>
> List cacheObjects = cache.withKeepBinary().query(new
> ScanQuery<>(
> //filter for just Blade Enabled Users
> new IgniteBiPredicate() {
> @Override
> public boolean apply(String s, BinaryObject
> binaryObject) {
> return ((HashSet) binaryObject.field("field1"
> )).contains("34");
> }
> }),
> // Transformer
> new IgniteClosure, BinaryObject>()
> {
> @Override
> public BinaryObject apply(Cache.Entry
> stringBinaryObjectEntry) {
> return stringBinaryObjectEntry.getValue();
> }
> }).getAll();
>
> Weirdest thing is that the classes seem to be loading on the cluster, only
> when i modify something on the class. I can replicate it again when I start
> the cluster wiping the data  folder.  It appears so that the grid deployer
> is not deploying classes to the cluster until it sees that the class has
> changed , but shouldn’t it deploy first time?.
>
>
>
>
>
> Thanks,
>
> Siva.
>
>
>
> *From:* vtcher...@gmail.com 
> *Sent:* Monday, May 10, 2021 1:07
> *To:* user@ignite.apache.org
> *Subject:* Re: Peer ClassLoading Issue | Apache Ignite 2.10 with Spring
> Boot 2.3
>
>
>
> This message originated from outside our organisation and is from web
> based email - vtcher...@gmail.com
>
> Hi Siva,
>
>
>
> Thank you for reading my blog post. I have no idea what is the problem in
> your case, just wanna share some experience.
>
>
>
> I do not use any user POJOs on the remote nodes. Instead, I create POJO on
> the thick client node, convert it in BinaryObject and change that object on
> the remote node by object.toBuilder().setField().build(). I use key value
> API only.
>
>
>
> So no class not found issues arise. Hope that helps
>
>
>
> Vladimir Chernyi
>
> 8:28, 9 мая 2021 г., "siva.velich...@barclays.com" <
> siva.velich...@barclays.com>:
>
> Hi,
>
>
>
> We are trying to use ignite for the first time in our project. We are
> trying to use ignite with persistence enabled.
>
>
>
> Architecture is as follows.
>
>
>
> SpringBoot 2.3 application (thick client ) tries to connect to apace
> ignite cluster (3 nodes ) with persistence enabled and peer class loading
> enabled.
>
>
>
> There seems to be a weird  issue with peer class loading.
>
>
>
> We are trying to load huge data following the same approach as here -
> https://www.gridgain.com/resources/blog/how-fast-load-large-datasets-apache-ignite-using-key-value-api
> <https://clicktime.symantec.com/33sZ5STPWrhi61h4sP3SJYy6H2?u=https%3A%2F%2Fwww.gridgain.com%2Fresources%2Fblog%2Fhow-fast-load-large-datasets-apache-ignite-using-key-value-api>
>
>
>
> Cache Configuration
>
>
>
> cacheConfiguration.setName(CacheIdentifiers.*USER_IGNITE_CACHE*
> .toString());
> cacheConfiguration.setIndexedTypes(String.class, IgniteUser1.class);
> cacheConfiguration.setCacheMode(CacheMode.*PARTITIONED*);
> cacheConfiguration.setStoreKeepBinary(true);
> RendezvousAffinityFunction rendezvousAffinityFunction = new
> RendezvousAffinityFunction();
> rendezvousAffinityFunction.setPartitions(512);
> cacheConfiguration.setBackups(1);
> cacheConfiguration.setAffinity(rendezvousAffinityFunction);
>
>
>
>
>
> Scenario 1.
>
>
>
> Start the cluster à activate the cluster à start the thick client à
>  Loading clients/ignite.cluster fails
>
>
>
> Exception o

Re: Maximising online ml performance

[Vladimir Pligin]

+ zaleslaw.sin@ Hi, It seems that for some reason we've missed this thread (something with nabble maybe?).Maybe we could ask Alexey Zinoviev to have a look at this one if he is available. Thanks a lot in advance. 13.03.2021, 16:36, "rothnorton" :After reading the examples etc, more generally, i have two questions:1) What is the best way to stream json data into a machine learningalgorithm to minimise latency of receiving date to the result.Including serialisation into cache from json for vector representation forml algs, cache setup, datastore style etc. Its unclear from thedocumentation about this.I'm happy to benchmark different methods, but maybe someone already knows?2) How does columnstore ml compare to row based ml, and if its better forthings like linear regression, is there a comparison?And for both of these, what is the optimal setup (if any) for minimisinglatency for realtime machine learning, and can the apache igniteinfrastructure be modified to achieve this? --Sent from: http://apache-ignite-users.70518.x6.nabble.com/  -- Warm Regards, Vladimir Pligin 

[Webinar] What’s new in Apache Ignite 2.10

[Vladimir Pligin]

Hi Igniters,

Tomorrow I'm going to present new features of Apache Ignite 2.10.
It'll be a webinar with a demo session. I'll show how to leverage new
functionality.

Please join us tomorrow, it's scheduled for the 20th April at 10:00 AM PDT.
More details:
https://www.gridgain.com/resources/webinars/whats-new-in-apache-ignite-210.

See you folks there.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: very fast loading of very big table

[Vladimir Tchernyi]

Pavel,

thanks for mentioning the patterns. Of course, I spent a lot of time
reading documentation, [2] at the very beginning and [1] a couple of months
ago. Here is the origin of my pain-in-the-neck about a complete GitHub
example - none of [1] and [2] give an answer about my problem. The keyword
in my case is ASAP, there should be a multithreaded example. Of course, the
real-world example must not use a primitive types as a cache values, I
tried to illustrate that in [3].

I'd built one with the data streamer [1], it seems I was limited by network
adapter performance (see my previous post in this thread). That is the
reason I decided to move SQL queries to the data nodes.

Vladimir

пт, 26 февр. 2021 г. в 10:54, Pavel Tupitsyn :

> Vladimir,
>
> I think all real-world use cases are very valuable to the community.
> However, we should be careful to avoid misleading conclusions.
>
> We have well-known patterns for loading data from other systems:
> DataStreamer [1] and CacheStore [2].
> The article [3] seems a bit confusing to me, since none of those two
> patterns are mentioned there.
> When proposing a custom approach, it would be great to compare it to the
> standard alternatives.
>
> [1] https://ignite.apache.org/docs/latest/data-streaming
> [2] https://ignite.apache.org/docs/latest/persistence/custom-cache-store
> [3]
> https://www.gridgain.com/resources/blog/how-fast-load-large-datasets-apache-ignite-using-key-value-api
>
> On Fri, Feb 26, 2021 at 9:19 AM Vladimir Tchernyi 
> wrote:
>
>> Hi Pavel,
>>
>> the code [1] you shared is a kind of in-memory experiment with all the
>> processes inside a single JVM. My work differs - it is from the big retail
>> business, and hence it is 100% practice-oriented. True to say, it's
>> oriented to the state of things inside my company, and that is my question
>> - will my results be interesting to the community? I have seen a lot of
>> questions on the user list regarding data loading and difficulties here
>> seem to be a blocker in extending Ignite's popularity.
>>
>> Please let me know if my case is not common in the industry. We have a
>> big bare-metal Windows MSSQL server and a number of bare metal hosts, each
>> with the virtualization software and a single CentOs virtual server inside.
>> These CentOs hosts currently form an Ignite cluster with 4 data nodes and 1
>> client node. The example [2] I published last year is intended to solve the
>> business problem we have out here:
>> 1) the data currently present in the cluster have zero value;
>> 2) actual data is in the database and must be loaded in the cluster ASAP.
>> We use BinaryObject as cache key and value;
>> 3) cluster performs some data processing and writes the result to the
>> database.
>>
>> Unfortunately, the code [2] does not 100% OK in my case, it tends to say
>> "is node still alive" and to drop the client node off the cluster. The
>> performance of the MSSQL and network is what it is, I consider it as a
>> given restriction. It seems I got some progress when managed to move the
>> data loading process from a single client node to multiple data nodes. When
>> the extra data nodes will be added, I expect the load performance will be
>> better. Of course, until my big MSSQL will be able to hold the load. So I
>> want to know how interesting my results will be if it will be published.
>>
>> WDYT?
>>
>> [1] https://gist.github.com/ptupitsyn/4f54230636178865fc93c97e4d419f15
>> [2] https://github.com/vtchernyi/FastDataLoad
>>
>> чт, 25 февр. 2021 г. в 11:01, Pavel Tupitsyn :
>>
>>> Vladimir,
>>>
>>> Thanks for getting back to us. A full example that clarifies the
>>> situation will be great!
>>>
>>> > Can you share your code as a GitHub project? Maybe with the script to
>>> reproduce 6 GB of data.
>>>
>>> It is super trivial, I just wanted to get a sense of the throughput and
>>> check if we have some kind of a regression in recent versions (we don't) [1]
>>> Also I realised that the data size can be counted very differently - do
>>> we account for DB overhead and how?
>>>
>>> [1] https://gist.github.com/ptupitsyn/4f54230636178865fc93c97e4d419f15
>>>
>>> On Thu, Feb 25, 2021 at 10:49 AM Vladimir Tchernyi 
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I'd spent some time thinking about the community comments on my post.
>>>> It seems that Ignite is really not a bottleneck here. The performance of my
>>>> production MSSQL is a given restriction and the problem is to 

Re: very fast loading of very big table

[Vladimir Tchernyi]

Hi Pavel,

the code [1] you shared is a kind of in-memory experiment with all the
processes inside a single JVM. My work differs - it is from the big retail
business, and hence it is 100% practice-oriented. True to say, it's
oriented to the state of things inside my company, and that is my question
- will my results be interesting to the community? I have seen a lot of
questions on the user list regarding data loading and difficulties here
seem to be a blocker in extending Ignite's popularity.

Please let me know if my case is not common in the industry. We have a big
bare-metal Windows MSSQL server and a number of bare metal hosts, each with
the virtualization software and a single CentOs virtual server inside.
These CentOs hosts currently form an Ignite cluster with 4 data nodes and 1
client node. The example [2] I published last year is intended to solve the
business problem we have out here:
1) the data currently present in the cluster have zero value;
2) actual data is in the database and must be loaded in the cluster ASAP.
We use BinaryObject as cache key and value;
3) cluster performs some data processing and writes the result to the
database.

Unfortunately, the code [2] does not 100% OK in my case, it tends to say
"is node still alive" and to drop the client node off the cluster. The
performance of the MSSQL and network is what it is, I consider it as a
given restriction. It seems I got some progress when managed to move the
data loading process from a single client node to multiple data nodes. When
the extra data nodes will be added, I expect the load performance will be
better. Of course, until my big MSSQL will be able to hold the load. So I
want to know how interesting my results will be if it will be published.

WDYT?

[1] https://gist.github.com/ptupitsyn/4f54230636178865fc93c97e4d419f15
[2] https://github.com/vtchernyi/FastDataLoad

чт, 25 февр. 2021 г. в 11:01, Pavel Tupitsyn :

> Vladimir,
>
> Thanks for getting back to us. A full example that clarifies the situation
> will be great!
>
> > Can you share your code as a GitHub project? Maybe with the script to
> reproduce 6 GB of data.
>
> It is super trivial, I just wanted to get a sense of the throughput and
> check if we have some kind of a regression in recent versions (we don't) [1]
> Also I realised that the data size can be counted very differently - do we
> account for DB overhead and how?
>
> [1] https://gist.github.com/ptupitsyn/4f54230636178865fc93c97e4d419f15
>
> On Thu, Feb 25, 2021 at 10:49 AM Vladimir Tchernyi 
> wrote:
>
>> Hi,
>>
>> I'd spent some time thinking about the community comments on my post. It
>> seems that Ignite is really not a bottleneck here. The performance of my
>> production MSSQL is a given restriction and the problem is to ensure fast
>> loading by executing multiple parallel queries. I'll test my code in
>> production for a couple of months for possible problems. If it will be OK,
>> probably the complete/downloadable/compilable GitHub example will be useful
>> for the community.
>>
>> WDYT?
>>
>> пт, 19 февр. 2021 г. в 21:47, Vladimir Tchernyi :
>>
>>> Pavel,
>>>
>>> maybe it's time to put your five-cent in. Can you share your code as a
>>> GitHub project? Maybe with the script to reproduce 6 GB of data.
>>>
>>> As for MSSQL data retrieval being the bottleneck - don't think so, I got
>>> 15 min load time for 1 node and 3.5 min time for 4 nodes. Looks like a
>>> linear dependency (the table and the RDBMS server were the same).
>>> --
>>> Vladimir
>>>
>>> пт, 19 февр. 2021 г. в 19:47, Pavel Tupitsyn :
>>>
>>>> > First of all, I tried to select the whole table as once
>>>>
>>>> Hmm, it looks like MSSQL data retrieval may be the bottleneck here, not
>>>> Ignite.
>>>>
>>>> Can you run a test where some dummy data of the same size as real data
>>>> is generated and inserted into Ignite,
>>>> so that we test Ignite perf only, excluding MSSQL from the equation?
>>>> For example, streaming 300 million entries (total size 6 GB) takes
>>>> around 1 minute on my machine, with a simple single-threaded DataStreamer.
>>>>
>>>> On Fri, Feb 19, 2021 at 4:49 PM Vladimir Tchernyi 
>>>> wrote:
>>>>
>>>>> Hi folks,
>>>>> thanks for your interest in my work.
>>>>>
>>>>> I didn't try COPY FROM since I've tried to work with Ignite SQL a
>>>>> couple of years ago and didn't succeed. Probably because examples 
>>>>> available
>>>>&g

Re: very fast loading of very big table

[Vladimir Tchernyi]

Hi,

I'd spent some time thinking about the community comments on my post. It
seems that Ignite is really not a bottleneck here. The performance of my
production MSSQL is a given restriction and the problem is to ensure fast
loading by executing multiple parallel queries. I'll test my code in
production for a couple of months for possible problems. If it will be OK,
probably the complete/downloadable/compilable GitHub example will be useful
for the community.

WDYT?

пт, 19 февр. 2021 г. в 21:47, Vladimir Tchernyi :

> Pavel,
>
> maybe it's time to put your five-cent in. Can you share your code as a
> GitHub project? Maybe with the script to reproduce 6 GB of data.
>
> As for MSSQL data retrieval being the bottleneck - don't think so, I got
> 15 min load time for 1 node and 3.5 min time for 4 nodes. Looks like a
> linear dependency (the table and the RDBMS server were the same).
> --
> Vladimir
>
> пт, 19 февр. 2021 г. в 19:47, Pavel Tupitsyn :
>
>> > First of all, I tried to select the whole table as once
>>
>> Hmm, it looks like MSSQL data retrieval may be the bottleneck here, not
>> Ignite.
>>
>> Can you run a test where some dummy data of the same size as real data is
>> generated and inserted into Ignite,
>> so that we test Ignite perf only, excluding MSSQL from the equation?
>> For example, streaming 300 million entries (total size 6 GB) takes around
>> 1 minute on my machine, with a simple single-threaded DataStreamer.
>>
>> On Fri, Feb 19, 2021 at 4:49 PM Vladimir Tchernyi 
>> wrote:
>>
>>> Hi folks,
>>> thanks for your interest in my work.
>>>
>>> I didn't try COPY FROM since I've tried to work with Ignite SQL a couple
>>> of years ago and didn't succeed. Probably because examples available aren't
>>> complete/downloadable/compilable (the paper [1] contains GitHub repo, that
>>> is my five cents in changing the status quo). My interest is in KV API.
>>>
>>> I did try a data streamer, and that was my first try. I did not notice a
>>> significant time reduction in using code from my paper [1] versus data
>>> streamer/receiver. There was some memory economy with the streamer, though.
>>> I must say my experiment was made on a heavily loaded production mssql
>>> server. Filtered query with 300K rows resultset takes about 15 sec. The
>>> story follows.
>>>
>>> First of all, I tried to select the whole table as once, I got the
>>> network timeout and the client node was dropped off the cluster (is node
>>> still alive?).
>>> So I'd partitioned the table and executed a number of queries one-by-one
>>> on the client node, each query for the specific table partition. That
>>> process took about 90 min. Inacceptable time.
>>>
>>> Then I tried to execute my queries in parallel on the client node, each
>>> query executing dataStreamer.addData() for a single dataStreamer. The
>>> timing was not less than 15 min. All the attempts were the same, probably
>>> that was the network throughput limit on the client node (same interface
>>> used for the resultset and for cluster intercom). Say it again - that was
>>> the production environment.
>>>
>>> Final schema:
>>> * ComputeTask.map() schedules ComputeJobs amongst cluster nodes, one job
>>> for one table partition;
>>> * each job executes SQL query, constructs a map with binary object key
>>> and value. Then the job executes targetCache.invokeAll() specifying the
>>> constructed map and the static EntryProcessor class. The EntryProcessor
>>> contains the logic for cache binary entry update;
>>> * ComputeTask.reduce() summarizes the row count reported by each job.
>>>
>>> The schema described proved to be network error-free in my production
>>> network and gives acceptable timing.
>>>
>>> Vladimir
>>>
>>> [1]
>>> https://www.gridgain.com/resources/blog/how-fast-load-large-datasets-apache-ignite-using-key-value-api
>>>
>>> пт, 19 февр. 2021 г. в 16:41, Stephen Darlington <
>>> stephen.darling...@gridgain.com>:
>>>
>>>> I think it’s more that that putAll is mostly atomic, so the more
>>>> records you save in one chunk, the more locking, etc. happens. Distributing
>>>> as compute jobs means all the putAlls will be local which is beneficial,
>>>> and the size of each put is going to be smaller (also beneficial).
>>>>
>>>> But that’s a lot of work that the data streamer already does for yo

Re: very fast loading of very big table

[Vladimir Tchernyi]

Pavel,

maybe it's time to put your five-cent in. Can you share your code as a
GitHub project? Maybe with the script to reproduce 6 GB of data.

As for MSSQL data retrieval being the bottleneck - don't think so, I got 15
min load time for 1 node and 3.5 min time for 4 nodes. Looks like a linear
dependency (the table and the RDBMS server were the same).
--
Vladimir

пт, 19 февр. 2021 г. в 19:47, Pavel Tupitsyn :

> > First of all, I tried to select the whole table as once
>
> Hmm, it looks like MSSQL data retrieval may be the bottleneck here, not
> Ignite.
>
> Can you run a test where some dummy data of the same size as real data is
> generated and inserted into Ignite,
> so that we test Ignite perf only, excluding MSSQL from the equation?
> For example, streaming 300 million entries (total size 6 GB) takes around
> 1 minute on my machine, with a simple single-threaded DataStreamer.
>
> On Fri, Feb 19, 2021 at 4:49 PM Vladimir Tchernyi 
> wrote:
>
>> Hi folks,
>> thanks for your interest in my work.
>>
>> I didn't try COPY FROM since I've tried to work with Ignite SQL a couple
>> of years ago and didn't succeed. Probably because examples available aren't
>> complete/downloadable/compilable (the paper [1] contains GitHub repo, that
>> is my five cents in changing the status quo). My interest is in KV API.
>>
>> I did try a data streamer, and that was my first try. I did not notice a
>> significant time reduction in using code from my paper [1] versus data
>> streamer/receiver. There was some memory economy with the streamer, though.
>> I must say my experiment was made on a heavily loaded production mssql
>> server. Filtered query with 300K rows resultset takes about 15 sec. The
>> story follows.
>>
>> First of all, I tried to select the whole table as once, I got the
>> network timeout and the client node was dropped off the cluster (is node
>> still alive?).
>> So I'd partitioned the table and executed a number of queries one-by-one
>> on the client node, each query for the specific table partition. That
>> process took about 90 min. Inacceptable time.
>>
>> Then I tried to execute my queries in parallel on the client node, each
>> query executing dataStreamer.addData() for a single dataStreamer. The
>> timing was not less than 15 min. All the attempts were the same, probably
>> that was the network throughput limit on the client node (same interface
>> used for the resultset and for cluster intercom). Say it again - that was
>> the production environment.
>>
>> Final schema:
>> * ComputeTask.map() schedules ComputeJobs amongst cluster nodes, one job
>> for one table partition;
>> * each job executes SQL query, constructs a map with binary object key
>> and value. Then the job executes targetCache.invokeAll() specifying the
>> constructed map and the static EntryProcessor class. The EntryProcessor
>> contains the logic for cache binary entry update;
>> * ComputeTask.reduce() summarizes the row count reported by each job.
>>
>> The schema described proved to be network error-free in my production
>> network and gives acceptable timing.
>>
>> Vladimir
>>
>> [1]
>> https://www.gridgain.com/resources/blog/how-fast-load-large-datasets-apache-ignite-using-key-value-api
>>
>> пт, 19 февр. 2021 г. в 16:41, Stephen Darlington <
>> stephen.darling...@gridgain.com>:
>>
>>> I think it’s more that that putAll is mostly atomic, so the more records
>>> you save in one chunk, the more locking, etc. happens. Distributing as
>>> compute jobs means all the putAlls will be local which is beneficial, and
>>> the size of each put is going to be smaller (also beneficial).
>>>
>>> But that’s a lot of work that the data streamer already does for you and
>>> the data streamer also batches updates so would still be faster.
>>>
>>> On 19 Feb 2021, at 13:33, Maximiliano Gazquez 
>>> wrote:
>>>
>>> What would be the difference between doing cache.putAll(all rows) and
>>> separating them by affinity key+executing putAll inside a compute job.
>>> If I'm not mistaken, doing putAll should end up splitting those rows by
>>> affinity key in one of the servers, right?
>>> Is there a comparison of that?
>>>
>>> On Fri, Feb 19, 2021 at 9:51 AM Taras Ledkov 
>>> wrote:
>>>
>>>> Hi Vladimir,
>>>> Did you try to use SQL command 'COPY FROM ' via thin JDBC?
>>>> This command uses 'IgniteDataStreamer' to write data into cluster and
&g

Re: very fast loading of very big table

[Vladimir Tchernyi]

Hi folks,
thanks for your interest in my work.

I didn't try COPY FROM since I've tried to work with Ignite SQL a couple of
years ago and didn't succeed. Probably because examples available aren't
complete/downloadable/compilable (the paper [1] contains GitHub repo, that
is my five cents in changing the status quo). My interest is in KV API.

I did try a data streamer, and that was my first try. I did not notice a
significant time reduction in using code from my paper [1] versus data
streamer/receiver. There was some memory economy with the streamer, though.
I must say my experiment was made on a heavily loaded production mssql
server. Filtered query with 300K rows resultset takes about 15 sec. The
story follows.

First of all, I tried to select the whole table as once, I got the network
timeout and the client node was dropped off the cluster (is node still
alive?).
So I'd partitioned the table and executed a number of queries one-by-one on
the client node, each query for the specific table partition. That process
took about 90 min. Inacceptable time.

Then I tried to execute my queries in parallel on the client node, each
query executing dataStreamer.addData() for a single dataStreamer. The
timing was not less than 15 min. All the attempts were the same, probably
that was the network throughput limit on the client node (same interface
used for the resultset and for cluster intercom). Say it again - that was
the production environment.

Final schema:
* ComputeTask.map() schedules ComputeJobs amongst cluster nodes, one job
for one table partition;
* each job executes SQL query, constructs a map with binary object key and
value. Then the job executes targetCache.invokeAll() specifying the
constructed map and the static EntryProcessor class. The EntryProcessor
contains the logic for cache binary entry update;
* ComputeTask.reduce() summarizes the row count reported by each job.

The schema described proved to be network error-free in my production
network and gives acceptable timing.

Vladimir

[1]
https://www.gridgain.com/resources/blog/how-fast-load-large-datasets-apache-ignite-using-key-value-api

пт, 19 февр. 2021 г. в 16:41, Stephen Darlington <
stephen.darling...@gridgain.com>:

> I think it’s more that that putAll is mostly atomic, so the more records
> you save in one chunk, the more locking, etc. happens. Distributing as
> compute jobs means all the putAlls will be local which is beneficial, and
> the size of each put is going to be smaller (also beneficial).
>
> But that’s a lot of work that the data streamer already does for you and
> the data streamer also batches updates so would still be faster.
>
> On 19 Feb 2021, at 13:33, Maximiliano Gazquez 
> wrote:
>
> What would be the difference between doing cache.putAll(all rows) and
> separating them by affinity key+executing putAll inside a compute job.
> If I'm not mistaken, doing putAll should end up splitting those rows by
> affinity key in one of the servers, right?
> Is there a comparison of that?
>
> On Fri, Feb 19, 2021 at 9:51 AM Taras Ledkov  wrote:
>
>> Hi Vladimir,
>> Did you try to use SQL command 'COPY FROM ' via thin JDBC?
>> This command uses 'IgniteDataStreamer' to write data into cluster and
>> parse CSV on the server node.
>>
>> PS. AFAIK IgniteDataStreamer is one of the fastest ways to load data.
>>
>> Hi Denis,
>>
>> Data space is 3.7Gb according to MSSQL table properries
>>
>> Vladimir
>>
>> 9:47, 19 февраля 2021 г., Denis Magda 
>> :
>>
>> Hello Vladimir,
>>
>> Good to hear from you! How much is that in gigabytes?
>>
>> -
>> Denis
>>
>>
>> On Thu, Feb 18, 2021 at 10:06 PM  wrote:
>>
>> Sep 2020 I've published the paper about Loading Large Datasets into
>> Apache Ignite by Using a Key-Value API (English [1] and Russian [2]
>> version). The approach described works in production, but shows
>> inacceptable perfomance for very large tables.
>>
>> The story continues, and yesterday I've finished the proof of concept for
>> very fast loading of very big table. The partitioned MSSQL table about 295
>> million rows was loaded by the 4-node Ignite cluster in 3 min 35 sec. Each
>> node had executed its own SQL queries in parallel and then distributed the
>> loaded values across the other cluster nodes.
>>
>> Probably that result will be of interest for the community.
>>
>> Regards,
>> Vladimir Chernyi
>>
>> [1]
>> https://www.gridgain.com/resources/blog/how-fast-load-large-datasets-apache-ignite-using-key-value-api
>> [2] https://m.habr.com/ru/post/526708/
>>
>>
>>
>> --
>> Отправлено из мобильного приложения Яндекс.Почты
>>
>> --
>> Taras Ledkov
>> Mail-To: tled...@gridgain.com
>>
>>
>
>

Re: IgniteDataStreamer.keepBinary proposal

[Vladimir Tchernyi]

Hi Denis,

I think the code examples we already have do not show the nature of Ignite
as a DISTRIBUTED database. These examples are oriented on a single-node
start. An inexperienced user can have a false impression that a single
Ignite node can outperform, for example, a commercial database server.

IMHO the documentation should be written for a multinode Ignite cluster. I
do not understand what is the purpose to show how to stream 100_000 integer
values in a cache defined as . In the real world, I need
to stream structured records (Kafka Avro messages), and I will create a
POJO to hold each message. It is known that Ignite does not
peer-deploy user POJOs, so using BinaryObject is the only way to forward my
POJOs to the remote nodes (correct me if I am wrong).

I trust Ignite and I managed to create really fast Ignite app in
production. But recently I faced again the long-forgotten feeling - the
page is nice but hard to use. Hope my experience will help to
improve documentation.

Vladimir

PS
as for contributing, I need some time to get my Kafka Ignite app to
production to be sure of it. After that, I will be ready to contribute

сб, 5 дек. 2020 г. в 06:31, Denis Magda :

> Hi Vladimir,
>
> Most of the code snippets are already arranged in complete and
> ready-for-usage samples:
>
> https://github.com/apache/ignite/tree/master/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets
>
> Anyway, those are code snippets that are injected into quite generic
> documentation pages. Your case represents a situation when someone needs to
> work with binary objects and streaming APIs. What if we add a data streamer
> example for BinaryObjects into Ignite's examples and put a reference to
> that example from the documentation page? Are you interested in
> contributing the example?
> https://github.com/apache/ignite/tree/master/examples
>
> -
> Denis
>
>
> On Fri, Dec 4, 2020 at 2:58 AM Vladimir Tchernyi 
> wrote:
>
>> Hi, community
>>
>>
>>
>> I've just finished drilling a small page [1] about Ignite data streaming
>> and I want to share my impressions. The situation is common for many Ignite
>> documentation pages, impressions are the same.
>>
>>
>>
>> My problem was to adapt IgniteDataStreamer to data loading using the
>> binary format as described in my article [2]. I try to use the same
>> approach:
>>
>> 1) load data on the client node;
>>
>> 2) convert it to the binary form;
>>
>> 3) use IgniteDataStreamer/StreamReceiver pair (instead of
>> ComputeTaskAdapter/ComputeJobAdapter) to ingest data in the cache.
>>
>>
>>
>> I modified my production code using IgniteDataStreamer> BinaryObject> and  StreamReceiver, tried to
>> start on the dev cluster made of 2 server nodes and 1 client node. That is
>> it: ClassNotFoundException for the class that exists on the client node
>> only.
>>
>>
>>
>> The solution to the problem seems to be in setting
>> streamer.keepBinary(true), but page [1] never says about it. I found that
>> setter in the IgniteDataStreamer source code after a single day of
>> troubleshooting. Definitely, "In Ignite We Trust" - what else reason would
>> drive me to spend so much time?
>>
>>
>>
>> The code snippets on the page [1] are hard to implement in real-world
>> applications because of using only primitive types String, Integer, etc.
>> These are more like unit tests.
>>
>>
>>
>> My proposal - it would be great to create a small GitHub repo containing
>> a complete compilable code example, one repo for every page. I think such
>> repos will keep the newbie Ignite users inside the project and prevent them
>> from leaving.
>>
>>
>>
>> Regards,
>>
>> Vladimir Tchernyi
>>
>> --
>>
>> [1] https://ignite.apache.org/docs/latest/data-streaming
>>
>> [2]
>> https://www.gridgain.com/resources/blog/how-fast-load-large-datasets-apache-ignite-using-key-value-api
>>
>>
>>
>

IgniteDataStreamer.keepBinary proposal

[Vladimir Tchernyi]

Hi, community



I've just finished drilling a small page [1] about Ignite data streaming
and I want to share my impressions. The situation is common for many Ignite
documentation pages, impressions are the same.



My problem was to adapt IgniteDataStreamer to data loading using the binary
format as described in my article [2]. I try to use the same approach:

1) load data on the client node;

2) convert it to the binary form;

3) use IgniteDataStreamer/StreamReceiver pair (instead of
ComputeTaskAdapter/ComputeJobAdapter) to ingest data in the cache.



I modified my production code using IgniteDataStreamer and  StreamReceiver, tried to
start on the dev cluster made of 2 server nodes and 1 client node. That is
it: ClassNotFoundException for the class that exists on the client node
only.



The solution to the problem seems to be in setting
streamer.keepBinary(true), but page [1] never says about it. I found that
setter in the IgniteDataStreamer source code after a single day of
troubleshooting. Definitely, "In Ignite We Trust" - what else reason would
drive me to spend so much time?



The code snippets on the page [1] are hard to implement in real-world
applications because of using only primitive types String, Integer, etc.
These are more like unit tests.



My proposal - it would be great to create a small GitHub repo containing a
complete compilable code example, one repo for every page. I think such
repos will keep the newbie Ignite users inside the project and prevent them
from leaving.



Regards,

Vladimir Tchernyi

--

[1] https://ignite.apache.org/docs/latest/data-streaming

[2]
https://www.gridgain.com/resources/blog/how-fast-load-large-datasets-apache-ignite-using-key-value-api

Re: Query on CVE-2020-5421

[Vladimir Pligin]

This issue is definitely web-oriented. It's about the 'Content-Disposition'
custom HTTP header which raised a big deal of concerns about security itself
over time. As far as I currently understand it affects only web part(s) of
Spring, more particularly spring-webmvc. It's not being used by Ignite. So I
suppose it's safe to think that Ignite is not affected by that one.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Inserting date into ignite with spark jdbc

[Vladimir Pligin]

Hi,

It seems that a dataset internally uses built-in
org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider instead
of org.apache.ignite.spark.impl.IgniteRelationProvider in case you force it
to use JDBC. The provider from spark obviously doesn't tolerate Ignite
custom properties. To be honest I'm sure how it should work, I'll need to
think about it.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: ReadFromBackup, Primary_SYNC, Backups

[Vladimir Pligin]

Manesh,

>> When running a large SELECT from a thick client node,  could data be
>> fetched from the backup instead of primary partitions?

readFromBackup=true affects only cache operations, it doesn't work with SQL
queries.

>> in a two server system, if readFromBackup = False for a cache, and if one
>> server fails, 
would 2nd server stop serving client requests since some partition data is
in the backup?

no, it will go on serving requests, former backup partitions will turn into
primary

>> is it possible that readFromBackup=True and Primary_Sync mode for a cache
>> could give inconsistent data for cacheMode Replicated in a 2 server node?

there's no need to explicitly use readFromBackup with replicated caches.
What do you exactly mean by "inconsistent data"? Could you please describe
your scenario?

>> if I increase backup= 10, in a 2 server system, would it mean that there
>> are 10 backups.
I am guessing, ignite would keep a single back up on each server, not 5 and
5 on each server. 
a new backup is created for that cache for any new node joining the cluster.
is this right understanding?

Yes, that's correct.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite JDBC connection pooling mechanism

[Vladimir Pligin]

In general it should be ok to use connection pooling with Ignite. Is your
network ok? It look like a connection is being closed because of network
issues.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite Client Node OOM Issue

[Vladimir Pligin]

What load profile do you have? Do you use SQL queries? Is it possible to
share your client node configuration?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Limit ignite-rest-http threads

[Vladimir Pligin]

Hi Ashish,

Yes, it works that way because Ignite launches Jetty server under the hood
to handle HTTP requests.
Could you please clarify what concerns you exactly? In general a number of
threads doesn't indicate anything "bad" or "good". 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite JDBC connection pooling mechanism

[Vladimir Pligin]

I wasn't able to reproduce that hurriedly. What Hikari settings do you have?
Maybe you have a reproducer?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite JDBC connection pooling mechanism

[Vladimir Pligin]

Hi,

What java version do you use? What about Hikari & Ignite versions? Do you
have SSL enabled?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Occasional duplicate ID with IgniteAtomicSequence in a cluster

[Vladimir Pligin]

Do you have a reproducer of such behavior? It would be really helpful.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : MutableEntry.getValue().toBuilder throws NullPointerException

[Vladimir Pligin]

Do you get it on every cache or with some subset of caches?
Do you have assertions enabled? 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Purging a partition

[Vladimir Pligin]

To be honest it doesn't look trivial.
I'm not sure at the moment why one could need that.
Could you please describe your use-case?
It will definitely help us to design a solution somehow.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Binary memory restoration

[Vladimir Pligin]

It's possible that it happens because of
https://issues.apache.org/jira/browse/IGNITE-13068.
We need to scan the entire SQL primary index during startup in case you have
at least on query entity configured.
As far as I can see it's going to be a part of the Ignite 2.9 release.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Map 10 caches to 10 different ignite nodes

[Vladimir Pligin]

Yes, it's possible to achieve that with a node filter.
You should change your cache configurations.
Here's the  javadoc

 
. After that you'll be able to leverage attributes to filter nodes. A cache
will start on on certain nodes for which the predicate return true.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Preload all indexes to memory

[Vladimir Pligin]

As far as know there's no such functionality at the moment.
The only way that comes to my mind is to try making a SQL query that scans
it.
For example if you created a index like that:

CREATE INDEX indexName ON tableName (field1, field2);
Then you can try to execute:

SELECT field1, field2 FROM tableName USE INDEX(indexName);

By the way I suppose it's going to be possible soon, have a look at
https://cwiki.apache.org/confluence/display/IGNITE/IEP-40+Cache+warm-up



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite Queue (Documentation or Code defect)?

[Vladimir Pligin]

Hi guys,

That has been done.
https://issues.apache.org/jira/browse/IGNITE-13396



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: can web session filter be configured to create session on demand

[Vladimir Pligin]

Hi Cameron,

I suppose the correct way of achieving the desired behavior is to modify
WebSessionFilter from the Ignite.
For example it's possible to introduce a new context param with a list of
urls that shouldn't be affected by the filter.
And pass a request to a chain in the doFilter method if the request matches.
I suppose you can look here
(https://github.com/apache/ignite/blob/master/modules/web/src/main/java/org/apache/ignite/cache/websession/WebSessionFilter.java#L369).
 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: read-though tutorial for a big table

[Vladimir Tchernyi]

Hi Denis,

Some progress had happened and I have some material to share with the
community. I think it will be interesting to newbies. It is about loading
big tables from rdbms and creating cache entries based on table info. This
approach was tested in production and showed good timing being paired with
MSSQL, tables from tens to hundreds million rows.

The loading jar process:
* starts Ignite client node;
* creates user POJO according to business logic;
* converts POJOs to BinaryObjects;
* uses affinity function and creates separate key-value HashMap for every
cache partition;
* uses ComputeTaskAdaper/ComputeJobAdaper to place hashMaps on
corresponding data node.

I would like to publish some tutorial, say on GridGain website in english
and russian version on habr.com.

WDYT?

чт, 12 мар. 2020 г. в 08:25, :

> Hello Denis,
>
> That is possible, my writing activities should be continued. The only
> question is to get my local project to production, there is no sense in
> writing another model example. So I hope there will be a progress in the
> nearest future
>
> Vladimir
>
> 2:25, 12 марта 2020 г., Denis Magda :
>
> Hello Vladimir,
>
> Just to clarify, are you suggesting to create a tutorial for data loading
> scenarios when data resides in an external database?
>
> -
> Denis
>
>
> On Tue, Mar 10, 2020 at 11:41 PM  wrote:
>
> Andrei, Evgenii, thanks for answer.
>
> Aa far as I see, there is no ready to use tutorial. I managed to do
> multi-threaded cache load procedure, out-of-the-box loadCache method is
> extremely slow.
>
> I spent about a month studying write-through topics, and finally got the
> same as "capacity planning" says: 0.8Gb mssql table on disk expands to
> 2.3Gb, size in ram is 2.875 times bigger.
>
> Is it beneficial to use BinaryObject instead of user pojo? If yes, how to
> create BinaryObject without pojo definition and deserialize it back to pojo?
> It would be great to have kind of advanced github example like this
>
> https://github.com/dmagda/MicroServicesExample
>
> It helped a lot in understanding. Current documentation links do not help
> to build a real solution, they are mostly like a reference, with no option
> to compile and debug
>
> Vladimir
>
> 2:51, 11 марта 2020 г., Evgenii Zhuravlev :
>
> When you're saying that the result was poor, do you mean that data
> preloading took too much time, or it's just about get operations?
>
> Evgenii
>
> вт, 10 мар. 2020 г. в 03:29, aealexsandrov :
>
> Hi,
>
> You can read the documentation articles:
>
> https://apacheignite.readme.io/docs/3rd-party-store
>
> In case if you are going to load the cache from 3-rd party store (RDBMS)
> then the default implementation of CacheJdbcPojoStore can take a lot of
> time
> for loading the data because it used JDBC connection inside (not pull of
> these connections).
>
> Probably you should implement your own version of CacheStore that will read
> data from RDBMS in several threads, e.g using the JDBC connection pull
> there. Sources are open for you, so you can copy the existed implementation
> and modify it:
>
>
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/cache/store/jdbc/CacheJdbcPojoStore.java
>
> Otherwise, you can do the initial data loading using some streaming tools:
>
> 1)Spark integration with Ignite -
> https://apacheignite-fs.readme.io/docs/ignite-data-frame
> 2)Kafka integration with Ignite -
> https://apacheignite-mix.readme.io/docs/kafka-streamer
>
> BR,
> Andrei
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
>
>
> --
> Отправлено из мобильного приложения Яндекс.Почты
>
>
>
> --
> Отправлено из мобильного приложения Яндекс.Почты

Re: Ignite 2.7.6: Memory Leak with Direct buffers in TCPCommunication SPI

[Vladimir Pligin]

By the way it would be great if you share a couple of heap histograms between
restarts. Something like jmap -histo  > histo. Is it possible?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite 2.7.6: Memory Leak with Direct buffers in TCPCommunication SPI

[Vladimir Pligin]

Hi Manesh,

Thanks for sharing your case. We'll try to reproduce it.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Long running query

[Vladimir Pligin]

Hi,

The most useful information here is the plan that the RDBMS uses.
Is it possible to share it?

I suppose that using of either the IDX_ORG_MEDP_DRUG index or the
IDX_DRUG_ID index is not absolutely correct.
My bet is an index on (DRUG_ID, ORG_ID) could help here but it's to be
checked.

And it's important to clarify whether you really need a left join here.
Inner join would help to use separate "where" clause to have different
indexes for the join and for the filtering.
What do you think, does it make any sense here? 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: How to failover/scale cluster in Apache Ignite

[Vladimir Pligin]

Hi, I'll try to do my best to help you.

>> Is kill -9  the right way to kill a node?

No, I don't think this is the right way. 

>> How about re-adding new nodes that were previously killed?

You should clean a node's work directory before re-adding.

>> How long does it take for the nodes to synchronise?

It depends on your network, data volume, disk(s) speed, data storage
configuration etc.

>> How do we know when a rebalance is completed?

You'll see a message in a log. Or you can use WebConsole.


By the way it would be great if you provide some sort of a reproducer to
help us review your scenario.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Failed to clean IP finder up.

[Vladimir Pligin]

Hi,

Yeah, it seems to be that bug.
As far as can see it's resolved and marked to be released as a part of 2.8.
I haven't managed to find that commit in other branches unfortunately. 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite 2.7.0 : thick client has all system threads busy indefinitely

[Vladimir Pligin]

Hi, could you please share your configurations of both server and client?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Log4j2 problem

[Vladimir Pligin]

Hi,

Could you please share your logging configurations? I'll make a try to
reproduce.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: CacheEntryProcessor causes cluster node to stop

[Vladimir Pligin]

Hi,

It seems that the root cause of the issue happens on node with
id=fb70df1c-0311-48b4-bb33-88cdf85438ce.
That node becomes unresponsive. Could you please share logs from that node,
it could help?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Web Agent connected to Cluster but console doesn't show

[Vladimir Pligin]

Yes, I suppose it could be. Have you finally managed to put it all together?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Web Agent connected to Cluster but console doesn't show

[Vladimir Pligin]

And do you use the corresponding versions of WC and agent (2.0) or you're
trying a newer versions?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Web Agent connected to Cluster but console doesn't show

[Vladimir Pligin]

What version do you use?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Distributed Cluster Deployment

[Vladimir Pligin]

Hi,

Spring here is just a convenient way of configuration building.
Ignite is not tightly bound to it. You're able to construct everything
programmatically.
For example here https://apacheignite.readme.io/docs/tcpip-discovery you can
switch example from "XML" to "Java".
And Spring xml configuration 
 

  ...
  

  

  

  

  


turns into 

TcpDiscoverySpi spi = new TcpDiscoverySpi();
 
TcpDiscoveryMulticastIpFinder ipFinder = new
TcpDiscoveryMulticastIpFinder();
 
ipFinder.setMulticastGroup("228.10.10.157");
 
spi.setIpFinder(ipFinder);
 
IgniteConfiguration cfg = new IgniteConfiguration();
 
cfg.setDiscoverySpi(spi);
 
Ignition.start(cfg);

Does it make sense? 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite statistics in the logs

[Vladimir Pligin]

Hi,

As far as I know this information should be shown in Ignite logs with
default setup.
The only thing you need to do is to run ignite.sh with -v option or set
system property IGNITE_QUIET to false if you run a node without ignite.sh.
In my setup it looks like

[18:47:30,425][INFO][grid-timeout-worker-#39][IgniteKernal] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=029b4aeb, uptime=30:41:07.843]
^-- H/N/C [hosts=1, nodes=2, CPUs=16]
^-- CPU [cur=0%, avg=0.01%, GC=0%]
^-- PageMemory [pages=200]
^-- Heap [used=77MB, free=92.35%, comm=1010MB]
^-- Non heap [used=69MB, free=90.67%, comm=71MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=6, qSize=0]

Does it make sense for you?
-
Vladimir




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Error installing Ignite on K8s

[Vladimir Pligin]

Hi,

As far as can see you're facing an issue related to k8s service
configuration.
It's quite difficult to say whether you have issues with Ignite pods at this
stage.
So let's try to figure it out step by step. I suppose that the error is
related to load balancing.
As far as I know AWS provides multiple load balancer types and it seems like
the particular one doesn't support session affinity based on client IP.
Could you please check whether this one works for you
https://pastebin.com/uEfL0rDd?
I've changed a type of load balancer to network load balancer. 

Depending on the result let's discuss further steps. 

-
Vladimir 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite with Hibernate 5.3

[Vladimir Pligin]

Hi Tomasz,

It seems like Hibernate 5.3 support has been already added to Ignite in the
master branch.
I suppose this feature will be delivered as a part of upcoming Ignite
release. Here is the corresponding ticket
https://issues.apache.org/jira/browse/IGNITE-9893, fix version is 2.8.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Clarity on Ignite filesytem

[Vladimir Ozerov]

Hi Maros,

Over the years of IGFS existence we came to conclusion that it doesn't fit
to typical Ignite use cases well. Currently development of this component
is frozen. Most likely it will be removed in nearest future.
Hence, no plans to improve it or extend with new APIs.

On Mon, Apr 1, 2019 at 11:14 AM maros.urbanec 
wrote:

> Hi all, I've been recently looking into Ignite Filesystem (IGFS for short)
> and couldn't find much assistance by Google searches.
>
> Reading the documentation, it wasn't immediately obvious whether IGFS as a
> cache for HDFS is an option or a must. In other words - is it possible to
> use IGFS without Hadoop or any other secondary file stores; storing files
> in
> Ignite cache only?
>
> Secondarily, I've tried to start Ignite with
> examples\config\filesystem\example-igfs.xml configuration, only for Java
> client to obstinately claim "IGFS is not configured: igfs". The
> configuration is being read by Ignite, since enabling shared memory
> endpoint
> end up in error (I'm running on Windows). Is there any step I'm missing?
>
> As a last question - there is no C++ API for IGFS at this point. However, a
> recurring talk on these mailing lists is an integration of IGFS as a
> userspace filesystem. Is this item stipulated on a roadmap?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: When to use REPLICATED vs PARTITIONED?

[Vladimir Pligin]

Hi again!

I'm sorry for being absent for so long. 
Maybe the reason here that you data is collocated somehow using id of a
city.
Replicated cache suits well if you need to join to some static dictionary
and it's impossible or difficult to distribute other data in manner
corresponding to that dictionary. My assumption here is that Ignite joins
only subsets (about 1/2) of city cache in partitioned case. But when you
have replicated one the Ignite should join entire dataset. What do you think
does it make sense here?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: When to use REPLICATED vs PARTITIONED?

[Vladimir Pligin]

Hi David,

As far as I know your understanding is correct in general.
Let's try to figure out what's happening in this particular case.
Could you please share execution plans of the query with both partitioned
and replicated templates.
And it also would be great if you share execution time for both.


Vladimir



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: SQL Query plan confusion

[Vladimir Ozerov]

Hi Jose,

I am a bit lost in two provided explains. Both "godd" and "bad" contain "k
= 'trans.cust.first_name'" condition. Could you please confirm that they
are correct? Specifically, I cannot understand why this condition is
present in "good" explain.

On Tue, Nov 27, 2018 at 12:06 PM joseheitor  wrote:

> 1. - I have a nested join query on a table of 8,000,000 records which
> performs similar or better than PostreSQL (~10ms) on my small test setup
> (2x
> nodes, 8GB, 2CPU):
>
> SELECT
> mainTable.pk, mainTable.id, mainTable.k, mainTable.v
> FROM
> public.test_data AS mainTable
> INNER JOIN (
> SELECT
> lastName.id
> FROM
> (SELECT id FROM public.test_data WHERE k =
> 'trans.cust.last_name' AND v
> = 'Smythe-Veall') AS lastName
> INNER JOIN
> (SELECT id FROM
> public.test_data WHERE k = 'trans.date' AND v =
> '2017-12-21') AS transDate ON transDate.id = lastName.id
> INNER JOIN
> (SELECT id FROM
> public.test_data WHERE k = 'trans.amount' AND cast(v
> AS integer) > 9) AS transAmount ON transAmount.id = lastName.id
> ) AS subTable ON mainTable.id = subTable.id
> ORDER BY 1, 2
>
>
> 2. - By simply adding a WHERE clause at the end, the performance becomes
> catastrophic on Ignite (~10s for subsequent queries - first query takes
> many
> minutes). On PostgreSQL performance does not change...
>
> SELECT
> mainTable.pk, mainTable.id, mainTable.k, mainTable.v
> FROM
> public.test_data AS mainTable
> INNER JOIN (
> SELECT
> lastName.id
> FROM
> (SELECT id FROM public.test_data WHERE k =
> 'trans.cust.last_name' AND v
> = 'Smythe-Veall') AS lastName
> INNER JOIN
> (SELECT id FROM
> public.test_data WHERE k = 'trans.date' AND v =
> '2017-12-21') AS transDate ON transDate.id = lastName.id
> INNER JOIN
> (SELECT id FROM
> public.test_data WHERE k = 'trans.amount' AND cast(v
> AS integer) > 9) AS transAmount ON transAmount.id = lastName.id
> ) AS subTable ON mainTable.id = subTable.id
> *WHERE
> mainTable.k = 'trans.cust.first_name'*
> ORDER BY 1, 2
>
> What can I do to optimise this query for Ignite???
>
> (Table structure and query plans attached for reference)
>
> Thanks,
> Jose
> table.sql
> 
> good-join-query.txt
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t1652/good-join-query.txt>
>
> bad-join-query.txt
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t1652/bad-join-query.txt>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: How much heap to allocate

[Vladimir Ozerov]

Hi Eugene,

Answering your questions:
1) Grouping is performed on both mapper and reducer (coordinator). If you
group be affinity key, you may try setting "SqlFieldsQuery.colocated=true"
to bypass grouping on reducer
2) For this specific query H2 will store (customer_id, count(*),
sum(views)) for every customer_id. It is hard to guess how much space it
would take in heap, but I think it would be ~50-100 bytes per customer_id.
So if you have N customers, it would be (100 * N) bytes
3) Please see
https://apacheignite-sql.readme.io/docs/performance-and-debugging

Vladimir.

On Thu, Aug 30, 2018 at 5:57 PM eugene miretsky 
wrote:

> Thanks against for the detailed response!
>
> Our main use case is preforming large SQL queries over tables with 200M+
> rows  - wanted to give you a bit more details and context you can pass along
>
> A simple example would be:
>
>- Table: customer_id, date, category, views, clicks ( pkey =
>"customer_id, date", affinity key = date )
>- Query: SELECT count(*) where date < X AND categroy in (C1, C2, C3)
>GROUP BY customer_id HAVING SUM(views) > 20
>
> My main concernse are
> 1) How is the group by performed. You mentioend that it is performend on
> the coordinator, I was coping that singe we are grouping using an colomn
> that is an affintiy key, each node will be able to do it's own group by
> 2) How much heap should I allocate for the group by stage
> 3) General performance tips
>
> Cheers,
> Eugene
>
>
> On Thu, Aug 30, 2018 at 1:32 AM Denis Magda  wrote:
>
>> Eugene,
>>
>> Just want to be sure you know about the existence of the following pages
>> which elaborate on Ignite memory architecture in details:
>>
>>-
>>
>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-Entriesandpagesindurablememory
>>-
>>
>> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood
>>
>>
>>
>>> 1) Are indexs loaded into heap (when used)?
>>>
>>
>> Something might be copied to disk but in most of the cases we perform
>> comparisons and other operations directly off-heap.
>> See org.apache.ignite.internal.processors.query.h2.database.InlineIndexHelper
>> and related classes.
>>
>> 2) Are full pages loaded into heap, or only the matching records?
>>>
>>
>> Matching records (result set) are presently loaded. The pages are not.
>>
>>
>>> 3) When the query needs more processing than the exisiting index
>>> (non-indexed columns, groupBy, aggreag) where/how does it happen?
>>>
>>
>> We will be doing a full scan. Grouping and aggregations are finalized on
>> the query coordinator which needs to get a full result set.
>>
>> 4) How is the query coordinator chosen? Is it the client node? How about
>>> when using the web console?
>>>
>>
>> That's your application. Web Console uses Ignite SQL APIs as well.
>>
>>
>>> 5) What paralalism settings would your recomend, we were thinking to set
>>> parallelJobsNumber  to 1  and task parallelism to number of cores * 2 -
>>> this way we can make sure that each job gets al the heap memory instead of
>>> all jobs fighting each other. Not sure if it makes sense, and it will also
>>> prevent us from making real time transactional transactional queries.(we
>>> are hoping to use ignite for both olap and simple real time queries)
>>
>>
>> I would start a separate discussion for this bringing this question to
>> the attention of our SQL experts. I'm not the one of them.
>>
>> --
>> Denis
>>
>> On Mon, Aug 27, 2018 at 8:54 PM eugene miretsky <
>> eugene.miret...@gmail.com> wrote:
>>
>>> Denis, thanks for the detailed response.
>>>
>>> A few more follow up questions
>>> 1) Are indexs loaded into heap (when used)?
>>> 2) Are full pages loaded into heap, or only the matching records?
>>> 3) When the query needs more processing than the exisiting index
>>> (non-indexed columns, groupBy, aggreag) where/how does it happen?
>>> 4) How is the query coordinator chosen? Is it the client node? How about
>>> when using the web console?
>>> 5) What paralalism settings would your recomend, we were thinking to set
>>> parallelJobsNumber  to 1  and task parallelism to number of cores * 2 -
>>> this way we can make sure that each job gets al the heap memory instead of
>>> all jobs fighting each other. Not sure if it makes sense, and 

Re: Difference between replicated and local cache mode regarding time execution of query

[Vladimir Ozerov]

Hi Moti,

Could you please attach execution plans for both LOCAL and REPLICATED cases?

On Wed, Sep 5, 2018 at 6:13 PM ilya.kasnacheev 
wrote:

> Hello!
>
> Unfortunately I'm not aware why you see such a big difference in this case.
> It should be comparable. Maybe SQL people will chime in?
>
> Regards,
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite SQL Queries not getting all data back in ignite 2.4 and 2.6

[Vladimir Ozerov]

Dmitriy,

I wanted to suggest this, but looks like we do not have direct links for
regular distribution. User can download it from TeamCity, but
username/password is needed for this. May be I missing something.

On Thu, Aug 16, 2018 at 11:44 AM Dmitriy Setrakyan 
wrote:

> I also want to point out that Ignite has nightly builds, so you can try
> them instead of doing your own build as well.
>
> https://ignite.apache.org/download.cgi#nightly-builds
>
> D.
>
> On Thu, Aug 16, 2018 at 1:38 AM, Vladimir Ozerov 
> wrote:
>
>> Hi,
>>
>> There were a lot of changes in the product since 2.3 which may affect it.
>> Most important change was baseline topology, as already mentioned.
>> I am aware of a case when incorrect result might be returned [1], which
>> is already fixed in *master*. Not sure if this is the same issue, but
>> you may try to build Ignite from recent master and check if the problem is
>> still there.
>>
>> Is it possible to create isolated reproducer for this issue?
>>
>> [1] https://issues.apache.org/jira/browse/IGNITE-8900
>>
>> On Wed, Aug 15, 2018 at 11:34 PM bintisepaha 
>> wrote:
>>
>>> Thanks for getting back, but we do not use Ignite's native persistence.
>>> Anything else changed from 2.3 to 2.4 to cause this around SQL Queries?
>>>
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>>
>

Re: Ignite SQL Queries not getting all data back in ignite 2.4 and 2.6

[Vladimir Ozerov]

Hi,

There were a lot of changes in the product since 2.3 which may affect it.
Most important change was baseline topology, as already mentioned.
I am aware of a case when incorrect result might be returned [1], which is
already fixed in *master*. Not sure if this is the same issue, but you may
try to build Ignite from recent master and check if the problem is still
there.

Is it possible to create isolated reproducer for this issue?

[1] https://issues.apache.org/jira/browse/IGNITE-8900

On Wed, Aug 15, 2018 at 11:34 PM bintisepaha  wrote:

> Thanks for getting back, but we do not use Ignite's native persistence.
> Anything else changed from 2.3 to 2.4 to cause this around SQL Queries?
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite running on JDK10?

[Vladimir Ozerov]

HI,

Please try adding add these flags to JVM startup arguments.

On Fri, Aug 10, 2018 at 5:31 PM KJQ  wrote:

> As a note, I downgraded all of the Docker containers to use JDK 9 (9.0.4)
> and
> I still get the same problem running the SpringBoot 2 application.  Running
> in my IDE a test case works perfectly fine.
>
> *Caused by: java.lang.RuntimeException: jdk.internal.misc.JavaNioAccess
> class is unavailable.*
>
> *Caused by: java.lang.IllegalAccessException: class
> org.apache.ignite.internal.util.GridUnsafe cannot access class
> jdk.internal.misc.SharedSecrets (in module java.base) because module
> java.base does not export jdk.internal.misc to unnamed module @78a89eea*
>
>
>
> -
> KJQ
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Delete SQL is failing with IN clause for a table which has composite key

[Vladimir Ozerov]

Hi,

The problem is already fixed [1]. Fix will be available in AI 2.5.

[1] https://issues.apache.org/jira/browse/IGNITE-8147

On Wed, Apr 11, 2018 at 9:21 AM, Naveen  wrote:

> Hi
>
> These are the queries I have used, which I was getting the error every time
> I execute.
> Nothing else to share
>
> Thanks
> Naveen
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: SQL Transactional Support for Commit and rollback operation.

[Vladimir Ozerov]

Hi,

Transactional SQL is extremely huge multi-man-year effort. We hope to have
it in the product in Q3. We are working on it in normal (not "ASAP") mode,
as our ultimate goal is great product.

On Wed, Apr 11, 2018 at 2:32 PM, joseheitor  wrote:

> Yes - please clarify?
>
> We are also developing a product for which we are keen to use Ignite
> (instead of PostgreSQL), but it is a complete BLOCKER if there is no
> persistent SQL Transactional support.
>
> So we also require this urgently!
> (Please, please, please...)
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Upgrade from 2.1.0 to 2.4.0 resulting in error within transaction block

[Vladimir Ozerov]

I've just fixed possible root cause in master [1]. However, as exact use
case details is not known, may be it was something else.
Is it possible to provide more info on the use case: cache configuratioh,
model classes?

[1] https://issues.apache.org/jira/browse/IGNITE-8147

On Mon, Apr 2, 2018 at 12:21 PM, Yakov Zhdanov  wrote:

> Cross posting to dev.
>
> Vladimir Ozerov, can you please take a look at NPE from query processor
> (see below - GridQueryProcessor.typeByValue(GridQueryProcessor.java:1901)
> )?
>
> --Yakov
>
> 2018-03-29 0:19 GMT+03:00 smurphy :
>
>> Code works in Ignite 2.1.0. Upgrading to 2.4.0 produces the stack trace
>> below. The delete statement that is causing the error is:
>>
>> SqlFieldsQuery sqlQuery = new SqlFieldsQuery("delete from EngineFragment
>> where " + criteria());
>> fragmentCache.query(sqlQuery.setArgs(criteria.getArgs()));
>>
>> The code above is called from within a transactional block managed by a
>> PlatformTransactionManager which is in turn managed by Spring's
>> ChainedTransactionManager. If the @Transactional annotation is removed
>> from
>> the surrounding code, then the code works ok...
>>
>> 2018-03-28 15:50:05,748 WARN  [engine 127.0.0.1] progress_monitor_2
>> unknown
>> unknown {ProgressMonitorImpl.java:112} - Scan
>> [ec7af5e8-a773-40fd-9722-f81103de73dc] is unable to process!
>> javax.cache.CacheException: Failed to process key '247002'
>> at
>> org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>> Impl.query(IgniteCacheProxyImpl.java:618)
>> at
>> org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>> Impl.query(IgniteCacheProxyImpl.java:557)
>> at
>> org.apache.ignite.internal.processors.cache.GatewayProtected
>> CacheProxy.query(GatewayProtectedCacheProxy.java:382)
>> at
>> com.company.core.dao.ignite.IgniteFragmentDao.delete(IgniteF
>> ragmentDao.java:143)
>> at
>> com.company.core.dao.ignite.IgniteFragmentDao$$FastClassBySp
>> ringCGLIB$$c520aa1b.invoke()
>> at org.springframework.cglib.proxy.MethodProxy.invoke(MethodPro
>> xy.java:204)
>> at
>> org.springframework.aop.framework.CglibAopProxy$CglibMethodI
>> nvocation.invokeJoinpoint(CglibAopProxy.java:720)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation
>> .proceed(ReflectiveMethodInvocation.java:157)
>> at
>> org.springframework.dao.support.PersistenceExceptionTranslat
>> ionInterceptor.invoke(PersistenceExceptionTranslationInterce
>> ptor.java:136)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation
>> .proceed(ReflectiveMethodInvocation.java:179)
>> at
>> org.springframework.aop.framework.CglibAopProxy$DynamicAdvis
>> edInterceptor.intercept(CglibAopProxy.java:655)
>> at
>> com.company.core.dao.ignite.IgniteFragmentDao$$EnhancerBySpr
>> ingCGLIB$$ce60f71c.delete()
>> at
>> com.company.core.core.service.impl.InternalScanServiceImpl.p
>> urgeScanFromGrid(InternalScanServiceImpl.java:455)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>> ssorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>> thodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsin
>> gReflection(AopUtils.java:302)
>> at
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(
>> JdkDynamicAopProxy.java:202)
>> at com.sun.proxy.$Proxy417.purgeScanFromGrid(Unknown Source)
>> at com.company.core.core.async.tasks.PurgeTask.process(PurgeTas
>> k.java:85)
>> at sun.reflect.GeneratedMethodAccessor197.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>> thodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsin
>> gReflection(AopUtils.java:302)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation
>> .invokeJoinpoint(ReflectiveMethodInvocation.java:190)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation
>> .proceed(ReflectiveMethodInvocation.java:157)
>> at
>> org.sp

Re: COUNT DISTINCT could push down group expressions

[Vladimir Ozerov]

Hi,

I think that in some (or even all) cases it is possible to push down
DISTINCT keyword. However, it is not possible to GROUP BY in general case,
unless data is co-located by an attribute mentioned in GROUP BY statement.

On Sat, Mar 24, 2018 at 7:35 PM, besquared  wrote:

> Queries involving COUNT(DISTINCT ) could push down the expression +
> group by into the mapper. Not doing this This causes an explosion of data
> received by the the reducer. On moderately large tables this usually causes
> OOM but worse than that it's just infeasible to execute at all.
>
> In the example below (and in most I can think of at the moment) this could
> be rewritten by adding a grouping by the given DISTINCT expression (in this
> case account_id). I'd have to think a little more about the more general
> relational case but I wanted to post this here to see if there was already
> discussion going on about this specific optimization path.
>
> CREATE TABLE people ( ID LONG PRIMARY KEY, account_id LONG )
> EXPLAIN SELECT COUNT(DISTINCT account_id) AS account_ids FROM people
>
> Map Phase
>
> SELECT
> __Z0.ACCOUNT_ID AS __C0_0
> FROM PUBLIC.PEOPLE __Z0
> /* PUBLIC.PEOPLE.__SCAN_ */
>
> Reduction Phase
>
> SELECT
> COUNT(DISTINCT __C0_0) AS ACCOUNT_IDS
> FROM PUBLIC.__T0
> /* PUBLIC."merge_scan" */
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Affinity Key column to be always part of the Primary Key

[Vladimir Ozerov]

Because without AFFINITY KEY option we do not know order of fields within
composite PK which is very important for index creation.

вт, 20 марта 2018 г. в 19:58, Dmitriy Setrakyan :

> On Tue, Mar 20, 2018 at 2:09 PM, Vladimir Ozerov 
> wrote:
>
>> Internally Ignite is key-value storage. It use key to derive partition it
>> belongs to. By default the whole key is used. Alternatively you can use
>> @AffinityKey annotation in cache API or "affinityKey" option in CREATE
>> TABLE to specify *part of the key* to be used for affinity calculation.
>> Affinity column cannot belong to value because in this case single
>> key-value pair could migrate between nodes during updates and
>> IgniteCache.get(K) will not be able to locate the key in cluster.
>>
>
> Vladimir, while it makes sense that the key must be composed of the ID and
> Affinity Key, I still do not understand why we require that user declares
> them both as PRIMARY KEY. Why do you need to enforce that explicitly? In my
> view you can do it automatically, if you see that the table has both,
> PRIMARY KEY and AFFINITY KEY declared.
>
>

Re: Affinity Key column to be always part of the Primary Key

[Vladimir Ozerov]

Internally Ignite is key-value storage. It use key to derive partition it
belongs to. By default the whole key is used. Alternatively you can use
@AffinityKey annotation in cache API or "affinityKey" option in CREATE
TABLE to specify *part of the key* to be used for affinity calculation.
Affinity column cannot belong to value because in this case single
key-value pair could migrate between nodes during updates and
IgniteCache.get(K) will not be able to locate the key in cluster.

On Fri, Mar 16, 2018 at 4:56 PM, David Harvey  wrote:

> Yes, the affinity key must be part of the primary key.   Welcome to my
> world
>
> On Fri, Mar 16, 2018 at 3:23 AM, Naveen  wrote:
>
>> Hi Mike
>>
>> I have created a table called CITY
>>
>> : jdbc:ignite:thin://127.0.0.1> CREATE TABLE City (  city_id LONG PRIMARY
>> KEY, name VARCHAR)  WITH "template=replicated";
>> No rows affected (0.224 seconds)
>>
>> Creating a table called Person with affinity key as city_id
>>
>> 0: jdbc:ignite:thin://127.0.0.1> CREATE TABLE IF NOT EXISTS Person ( age
>> int, id int, city_id LONG , name varchar, company varchar,  PRIMARY KEY
>> (name, id)) WITH "template=partitioned,backups=1,affinitykey=city_id,
>> key_type=PersonKey, value_type=MyPerson";
>>
>> This is the exception I get
>>
>> Error: Affinity key column must be one of key columns: CITY_ID
>> (state=42000,code=0)
>> java.sql.SQLException: Affinity key column must be one of key columns:
>> CITY_ID
>> at
>> org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.send
>> Request(JdbcThinConnection.java:671)
>> at
>> org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.execu
>> te0(JdbcThinStatement.java:130)
>> at
>> org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.execu
>> te(JdbcThinStatement.java:299)
>> at sqlline.Commands.execute(Commands.java:823)
>> at sqlline.Commands.sql(Commands.java:733)
>> at sqlline.SqlLine.dispatch(SqlLine.java:795)
>> at sqlline.SqlLine.begin(SqlLine.java:668)
>> at sqlline.SqlLine.start(SqlLine.java:373)
>> at sqlline.SqlLine.main(SqlLine.java:265)
>> 0: jdbc:ignite:thin://127.0.0.1>
>>
>> And, when I change the primary key to include affinity id, below DDL is
>> working fine.
>> 0: jdbc:ignite:thin://127.0.0.1> CREATE TABLE IF NOT EXISTS Person ( age
>> int, id int, city_id LONG , name varchar, company varchar,  PRIMARY KEY
>> (name, id,city_id)) WITH
>> "template=partitioned,backups=1,affinitykey=city_id, key_type=PersonKey,
>> value_type=MyPerson";
>>
>> This is what I was trying to explain, is affinity key to be part of the
>> primary key ??
>>
>> If this is the case, whole my data model will change drastically.
>>
>> Thanks
>> Naveen
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
>
>
> *Disclaimer*
>
> The information contained in this communication from the sender is
> confidential. It is intended solely for use by the recipient and others
> authorized to receive it. If you are not the recipient, you are hereby
> notified that any disclosure, copying, distribution or taking action in
> relation of the contents of this information is strictly prohibited and may
> be unlawful.
>
> This email has been scanned for viruses and malware, and may have been
> automatically archived by *Mimecast Ltd*, an innovator in Software as a
> Service (SaaS) for business. Providing a *safer* and *more useful* place
> for your human generated data. Specializing in; Security, archiving and
> compliance. To find out more Click Here
> .
>

Re: query on BinaryObject index and table

[Vladimir Ozerov]

He Rajesh,

Method CacheConfiguration.setIndexedTypes() should only be used for classes
with SQL annotations. Since you operate on binary objects, you should use
CacheConfiguration.setQueryEntity(), and define QueryEntity with all
necessary fields. Also there is a property QueryEntity.tableName which you
can use to specify concrete table name.

Vladimir.


On Mon, Jan 22, 2018 at 7:41 PM, Denis Magda  wrote:

> The schema can be changed with ALTER TABLE ADD COLUMN command:
> https://apacheignite-sql.readme.io/docs/alter-table
>
> To my knowledge this is supported for schemas that were initially
> configured by both DDL and QueryEntity/Annotations.
>
> —
> Denis
>
>
> On Jan 22, 2018, at 5:44 AM, Ilya Kasnacheev 
> wrote:
>
> Hello Rajesh!
>
> Table name can be specified in cache configuration's query entity. If not
> supplied, by default it is equal to value type name, e.g. BinaryObject :)
>
> Also, note that SQL tables have fixed schemas. This means you won't be
> able to add a random set of fields in BinaryObject and be able to do SQL
> queries on them all. You will have to declare all fields that you are going
> to use via SQL, either by annotations or query entity:
> see https://apacheignite-sql.readme.io/docs/schema-and-indexes
>
> To add index, you should either specify it in annotations (via index=true)
> or in query entity.
>
> Regards,
> Ilya.
>
> --
> Ilya Kasnacheev
>
> 2018-01-21 15:12 GMT+03:00 Rajesh Kishore :
>
>> Hi Denis,
>>
>> This is my code:
>>
>> CacheConfiguration cacheCfg =
>> new CacheConfiguration<>(ORG_CACHE);
>>
>> cacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
>> cacheCfg.setBackups(1);
>> cacheCfg
>> .setWriteSynchronizationMode(CacheWriteSynchronizationMode.F
>> ULL_SYNC);
>> cacheCfg.setIndexedTypes(Long.class, BinaryObject.class);
>>
>> IgniteCache cache = ignite.getOrCreateCache(cacheC
>> fg);
>>
>> if ( UPDATE ) {
>>   System.out.println("Populating the cache...");
>>
>>   try (IgniteDataStreamer streamer =
>>   ignite.dataStreamer(ORG_CACHE)) {
>> streamer.allowOverwrite(true);
>> IgniteBinary binary = ignite.binary();
>> BinaryObjectBuilder objBuilder = binary.builder(ORG_CACHE);
>> ;
>> for ( long i = 0; i < 100; i++ ) {
>>   streamer.addData(i,
>>   objBuilder.setField("id", i)
>>   .setField("name", "organization-" + i).build());
>>
>>   if ( i > 0 && i % 100 == 0 )
>> System.out.println("Done: " + i);
>> }
>>   }
>> }
>>
>> IgniteCache binaryCache =
>> ignite.cache(ORG_CACHE).withKeepBinary();
>> BinaryObject binaryPerson = binaryCache.get(54l);
>> System.out.println("name " + binaryPerson.field("name"));
>>
>>
>> Not sure, If I am missing some context here , if I have to use sqlquery ,
>> what table name should I specify - I did not create table explicitly, do I
>> need to that?
>> How would I create the index?
>>
>> Thanks,
>> Rajesh
>>
>> On Sun, Jan 21, 2018 at 12:25 PM, Denis Magda  wrote:
>>
>>>
>>>
>>> > On Jan 20, 2018, at 7:20 PM, Rajesh Kishore 
>>> wrote:
>>> >
>>> > Hi,
>>> >
>>> > I have requirement that my schema is not fixed , so I have to use the
>>> BinaryObject approach instead of fixed POJO
>>> >
>>> > I am relying on OOTB file system persistence mechanism
>>> >
>>> > My questions are:
>>> > - How can I specify the indexes on BinaryObject?
>>>
>>> https://apacheignite-sql.readme.io/docs/create-index
>>> https://apacheignite-sql.readme.io/docs/schema-and-indexes
>>>
>>> > - If I have to use sql query for retrieving objects , what table name
>>> should I specify, the one which is used for cache name does not work
>>> >
>>>
>>> Was the table and its queryable fields/indexes created with CREATE TABLE
>>> or Java annotations/QueryEntity?
>>>
>>> If the latter approach was taken then the table name corresponds to the
>>> Java type name as shown in this doc:
>>> https://apacheignite-sql.readme.io/docs/schema-and-indexes
>>>
>>> —
>>> Denis
>>>
>>> > -Rajesh
>>>
>>>
>>
>
>

Re: Cannot insert data into table using JDBC

[Vladimir Ozerov]

Hi Michael,

The issue is almost fixed. The fix will be available as a part of Apache
Ignite 2.4 release.

On Tue, Dec 19, 2017 at 1:48 PM, Michael Jay <841519...@qq.com> wrote:

> Hi,has it been solved or is it a bug? I just met with the same
> problem.When I
> set "streaming=false" ,it worked, data can be inserted via
> JdbcClientDriver.
> However, when streaming=true, I got message that said "schema not found" .
> Thanks.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: BinaryObjectImpl.deserializeValue with specific ClassLoader

[Vladimir Ozerov]

The ticket is on the radar, but not in immediate plans. The problem might
sounds simple at first glance, but we already spent considerable time on
implementation and review, because we heavily rely on classes caching, and
a lot of internal BinaryMarshaller infrastructure should be reworked to
allow for this change. Hopefully, we will have it in Apache Ignite 2.5.

On Thu, Jan 4, 2018 at 2:28 AM, Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Ticket is still open. Vladimir, looks like it's assigned to you. Do you
> have any plans to work on it?
>
> https://issues.apache.org/jira/browse/IGNITE-5038
>
> -Val
>
> On Wed, Jan 3, 2018 at 1:26 PM, Abeneazer Chafamo <
> chaf...@nodalexchange.com> wrote:
>
>> Is there any update on the suggested functionality to resolve cache entry
>> classes based on the caller's context first instead of relying on Ignite's
>> classloader?
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
>

Re: insert so slow via JDBC thin driver

[Vladimir Ozerov]

Streaming mode for thin JDBC driver is expected in Apache Ignite 2.5.
Meanwhile you can load data using thick driver which support streaming, and
then switch to thin driver for normal operations it you prefer it over
thick one.


On Fri, Dec 15, 2017 at 9:09 PM, Denis Magda  wrote:

> Hi Michael,
>
> Yes, I heard from Ignite SQL experts that Ignite thin client is not
> optimal for data loading yet. However, there is already some work in
> progress to speed up the loading and batching of a myriad of rows.
>
> Vladimir, could you please step in and comment here.
>
> —
> Denis
>
> > On Dec 14, 2017, at 1:02 AM, Michael Jay <841519...@qq.com> wrote:
> >
> > Hello, I am a new Ignite leaner. I want to insert 50,000,000 rows into a
> > table. Here,i got a problem.
> > When one host and one sever node, the speed of insert is about 2,000,000
> per
> > minute,  the usage of cpu is 30-40%; however two hosts and two sever
> nodes,
> > about 100,000 per minute,and the usage of cpu is only 5%. It's so
> slow,what
> > can i do to improve the performance? Thanks.
> >
> > my default-config.xml:
> >   
> >   
> >   
> >   
> > > class="org.apache.ignite.configuration.MemoryConfiguration">
> >value="19327352832"/>
> >
> >value="16"/>
> >
> >   
> >   
> >   
> >   
> >
> >
> > > class="org.apache.ignite.configuration.CacheConfiguration">
> >   
> >value="PARTITIONED"/>
> >value="4"/>
> >
> >
> >
> >value="false"/>
> >value="false"/>
> >value="false"/>
> >
> >name="rebalanceBatchSize" value="#{256L * 1024 * 1024}"/>
> >
> >value="0"/>
> >
> >name="rebalanceThreadPoolSize" value="8"/>
> >
> >value="false"/>
> >
> >
> >
> >
> >
> > > class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> >
> >
> > > class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.
> TcpDiscoveryMulticastIpFinder">
> >
> >   
> >
> >
> >
>  10x.x.x.226:47500..47509
> >
>  10x.x.x.75:47500..47509
> >
> >
> >
> >
> >
> >
> >   
> > 
> >
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
>

Re: Trouble with v 2.3.0 on AIX

[Vladimir]

Logs uploaded. Node1 started first. Node2 couldn't join the cluster. Search
for first "java.lang.IllegalArgumentException: null" in node2.log 

node1.zip
  

node2.zip
  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: How can I get Ignite security plugin to work with JDBC thin client?

[Vladimir Ozerov]

Hi Caleb,

This appears to be a problem with our query execution engine, rather than
with thin JDBC driver. I created a ticket to fix it [1].

[1] https://issues.apache.org/jira/browse/IGNITE-6856

On Tue, Oct 31, 2017 at 4:49 PM, Andrey Mashenkov <
andrey.mashen...@gmail.com> wrote:

> Caleb,
>
> I've found authorization should work only when you use url like
> "jdbc:ignite://" with thick driver,
> and won't if  "jdbc:ignite:cfg://" is used.
>
> On Tue, Oct 31, 2017 at 4:33 PM, calebs  wrote:
>
>> The javadoc for the jdbc thick client states that property:
>> "ignite.client.credentials" is available to pass "Client credentials used
>> in
>> authentication process."
>>
>> Is this not being used for authentication/authorization?
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
>
>
> --
> Best regards,
> Andrey V. Mashenkov
>

Re: Trouble with v 2.3.0 on AIX

[Vladimir]

No single and simple configuration is available. But notging special: default
or slightly changed memory policy, replicated and partitioned caches (mostly
atomic), write-through persistence, tcp-discovery, continous queries,
services



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Trouble with v 2.3.0 on AIX

[Vladimir]

Hi,

Just upgraded from Ignite 2.1.0 to v.2.3.0. And now our several nodes cannot
start on AIX 7. The error:

ERROR 2017-11-02 11:59:01.331 [grid-nio-worker-tcp-comm-1-#22]
org.apache.ignite.internal.util.nio.GridDirectParser: Failed to read message
[msg=GridIoMessage [plc=0, topic=null, topicOrd=-1, ordered=false,
timeout=0, skipOnTimeout=false, msg=null],
buf=java.nio.DirectByteBuffer[pos=16841 lim=16844 cap=32768],
reader=DirectMessageReader [state=DirectMessageState [pos=0,
stack=[StateItem [stream=DirectByteBufferStreamImplV2
[baseOff=1100144027456, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=true, msg=GridDhtPartitionsFullMessage [parts=null,
partCntrs=null, partCntrs2=null, partHistSuppliers=null, partsToReload=null,
topVer=null, errs=null, compress=false, resTopVer=null, partCnt=0,
super=GridDhtPartitionsAbstractMessage [exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], discoEvt=null,
nodeId=e3ac3f40, evt=NODE_JOINED], lastVer=GridCacheVersion [topVer=0,
order=1509612963224, nodeOrder=0], super=GridCacheMessage [msgId=369,
depInfo=null, err=null, skipPrepare=false]]], mapIt=null, it=null,
arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0,
uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true],
state=0], StateItem [stream=DirectByteBufferStreamImplV2
[baseOff=1100144027456, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=true, msg=CacheGroupAffinityMessage [], mapIt=null, it=null,
arrPos=-1, keyDone=true, readSize=7, readItems=2, prim=0, primShift=0,
uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true],
state=0], StateItem [stream=DirectByteBufferStreamImplV2
[baseOff=1100144027456, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=true, msg=GridLongList [idx=0, arr=[]], mapIt=null, it=null,
arrPos=-1, keyDone=false, readSize=512, readItems=47, prim=0, primShift=0,
uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true],
state=0], StateItem [stream=DirectByteBufferStreamImplV2
[baseOff=1100144027456, arrOff=-1, tmpArrOff=40, tmpArrBytes=40,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], null, null, null,
null, null, null]], lastRead=true], ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=1,
bytesRcvd=404253, bytesSent=1989, bytesRcvd0=16886, bytesSent0=28,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-1,
igniteInstanceName=null, finished=false, hashCode=-2134841549,
interrupted=false, runner=grid-nio-worker-tcp-comm-1-#22]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=16841 lim=16844 cap=32768],
inRecovery=GridNioRecoveryDescriptor [acked=6, resendCnt=0, rcvCnt=0,
sentCnt=6, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode
[id=7683662b-16c9-42b7-aa0d-8328a60fc58e, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:6250], discPort=6250, order=1, intOrder=1,
lastExchangeTime=1509612963744, loc=false, ver=2.3.0#20171028-sha1:8add7fd5,
isClient=false], connected=true, connectCnt=11, queueLimit=131072,
reserveCnt=175, pairedConnections=false],
outRecovery=GridNioRecoveryDescriptor [acked=6, resendCnt=0, rcvCnt=0,
sentCnt=6, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode
[id=7683662b-16c9-42b7-aa0d-8328a60fc58e, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:6250], discPort=6250, order=1, intOrder=1,
lastExchangeTime=1509612963744, loc=false, ver=2.3.0#20171028-sha1:8add7fd5,
isClient=false], connected=true, connectCnt=11, queueLimit=131072,
reserveCnt=175, pairedConnections=false], super=GridNioSessionImpl
[locAddr=/127.0.0.1:6284, rmtAddr=/127.0.0.1:61790,
createTime=1509613141318, closeTime=0, bytesSent=28, bytesRcvd=16886,
bytesSent0=28, bytesRcvd0=16886, sndSchedTime=1509613141318,
lastSndTime=1509613141318, lastRcvTime=1509613141318, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@8e66d834, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true]]]
java.lang.IllegalArgumentException: null
at java.nio.Buffer.position(Buffer.java:255) ~[?:1.8.0]
at
org.apache.ignite.internal.direct.stream.v2.DirectByteBufferStreamImplV2.readArrayLE(DirectByteBufferStreamImplV2.java:1587)
~[ignite-core-2.3.0.jar:2.3.0]
at
org.apache.ignite.internal.direct.stream.v2.DirectByteBufferStreamImplV2.readArrayLE(DirectByteBufferStreamImplV2.java:1542)
~[ignite-core-2.3.0.jar:2.3.0]
at
org.apache.ignite.internal.direct.stream.v2.DirectByteBufferStreamImplV2.readLongArray(DirectByteBufferStreamImplV2.java:1013)
~[ignite-core-2.3.0.jar:2.3.0]
at
org.apache.ignite.internal.direct.DirectMessageReader.readLongArray(DirectMessageReader.java:212)
~[ignite-core-2.3.0.jar:2.3.0]
at
org.apache.ignite.interna

Re: Ignite 2.0.0 GridUnsafe unmonitor

[Vladimir Ozerov]

Guys,

Printing a warning in this case is really strange idea. First, how would
explain it in case of OPTIMISTIC/SERIALIZABLE transactions where deadlocks
are impossible? Second, what would you do in case tow sorted maps are
passed one by one in a transaction? User still may have a deadlock. :Last,
we are going towards SQL world, where "maps" simply do not exist, and
virtually any update could eailty lead to a deadlock.

Let's avoid strange warnings for normal usage scenario. Denis, please close
the ticket :-)))

Vladimir.

On Tue, Oct 31, 2017 at 8:34 PM, Denis Magda  wrote:

> Here is a ticket for the improvement:
> https://issues.apache.org/jira/browse/IGNITE-6804
>
> —
> Denis
>
> > On Oct 31, 2017, at 3:55 AM, Dmitry Pavlov 
> wrote:
> >
> > I agree with Denis, if we don't have such warning we should continiously
> warn users in wiki pages/blogs/presentations. It is simpler to warn from
> code.
> >
> > What do you think if we will issue warning only if size > 1. HashMap
> with 1 item will not cause deadlock. Moreover where can be some custom
> singleton Map provided by user.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > вт, 31 окт. 2017 г. в 7:18, Dmitriy Setrakyan  <mailto:dsetrak...@apache.org>>:
> > Denis,
> >
> > We should definitely print out a thorough warning if HashMap is passed
> into
> > a bulk method (instead of SortedMap). However, we should make sure that
> we
> > only print that warning once and not ever time the API is called.
> >
> > Can you please file a ticket for 2.4?
> >
> > D.
> >
> > On Thu, Oct 26, 2017 at 11:05 AM, Denis Magda  <mailto:dma...@apache.org>> wrote:
> >
> > > + dev list
> > >
> > > Igniters, that’s a relevant point below. Newcomers to Ignite tend to
> > > stumble on deadlocks simply because the keys are passed in an unordered
> > > HashMap. Propose to do the following:
> > > - update bulk operations Java doc.
> > > - print out a warning if a HashMap is used and its exceeds one element.
> >
> >
> > > Thoughts?
> > >
> > > —
> > > Denis
> > >
> > > > On Oct 21, 2017, at 6:16 PM, dark  ekdxhrl0...@gmail.com>> wrote:
> > > >
> > > > Many people seem to be more likely to send Cache entries in bulk via
> a
> > > > HashMap.
> > > > How do you expose a warning statement by checking if the TreeMap is
> > > putAll
> > > > inside the code?
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ <
> http://apache-ignite-users.70518.x6.nabble.com/>
> > >
> > >
>
>

Re: Query performance against table with/out backup

[Vladimir Ozerov]

Hi,

We know one serious source of slowdown when backups are enabled. See [1]
and [2]. It will be fixed in 2.4.

Vladimir.

[1] https://issues.apache.org/jira/browse/IGNITE-6624
[2] https://issues.apache.org/jira/browse/IGNITE-6626

On Thu, Oct 19, 2017 at 9:29 PM, blackfield 
wrote:

> Here, I am trying to ascertain that I set backup == 2 properly as I
> mentioned
> above that I do not see query performance difference between backup ==1 and
> backup == 2.
>
> I want to make sure that I configure my cache properly.
>
> When I set the backup==2 (to have three copies), I notice the following via
> visor.
>
> The Affinity Backups is still equal to 1. Is this a different property than
> number of backups? If it is not, how do one see the number of backups a
> cache is configured for?
>
> Invoking "cache -a" to see the detail cache stat, with backup==2, under the
> size column, the sum of entries on all nodes is equal to the number of rows
> in the table * 2.  It appears this is the case for backup >= 1?
>
> As in, only one set of backup will be stored in off heap regardless the
> number of backups are specified?
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Why SQL_PUBLIC is appending to Cache name while using JDBC thin driver

[Vladimir Ozerov]

Hi Austin,

All caches in Ignite must have unique names. This is why we add unique
prefix with schema name attached.

On Sat, Oct 7, 2017 at 1:17 PM, austin solomon 
wrote:

> Hi,
>
> I am using Ignite version 2.2.0, and I have created a table using
> IgniteJdbcThinDriver.
>
> When I checked the cache in Ignite Visor I'm seeing SQL_PUBLIC_{TABLE-NAME}
> is appended.
> Is their a way to get rid of this.
>
> I want to remove the SQL_PUBLIC from the cache name.
>
> Thanks,
> Austin
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: work around for problem where ignite query does not include objects added into Cache from within a transaction

[Vladimir Ozerov]

Hi,

Looks like you need the feature which is simply not implement in Ignite at
the moment. SELECTs are not transactional and there is not easy way to make
not-yet-committed updates visible to them. Solution 1 doesn't work as there
are not "temporal" caches in Ignite. Not-yet-committed updates are stored
in some internal data structures. Solution 2 would lead to dramatic
performance degradation.

I would suggest you to rethink the whole approach - SELECTs do have any
transactional guarantees at the moment. May be you will be able to split
single transaction into multiple and then run SQL query in the middle.

Vladimir.

On Sat, Sep 16, 2017 at 10:17 AM, kotamrajuyashasvi <
kotamrajuyasha...@gmail.com> wrote:

> Hi
>
> When a transaction is started and some rows are updated/inserted/deleted in
> a cache which is in transactional mode, And after that when I perform a
> select query, its not reflecting the results from of update/insert/delete
> done before with in the transaction.  I found out that ignite sql does not
> have this functionality yet. But I need a temporary workaround for this
> problem. I thought of two possible solutions. I do not know the feasibility
> of these solutions and also its impacts on performance or functionality.
> Hence I need your help.
>
> solution 1: Assuming all updates/inserts/deletes with in transaction are
> temporarily stored in some cache, and then flushed to original cache when
> commited or rolledback, Is there any way I could access that cache to
> perform select query on that temporary cache and then decide what rows
> should I include as result of select query.
>
> solution 2: When ever I start a Transaction, for every original cache, I
> initialize a new dynamic temporary cache which is not in  transactional
> mode[also does not have any persistent store] using 'getOrCreateCache(..)'
> as mentioned in docs. Now within transaction when ever any update/insert
> happens I insert into the corresponding temporary cache. During the select
> query I perform select query on both main and corresponding temporary
> cache.
> Then I take results from both the queries and decide the actual result.I
> also maintain a list of rows/keys deleted within transaction, and during
> select query I will exclude the keys present in this list. During commit I
> sync the temporary caches into original caches. Will creating temporary
> caches for every transaction degrade performance? Is there any way to
> create
> a temporary cache local to transaction or node so that it does not affect
> performance, since Im using temporary cache mainly for performing sql
> queries.
>
>
> Please suggest if there are any drawbacks to the above solutions and also
> if
> there are any other workarounds for this problem.
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: GROUP_CONCAT function is unsupported

[Vladimir Ozerov]

Hi,

This function is not supported yet. No immediate plans to fix it as there
are lots of tasks with higher priority and impact. But will definitely
support it at some point.

On Wed, Sep 6, 2017 at 11:38 AM, mhetea  wrote:

> Hello,
> I saw that the ticket was not fixed.
> Is there a workaround for this issue?
> I wouldn't like to iterate over the results and then concatenate them
> because the result list can be large.
> Thank you!
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: New API docs look for Ignite.NET

[Vladimir Ozerov]

Looks good to me. Clean layout, fast build - do we need anything else?

On Tue, Sep 5, 2017 at 12:06 PM, Oleg Ostanin  wrote:

> Great news, thanks a lot!
>
> On Tue, Sep 5, 2017 at 11:47 AM, Pavel Tupitsyn 
> wrote:
>
>> DocFX takes around 30 seconds on my machine.
>>
>> > if you already tried that
>> Yes, everything is done on my side, see JIRA ticket [4] and preview [5]
>> above.
>>
>> On Tue, Sep 5, 2017 at 11:45 AM, Ilya Suntsov 
>> wrote:
>>
>> > Pavel, thanks!
>> > It is the great news!
>> > Looks like DocFX will save 30-40 min.
>> >
>> > 2017-09-05 11:16 GMT+03:00 Pavel Tupitsyn :
>> >
>> > > Igniters and users,
>> > >
>> > > Historically we've been using Doxygen [1] to generate .NET API
>> > > documentation [2].
>> > >
>> > > Recently it became very slow on our code base (more than 30 minutes to
>> > > generate), and I could not find any solution or tweak to fix that.
>> Other
>> > > issues include outdated looks and limited customization possibilities.
>> > >
>> > > I propose to replace it with DocFX [3] [4]:
>> > > - Popular .NET Foundation project
>> > > - Good looks and usability out of the box
>> > > - Easy to set up
>> > >
>> > > Our docs will look like this: [5]
>> > > Let me know if you have any objections or suggestions.
>> > >
>> > > Pavel
>> > >
>> > >
>> > > [1] http://www.stack.nl/~dimitri/doxygen/
>> > > [2] https://ignite.apache.org/releases/latest/dotnetdoc/index.html
>> > > [3] https://dotnet.github.io/docfx/
>> > > [4] https://issues.apache.org/jira/browse/IGNITE-6253
>> > > [5] https://ptupitsyn.github.io/docfx-test/api/index.html
>> > >
>> >
>> >
>> >
>> > --
>> > Ilya Suntsov
>> >
>>
>
>

Re: SQL query is slow

[Vladimir Ozerov]

Hi Mihaela,

Index is not used in your case because you specify function-based
condition. Usually this is resolved by adding functional index, but Ignite
doesn't support it at the moment unfortunately. Is it possible to
"materialize" the condition "POSITION ('Z',manufacturerCode)>0" as
additional attribute and add an index on it? In this case SQL would look
like this and index will be used:

SELECT COUNT(_KEY) FROM IgniteProduct AS product
WHERE manufacturerCodeZ=1

Another important thing is selectivity - which fraction of records fall
under this condition?
Also I would recommend to change "COUNT(_KEY)" to "COUNT(*)".

Vladimir.

On Tue, Aug 29, 2017 at 6:05 PM, Andrey Mashenkov <
andrey.mashen...@gmail.com> wrote:

> It is possible returned dataset is too large and cause high network
> pressure that results in large query execution time.
>
> There is no recommendation for grid nodes count.
> Simple SQL queries can work slower on large grid as most of time is spent
> in inter-node communication.
> Heavy SQL queries may show better results on larger grid as every node
> will have smaller dataset.
>
> You can try to look at page memory statistics [1] to get estimate numbers.
>
> Really, there is an issue with large OFFSET as Ignite can't just skip
> entries and have to fetch all of them from nodes.
> OFFSET makes no sense without ORDER as Ignite fetch rows from other nodes
> in async way and row order should be preserved between such queries.
> OFFSET applies on query initiator node (reduce side) after results merged
> as there is no way to understand on map side what rows should be skiped.
>
>
> Looks like underlying H2 tries to use index scan, but I don't think index
> can help in case of functional condition.
> You can try to make Ignite to have inline values in index or use separate
> field with smaller type that can be inlined. By default, index inlining is
> enabled for 10 byte length values.
> See IGNITE_MAX_INDEX_PAYLOAD_SIZE_DEFAULT system property docs and [2].
>
> [1] https://apacheignite.readme.io/v2.1/docs/memory-metrics
> [2] https://issues.apache.org/jira/browse/IGNITE-6060
>
> On Tue, Aug 29, 2017 at 3:59 PM, mhetea  wrote:
>
>> Thank you for your response.
>> I used query parallelizm and the time reduced to ~2.3s, which is still too
>> much.
>> Regarding 1. is there any documentation about configuration parameters
>> (recommended number of nodes, how much data should be stored on each
>> node).
>> We currently have 2 nodes with 32GB RAM each. Every 1 million records from
>> our cache occupy about 1GB (is there a way to see how much memory a cache
>> actually occupies? we look now at the Allocated next memory segment log
>> info)
>> For 3. it seems that the index is hit  from the execution plan:
>>  /* "productCache".IGNITEPRODUCT_MANUFACTURERCODE_IDX */
>> No?
>>
>> We have this issue also when we use a large OFFSET (we execute this kind
>> of
>> query because we want paginated results)
>>
>> Also, this cache will be updated frequently so we expect it to grow in
>> size.
>>
>> Thank you!
>>
>>
>>
>> --
>> View this message in context: http://apache-ignite-users.705
>> 18.x6.nabble.com/SQL-query-is-slow-tp16475p16487.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>
>
> --
> Best regards,
> Andrey V. Mashenkov
>

Re: Cache performance in multi-threaded environment

[Vladimir Chmakov]

Hi Nikolai,

Thank you for the prompt reply. 
Hopefully I fixed the issues with subscription, thank you pointing at that.
You suggested option #1 for the cache instance usage when it is cached
locally and used by multiple threads. Actually, this is the way we
implemented our PoC and we will keep up with this approach – thank you for
the suggestion. Also your advice renders my second question invalid as the
approach I was asking about seems to be wrong, so the reproducer doesn’t
make much sense. 
We will keep testing with respect to the performance numbers and will advise
if we find something that would alarm us.

Regards, 
Vladimir






--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Cache-performance-in-multi-threaded-environment-tp15698p15738.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: NodeFilter for cache and GridDhtPartitionsExchangeFuture (Failed to wait for partition release future)

[Vladimir]

>>> What is the exact scenario when cache is not available? If this means
absence of node A, then you can lose after initialization of B,

It is just a test. In real project, of course, there are many nodes holding
the cache. Node A represents a part of cluster.


>>> I would recommend to use Ignite#getOrCreateCache method 

Yes, on the nodes keeping and serving the cache. "Client" nodes don't
provide the cache configuration. They can only use cache(). I moved the
cache acquring in a handler of ContextRefreshedEvent. Looks enough for now.


>>> In addition, it sounds like node B can be a client which eliminates a
>>> requirement to have a node filter.

No. It's a server node for another purposes. We don't want to spread some
caches across whole cluster bit only on dedicated nodes. Often they are the
ones which need faster access to the cache.


>>> This sounds weird, I think there is some other factor that we're
>>> missing. Can you create a simple GitHub project that reproduces this
>>> behavior and share it with us? 

Ok. Once I get time I'll create a special topic.

Thanks



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/NodeFilter-for-cache-and-GridDhtPartitionsExchangeFuture-Failed-to-wait-for-partition-release-future-tp14179p14489.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: NodeFilter for cache and GridDhtPartitionsExchangeFuture (Failed to wait for partition release future)

[Vladimir]

Node B must know wheter the cache is already created. If not it must not
start to avoid such runtime problems. That's why the cache is acquired at
the initialization. That looks like reasonable and comfortable way. There
are several interesting issues:

1) Why does node B acquire that cache store bean? This node does not hold,
service and create cache due to the node filter. Note that this node doesn't
even have the cache config. Node B is supposed to seldom use the cache as a
remote source without working directly with the database. I was forced to
create properly named datasource bean only to satisfy the requirement which
actually relates to other node (Node A)!

2) Why no problem is met when Node B matches the node filter?



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/NodeFilter-for-cache-and-GridDhtPartitionsExchangeFuture-Failed-to-wait-for-partition-release-future-tp14179p14308.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: NodeFilter for cache and GridDhtPartitionsExchangeFuture (Failed to wait for partition release future)

[Vladimir]

YourKits thinks:

Frozen threads found (potential deadlock)

It seems that the following threads have not changed their stack for more
than 10 seconds.
These threads are possibly (but not necessarily!) in a deadlock or hung.

exchange-worker-#29%null% <--- Frozen for at least 14s
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(String,
ObjectFactory) DefaultSingletonBeanRegistry.java:213
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(String,
Class, Object[], boolean) AbstractBeanFactory.java:302
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(String)
AbstractBeanFactory.java:197
org.springframework.context.support.AbstractApplicationContext.getBean(String)
AbstractApplicationContext.java:1081
org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadBeanFromAppContext(Object,
String) IgniteSpringHelperImpl.java:217
org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory.create()
CacheJdbcPojoStoreFactory.java:178
org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory.create()
CacheJdbcPojoStoreFactory.java:100
org.apache.ignite.internal.processors.cache.GridCacheProcessor.createCache(CacheConfiguration,
CachePluginManager, CacheType, CacheObjectContext, boolean)
GridCacheProcessor.java:1458
org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheStart(CacheConfiguration,
NearCacheConfiguration, CacheType, boolean, UUID, IgniteUuid,
AffinityTopologyVersion, QuerySchema) GridCacheProcessor.java:1931
org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheStart(DynamicCacheChangeRequest,
AffinityTopologyVersion) GridCacheProcessor.java:1833
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.onCacheChangeRequest(GridDhtPartitionsExchangeFuture,
boolean, Collection) CacheAffinitySharedManager.java:379
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onCacheChangeRequest(boolean)
GridDhtPartitionsExchangeFuture.java:688
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init()
GridDhtPartitionsExchangeFuture.java:529
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body()
GridCachePartitionExchangeManager.java:1806
org.apache.ignite.internal.util.worker.GridWorker.run() GridWorker.java:110
java.lang.Thread.run() Thread.java:745




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/NodeFilter-for-cache-and-GridDhtPartitionsExchangeFuture-Failed-to-wait-for-partition-release-future-tp14179p14272.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: NodeFilter for cache and GridDhtPartitionsExchangeFuture (Failed to wait for partition release future)

[Vladimir]

Here it is:

logs_dumps.zip
  



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/NodeFilter-for-cache-and-GridDhtPartitionsExchangeFuture-Failed-to-wait-for-partition-release-future-tp14179p14270.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

NodeFilter for cache and GridDhtPartitionsExchangeFuture (Failed to wait for partition release future)

[Vladimir]

Hi,

Could anyone please explain to me what happens with my small test cluster
when I use NodeFilter for a cache? A node cannot join the grid. Another
working node says (in short):

[exchange-worker-#29%null%] ... GridDhtPartitionsExchangeFuture: Failed to
wait for partition release future [topVer=AffinityTopologyVersion [topVer=2,
minorTopVer=2], node=...]. Dumping pending objects that might be the cause: 

GridCachePartitionExchangeManager: Pending exchange futures:
GridCachePartitionExchangeManager: Pending transactions: 
GridCachePartitionExchangeManager: Pending explicit locks:
GridCachePartitionExchangeManager: Pending cache futures:
GridCachePartitionExchangeManager: >>> GridDhtTxPrepareFuture [...
GridCachePartitionExchangeManager: >>> GridNearPessimisticTxPrepareFuture
[...
GridCachePartitionExchangeManager: >>> GridNearTxFinishFuture [...
GridCachePartitionExchangeManager: Pending atomic cache futures:
GridCachePartitionExchangeManager: Pending data streamer futures:
GridCachePartitionExchangeManager: Pending transaction deadlock detection
futures:
TcpCommunicationSpi: Communication SPI recovery descriptors:  [...]
   Communication SPI clients [...]
TcpCommunicationSpi: NIO server statistics [readerSesBalanceCnt=0,
writerSesBalanceCnt=0]


org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi: 
>> Selector info [idx=3, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0,
>> bytesSent0=0]
...
...


and spams it again, again and again. No pendings are posted into the log. No
locks, no transactions etc.


What I do is launching only 2 very simple nodes. Let's name them A and B.
Node A deploys two indexed caches with read/write-thorugh feature. Both
caches have the same node filter which bounds it strictly to node A. After,
node B starts and tries to get these caches. And the problem appears. I
thought I can get a proxy of a cache if it is not hold on the current node.
Can't I? What’s wrong?



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/NodeFilter-for-cache-and-GridDhtPartitionsExchangeFuture-Failed-to-wait-for-partition-release-future-tp14179.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: using hdfs as secondary file system

[Vladimir Ozerov]

Hi Antonio,

Is it possible to attach logs from server nodes?



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/using-hdfs-as-secondary-file-system-tp13580p13696.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: odd query plan with joins?

[Vladimir Ozerov]

Hi,

"1=1" is normal thing in the plan provided that index is used (see "/*
PropertyCache.PROPERTY_CITY_ID_IDX: CITY_ID = 59053 */"). Is it possible to
attach a reproducer?

Vladimir.

On Wed, Jun 14, 2017 at 5:10 AM, djardine  wrote:

> Hi,
> I'm attempting to run a query that does a join between 2 large tables (one
> with 6m rows, another with 80m rows). In the query plan I see a join "on
> 1=1" and separately i see filters for my join under the "where" clause. I'm
> not sure if this is standard output in the query plan or if it's doing a
> ridiculously expensive join where it's combining every possible permutation
> between the tables and then filtering it down. The query basically runs
> forever, never returning and eventually will kill the server node that it's
> running on (maybe OOM?). I've tried this on both PARTITIONED and REPLICATED
> clusters. "property_id" fields are indexed.
>
> The query is:
>
> SELECT p.property_id, sum(cm.days_r)::real/(sum(cm.days_r)+sum(cm.days_a))
> as occ_rate, p.bedrooms, cm.period, p.room_type
> FROM calendar_metric as cm
> JOIN "PropertyCache".property as p on
> p.property_id = cm.property_id
> WHERE
> cm.days_r > 0 AND p.bedrooms is not null AND (
> p.room_type = 'Entire
> home/apt' )
> AND cm.period BETWEEN '2016-1-1' and '2016-8-1'
> AND p.city_id = 59053
> GROUP BY cm.period, p.room_type, p.bedrooms, p.property_id
>
> The query plan shows:
>
> SELECT
> P__Z1.PROPERTY_ID AS __C0_0,
> SUM(CM__Z0.DAYS_R) AS __C0_1,
> P__Z1.BEDROOMS AS __C0_2,
> CM__Z0.PERIOD AS __C0_3,
> P__Z1.ROOM_TYPE AS __C0_4,
> SUM(CM__Z0.DAYS_R) AS __C0_5,
> SUM(CM__Z0.DAYS_A) AS __C0_6
> FROM CalendarMetricCache.CALENDAR_METRIC CM__Z0
> /* CalendarMetricCache.CALENDAR_METRIC_PERIOD_DAYS_R_IDX: PERIOD >=
> DATE
> '2016-01-01'
> AND PERIOD <= DATE '2016-08-01'
> AND DAYS_R > 0
>  */
> /* WHERE (CM__Z0.DAYS_R > 0)
> AND ((CM__Z0.PERIOD >= DATE '2016-01-01')
> AND (CM__Z0.PERIOD <= DATE '2016-08-01'))
> */
> INNER JOIN PropertyCache.PROPERTY P__Z1
> /* PropertyCache.PROPERTY_CITY_ID_IDX: CITY_ID = 59053 */
> ON 1=1
> WHERE (P__Z1.PROPERTY_ID = CM__Z0.PROPERTY_ID)
> AND ((P__Z1.CITY_ID = 59053)
> AND (((CM__Z0.PERIOD >= DATE '2016-01-01')
> AND (CM__Z0.PERIOD <= DATE '2016-08-01'))
> AND ((P__Z1.ROOM_TYPE = 'Entire home/apt')
> AND ((CM__Z0.DAYS_R > 0)
> AND (P__Z1.BEDROOMS IS NOT NULL)
> GROUP BY CM__Z0.PERIOD, P__Z1.ROOM_TYPE, P__Z1.BEDROOMS, P__Z1.PROPERTY_ID
>
> SELECT
> __C0_0 AS PROPERTY_ID,
> (CAST(SUM(__C0_1) AS REAL) / (SUM(__C0_5) + SUM(__C0_6))) AS OCC_RATE,
> __C0_2 AS BEDROOMS,
> __C0_3 AS PERIOD,
> __C0_4 AS ROOM_TYPE
> FROM PUBLIC.__T0
> /* CalendarMetricCache.merge_scan */
> GROUP BY __C0_3, __C0_4, __C0_2, __C0_0
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/odd-query-plan-with-joins-tp13680.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Failed to wait for initial partition map exchange

[Vladimir]

INFO  2017-06-13 20:36:22 [localhost-startStop-1]
org.apache.ignite.internal.IgniteKernal%svip - [[OS: Linux
4.1.12-61.1.18.el7uek.x86_64 amd64]]
INFO  2017-06-13 20:36:22 [localhost-startStop-1]
org.apache.ignite.internal.IgniteKernal%svip - [[PID: 7608]]
[20:36:22] VM information: Java(TM) SE Runtime Environment 1.8.0_102-b14
Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.102-b14
INFO  2017-06-13 20:36:22 [localhost-startStop-1]
org.apache.ignite.internal.IgniteKernal%svip - [[Language runtime: Java
Platform API Specification ver. 1.8]]
INFO  2017-06-13 20:36:22 [localhost-startStop-1]
org.apache.ignite.internal.IgniteKernal%svip - [[VM information: Java(TM) SE
Runtime Environment 1.8.0_102-b14 Oracle Corporation Java HotSpot(TM) 64-Bit
Server VM 25.102-b14]]
INFO  2017-06-13 20:36:22 [localhost-startStop-1]
org.apache.ignite.internal.IgniteKernal%svip - [[VM total memory: 2.1GB]]
INFO  2017-06-13 20:36:22 [localhost-startStop-1]
org.apache.ignite.internal.IgniteKernal%svip - [[Remote Management [restart:
off, REST: on, JMX (remote: off)]]]
INFO  2017-06-13 20:36:22 [localhost-startStop-1]
org.apache.ignite.internal.IgniteKernal%svip - [[System cache's MemoryPolicy
size is configured to 40 MB. Use MemoryConfiguration.systemCacheMemorySize
property to change the setting.]]
INFO  2017-06-13 20:36:22 [localhost-startStop-1]
org.apache.ignite.internal.IgniteKernal%svip - [[Configured caches [in
'default' memoryPolicy: ['ignite-sys-cache', 'ignite-atomics-sys-cache'
INFO  2017-06-13 20:36:22 [localhost-startStop-1]
org.apache.ignite.internal.IgniteKernal%svip - [[Local node user attribute
[IgSupport_LogicClusterGroups=com.bpcbt.common.support.ignite.beans.IgLogicClusterGroups@0]]]
WARN  2017-06-13 20:36:22 [pub-#14%svip%]
org.apache.ignite.internal.GridDiagnostic - [[Initial heap size is 154MB
(should be no less than 512MB, use -Xms512m -Xmx512m).]]
[20:36:22] Initial heap size is 154MB (should be no less than 512MB, use
-Xms512m -Xmx512m).
[20:36:23] Configured plugins:
INFO  2017-06-13 20:36:23 [localhost-startStop-1]
org.apache.ignite.internal.processors.plugin.IgnitePluginProcessor -
[[Configured plugins:]]
[20:36:23]   ^-- None
INFO  2017-06-13 20:36:23 [localhost-startStop-1]
org.apache.ignite.internal.processors.plugin.IgnitePluginProcessor - [[  ^--
None]]
[20:36:23] 
INFO  2017-06-13 20:36:23 [localhost-startStop-1]
org.apache.ignite.internal.processors.plugin.IgnitePluginProcessor - [[]]
INFO  2017-06-13 20:36:23 [localhost-startStop-1]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[Successfully
bound communication NIO server to TCP port [port=9343, locHost=/127.0.0.1,
selectorsCnt=4, selectorSpins=0, pairedConn=false]]]
WARN  2017-06-13 20:36:23 [localhost-startStop-1]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[Message
queue limit is set to 0 which may lead to potential OOMEs when running cache
operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth
on sender and receiver sides.]]
[20:36:23] Message queue limit is set to 0 which may lead to potential OOMEs
when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to
message queues growth on sender and receiver sides.
WARN  2017-06-13 20:36:23 [localhost-startStop-1]
org.apache.ignite.spi.checkpoint.noop.NoopCheckpointSpi - [[Checkpoints are
disabled (to enable configure any GridCheckpointSpi implementation)]]
WARN  2017-06-13 20:36:23 [localhost-startStop-1]
org.apache.ignite.internal.managers.collision.GridCollisionManager -
[[Collision resolution is disabled (all jobs will be activated upon
arrival).]]
[20:36:23] Security status [authentication=off, tls/ssl=off]
INFO  2017-06-13 20:36:23 [localhost-startStop-1]
org.apache.ignite.internal.IgniteKernal%svip - [[Security status
[authentication=off, tls/ssl=off]]]
INFO  2017-06-13 20:36:23 [localhost-startStop-1]
org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestProtocol
- [[Command protocol successfully started [name=TCP binary,
host=0.0.0.0/0.0.0.0, port=11214]]]
INFO  2017-06-13 20:36:23 [localhost-startStop-1]
org.apache.ignite.internal.IgniteKernal%svip - [[Non-loopback local IPs:
192.168.122.1, 192.168.209.65]]
INFO  2017-06-13 20:36:23 [localhost-startStop-1]
org.apache.ignite.internal.IgniteKernal%svip - [[Enabled local MACs:
0021F6321229, 5254000A6937]]
INFO  2017-06-13 20:36:23 [localhost-startStop-1]
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi - [[Successfully bound
to TCP port [port=9463, localHost=/127.0.0.1,
locNodeId=7fc5ea30-913d-4ff5-932b-62d81c6027db]]]
INFO  2017-06-13 20:36:24 [localhost-startStop-1]
org.apache.ignite.internal.processors.cache.GridCacheProcessor - [[Started
cache [name=ignite-sys-cache, memoryPolicyName=sysMemPlc, mode=REPLICATED]]]
INFO  2017-06-13 20:36:24 [localhost-startStop-1]
org.apache.ignite.internal.processors.cache.GridCacheProcessor - [[Started
cache [name=ignite-atomics-sys-cache, memoryPolicyName=sysMemPlc,
mode=REPLICATED]]]
INFO  2017-06-13 2

Node can't start. java.lang.NullPointerException in GridUnsafe.compareAndSwapLong()

[Vladimir]

Hi,

Having no problems on Windows and Linux application suddenly couldn't start
on IBM AIX with IBM J9 VM (build 2.8):

Caused by: java.lang.NullPointerException
at
org.apache.ignite.internal.util.GridUnsafe.compareAndSwapLong(GridUnsafe.java:1228)
~[ignite-core-2.0.0.jar!/:2.0.0]
at
org.apache.ignite.internal.util.OffheapReadWriteLock.readLock(OffheapReadWriteLock.java:122)
~[ignite-core-2.0.0.jar!/:2.0.0]
at
org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.readLock(PageMemoryNoStoreImpl.java:450)
~[ignite-core-2.0.0.jar!/:2.0.0]
at
org.apache.ignite.internal.processors.cache.database.tree.util.PageHandler.readLock(PageHandler.java:181)
~[ignite-core-2.0.0.jar!/:2.0.0]
at
org.apache.ignite.internal.processors.cache.database.tree.util.PageHandler.readPage(PageHandler.java:152)
~[ignite-core-2.0.0.jar!/:2.0.0]
at
org.apache.ignite.internal.processors.cache.database.DataStructure.read(DataStructure.java:319)
~[ignite-core-2.0.0.jar!/:2.0.0]
at
org.apache.ignite.internal.processors.cache.database.tree.BPlusTree.findDown(BPlusTree.java:1115)
~[ignite-core-2.0.0.jar!/:2.0.0]
at
org.apache.ignite.internal.processors.cache.database.tree.BPlusTree.doFind(BPlusTree.java:1084)
~[ignite-core-2.0.0.jar!/:2.0.0]
at
org.apache.ignite.internal.processors.cache.database.tree.BPlusTree.findOne(BPlusTree.java:1048)
~[ignite-core-2.0.0.jar!/:2.0.0]
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.find(IgniteCacheOffheapManagerImpl.java:1143)
~[ignite-core-2.0.0.jar!/:2.0.0]
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.read(IgniteCacheOffheapManagerImpl.java:361)
~[ignite-core-2.0.0.jar!/:2.0.0]
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:384)
~[ignite-core-2.0.0.jar!/:2.0.0]
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerGet0(GridCacheMapEntry.java:588)
~[ignite-core-2.0.0.jar!/:2.0.0]
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerGet(GridCacheMapEntry.java:474)
~[ignite-core-2.0.0.jar!/:2.0.0]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.localGet(GridPartitionedSingleGetFuture.java:380)
~[ignite-core-2.0.0.jar!/:2.0.0]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.mapKeyToNode(GridPartitionedSingleGetFuture.java:326)
~[ignite-core-2.0.0.jar!/:2.0.0]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.map(GridPartitionedSingleGetFuture.java:211)
~[ignite-core-2.0.0.jar!/:2.0.0]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.init(GridPartitionedSingleGetFuture.java:203)
~[ignite-core-2.0.0.jar!/:2.0.0]
at
org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedCache.getAsync(GridDhtColocatedCache.java:266)
~[ignite-core-2.0.0.jar!/:2.0.0]
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.get0(GridCacheAdapter.java:4482)
~[ignite-core-2.0.0.jar!/:2.0.0]
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:4463)
~[ignite-core-2.0.0.jar!/:2.0.0]
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:1405)
~[ignite-core-2.0.0.jar!/:2.0.0]
at
org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.getAtomic(DataStructuresProcessor.java:586)
~[ignite-core-2.0.0.jar!/:2.0.0]
at
org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.sequence(DataStructuresProcessor.java:396)
~[ignite-core-2.0.0.jar!/:2.0.0]
at
org.apache.ignite.internal.IgniteKernal.atomicSequence(IgniteKernal.java:3419)
~[ignite-core-2.0.0.jar!/:2.0.0]

Any workarounds?



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Node-can-t-start-java-lang-NullPointerException-in-GridUnsafe-compareAndSwapLong-tp13573.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: How to properly close igfs file after appending it

[Vladimir Ozerov]

Is it possible to provide a reproducer?

On Fri, Jun 9, 2017 at 9:11 AM, ishan-jain  wrote:

> Ivan please ellaborate
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/How-to-properly-close-igfs-file-after-appending-it-
> tp11613p13551.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Memory Leak or my Error?

[Vladimir Ozerov]

Hi Devis,

Behavior you saw doesn't depend on how many rows are returned, but rather
on how many threads are created. There should not be any problems when
switching from 100 to 2000 rows.

On Wed, Jun 7, 2017 at 9:20 PM, Devis Balsemin <
devis.balse...@flexvalley.com> wrote:

> Hi Vladimir,
>
> for now i have change my code in this way:
>
>
>
> From
>
> queryCursor = *cache*.query(*new *SqlQuery(cls, whereCondition).setArgs(
> whereParams));
>
>
>
> With
>
> cache.iterator() and Java 8 Stream/Filter/….
>
>
>
> My memory now works  very fine over 37.4% less of resources and CMS objects 
> in garbage
>
>
>
>
>
> What do you think about this?
>
>
>
> For your what will happen with over 2000 Rows (now I’m testing only with 100 
> rows)?
>
>  Have I  to wait for a decrease in performance with this approach?
>
>
>
> Thank for your help
>
> Devis
>
>
>
> [image: cid:image001.png@01D12220.7E4C4300]
>
>
>
>
>
> Devis Balsemin,  FlexValley SRL (Italy) ,
> Via Dante Alighieri 4, I-36075 Montecchio Maggiore, Italy | t: +39.0444.699622
> | f: +39.0444.1830543
>
>
>
> *Please consider your environmental responsibility before printing this
> e-mail*
>
> This communication (including any attachments) may contain privileged or
> confidential information of FlexValley SRL and is intended for a specific
> individual.  If you are not the intended recipient, you should delete this
> communication, including any attachments without reading or saving them in
> any manner, and you are hereby notified that any disclosure, copying, or
> distribution of this communication, or the taking of any action based on
> it, is strictly prohibited.
>
>
>
> *Da:* devis76 [mailto:devis.balse...@flexvalley.com]
> *Inviato:* martedì 6 giugno 2017 17:12
> *A:* user@ignite.apache.org
> *Oggetto:* R: Memory Leak or my Error?
>
>
>
> Hi Vladimir,
>
> sorry…
>
> when i call “submit”….
>
> I’m using a IgniteThreadPoolExecutor in this way….
>
>
>
> *threadPoolExecutor *= *new *IgniteThreadPoolExecutor(coreSize, maxSize,
> keepAlive, *arrayBlockingQueue*, *threadFactory*, *new *
> ThreadPoolExecutor.CallerRunsPolicy());
> *threadPoolExecutor*.allowCoreThreadTimeOut(*true*);
>
>
>
>
>
> [image: cid:image001.png@01D12220.7E4C4300]
>
>
>
>
>
> Devis Balsemin,  FlexValley SRL (Italy) ,
> Via Dante Alighieri 4, I-36075 Montecchio Maggiore, Italy | t: +39.0444.699622
> | f: +39.0444.1830543
>
>
>
> *Please consider your environmental responsibility before printing this
> e-mail*
>
> This communication (including any attachments) may contain privileged or
> confidential information of FlexValley SRL and is intended for a specific
> individual.  If you are not the intended recipient, you should delete this
> communication, including any attachments without reading or saving them in
> any manner, and you are hereby notified that any disclosure, copying, or
> distribution of this communication, or the taking of any action based on
> it, is strictly prohibited.
>
>
>
> *Da:* Vladimir Ozerov [via Apache Ignite Users] [mailto:[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=13429&i=0>]
> *Inviato:* martedì 6 giugno 2017 16:47
> *A:* devis76 <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=13429&i=1>>
> *Oggetto:* Re: Memory Leak or my Error?
>
>
>
> Hi Devis,
>
>
>
> Is it correct that your application create lots of threads? Currently we
> cache connections on per-thread level for performance reasons. If there are
> many threads created over and over again, it could consume more and more
> memory. Probably we should improve this piece of code.
>
>
>
> On Tue, Jun 6, 2017 at 4:38 PM, Devis Balsemin <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=13425&i=0>> wrote:
>
> Hi Vladimir,
>
>
>
> this is my cache configuration
>
>
>
>
>CacheConfiguration cacheCfg = *new *CacheConfiguration<>();
>cacheCfg.setTypes(*keyType*, *valueType*);
>
>cacheCfg.setBackups(0);
>cacheCfg.setName(ctx.name());
>cacheCfg.setCacheMode(*PARTITIONED*);
>cacheCfg.setEagerTtl(*false*);
>cacheCfg.setCopyOnRead(*true*);
>cacheCfg.setExpiryPolicyFactory(CreatedExpiryPolicy.*factoryOf*(*new *
> Duration(TimeUnit.*HOURS*, 24)));
>cacheCfg.setIndexedTypes(*keyType*, *valueType*);
>*ignite*.configuration().setCacheConfiguration(cacheCfg);
>*cache *= *ignite*.getOrCreateCache(cacheCfg);
>
>
>
>
>
> [image: cid:image001.png@01D12220.7E4C4300]
>
&g

Re: Is it necessary to configure setting HBase which use Secondary File System?

[Vladimir Ozerov]

Hi Takashi,

"igfs://" prefix should be used in your application code, in those places
where data is accessed. It is illegal to change "hbase.wal.dir" property,
as it breaks HBase internals.

On Thu, Jun 8, 2017 at 6:30 AM, Takashi Sasaki  wrote:

> Hello,
>
> I used igfs:// instead of hdfs:// for hbase.wal.dir property, then
> HBase Master Server throwed Exception.
>
> 2017-06-08 02:51:56,745 ERROR [main] master.HMasterCommandLine: Master
> exiting
> java.lang.RuntimeException: Failed construction of Master: class
> org.apache.hadoop.hbase.master.HMaster.
> at org.apache.hadoop.hbase.master.HMaster.
> constructMaster(HMaster.java:2577)
> at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(
> HMasterCommandLine.java:231)
> at org.apache.hadoop.hbase.master.HMasterCommandLine.run(
> HMasterCommandLine.java:137)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(
> ServerCommandLine.java:126)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2587)
> Caused by: java.io.IOException: File system is already initialized:
> org.apache.ignite.internal.processors.hadoop.impl.igfs.
> HadoopIgfsWrapper@1dbd580
> at org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem.
> initialize(IgniteHadoopFileSystem.java:215)
> at org.apache.hadoop.hbase.fs.HFileSystem.(
> HFileSystem.java:87)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.
> initializeFileSystem(HRegionServer.java:634)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.<
> init>(HRegionServer.java:576)
> at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:397)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>
> I checked Ignite source code.
> It seems to be called initialize method more than once, so the server
> throw the exception.
>
> I added properties to core-site.xml:
> 
>   fs.igfs.impl
>   org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem
> 
> 
>   fs.AbstractFileSystem.igfs.impl
>   org.apache.ignite.hadoop.fs.v2.IgniteHadoopFileSystem
> 
>
> I changed property hbase-site.xml:
> 
>   hbase.wal.dir
>   igfs://igfs@/user/hbase/WAL
> 
>
> Hadoop version: 2.7.3
> HBase version: 1.3.0
> Ignite version: 2.0.0
>
> How can I solve this problem?
> Give me advice if you like.
>
> Thanks,
> Takashi
>
> 2017-06-07 21:38 GMT+09:00 Takashi Sasaki :
> > Hello,
> >
> > I'm newbie of Ignite, so have some question.
> >
> > When I use Secondary File System to write HBase WAL, should I use
> > igfs:// instead of hdfs:// ?
> >
> > hbase-site.xml(default) is hdfs://.
> >
> > --
> > 
> >   hbase.wal.dir
> >   hdfs://[dnsname]:[port]/user/hbase/WAL
> > 
> > --
> >
> > Does the secondary file system require some configuration changes to
> Hbase?
> >
> > Please give me advice.
> >
> > Thanks,
> > Takashi
>

Re: igfs-meta behavior when node restarts

[Vladimir Ozerov]

Hi,

Is it possible to create isolated reproducer which will show gradual
increase of failed operations over time?

On Wed, Jun 7, 2017 at 6:39 AM, joewang  wrote:

> Would it be helpful if I uploaded logs collected from the cluster? I can
> point to the time when the behavior begins to occur.
>
> My concern with this is that the recurrent transactions (removing entries
> from the TRASH) and rollbacks are being performed over and over again for
> the same set of entries -- over time, this set continues to grow. In my
> observation, over the course of 3-4 days, the initial 4 TX op/s grows to
> ~200 TX op/s across all my nodes. I'm assuming if this grows unbounded, the
> cluster performance and stability will eventually be affected.
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/igfs-meta-behavior-when-node-restarts-
> tp13155p13446.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Failed to wait for initial partition map exchange

[Vladimir]

Unfortunately, it’s not a warning. New node cannot join the cluster. And
there is no heavy load on the cluster, no CPU/memory consumption. No any
network problem because I tested all the nodes on single machine.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Failed-to-wait-for-initial-partition-map-exchange-tp6252p13458.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: BinaryInvalidTypeException: Unknown pair [platformId=.., typeId=...] when using CacheJdbcPojoStoreFactory and read/write-through

[Vladimir]

Thanks. A simple BinaryConfiguration has helped in the case of one isolated
node or more nodes with the same configuration. The problem is that the
storable cache is configured with the node filter. So, only several nodes
share the cache. Another nodes aren't supposed to know anything about these
manually configured types in the BinaryConfiguration and haven't such
config. That causes:

IgniteSpiException: Local node's binary configuration is not equal to remote
node's binary configuration

Can I anyhow bind BinaryConfiguration to certain cache with node filter, not
to entire cluster?




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/BinaryInvalidTypeException-Unknown-pair-platformId-typeId-when-using-CacheJdbcPojoStoreFactory-and-rh-tp13384p13431.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Memory Leak or my Error?

[Vladimir Ozerov]

Hi Devis,

Is it correct that your application create lots of threads? Currently we
cache connections on per-thread level for performance reasons. If there are
many threads created over and over again, it could consume more and more
memory. Probably we should improve this piece of code.

On Tue, Jun 6, 2017 at 4:38 PM, Devis Balsemin <
devis.balse...@flexvalley.com> wrote:

> Hi Vladimir,
>
>
>
> this is my cache configuration
>
>
>
>
>CacheConfiguration cacheCfg = *new *CacheConfiguration<>();
>cacheCfg.setTypes(*keyType*, *valueType*);
>
>cacheCfg.setBackups(0);
>cacheCfg.setName(ctx.name());
>cacheCfg.setCacheMode(*PARTITIONED*);
>cacheCfg.setEagerTtl(*false*);
>cacheCfg.setCopyOnRead(*true*);
>cacheCfg.setExpiryPolicyFactory(CreatedExpiryPolicy.*factoryOf*(*new *
> Duration(TimeUnit.*HOURS*, 24)));
>cacheCfg.setIndexedTypes(*keyType*, *valueType*);
>*ignite*.configuration().setCacheConfiguration(cacheCfg);
>*cache *= *ignite*.getOrCreateCache(cacheCfg);
>
>
>
>
>
> [image: cid:image001.png@01D12220.7E4C4300]
>
>
>
>
>
> Devis Balsemin,  FlexValley SRL (Italy) ,
> Via Dante Alighieri 4, I-36075 Montecchio Maggiore, Italy | t: +39.0444.699622
> | f: +39.0444.1830543
>
>
>
> *Please consider your environmental responsibility before printing this
> e-mail*
>
> This communication (including any attachments) may contain privileged or
> confidential information of FlexValley SRL and is intended for a specific
> individual.  If you are not the intended recipient, you should delete this
> communication, including any attachments without reading or saving them in
> any manner, and you are hereby notified that any disclosure, copying, or
> distribution of this communication, or the taking of any action based on
> it, is strictly prohibited.
>
>
>
> *Da:* Vladimir Ozerov [mailto:voze...@gridgain.com]
> *Inviato:* martedì 6 giugno 2017 15:02
> *A:* user@ignite.apache.org
> *Oggetto:* Re: Memory Leak or my Error?
>
>
>
> Hi Devis,
>
>
>
> Can you show GC roots of these Session objects?
>
>
>
> On Tue, Jun 6, 2017 at 3:52 PM, Devis Balsemin <
> devis.balse...@flexvalley.com> wrote:
>
> Hi,
>
> I’m using Ignite 1.7.
>
> I have a function nativeSQL that is called in many point of my programs.
>
> But after 2/3days I receive my OOM  (this function is called every 0.25ms
> X 10 concurrent users).
>
>
>
> My dubmp show me
>
>
>
> Class Name |
> Objects | Shallow Heap | Retained Heap
>
> 
> 
>
> org.h2.engine.Session  |
> 201.406 |   51.559.936 | >= 95.063.648
>
> char[] |
> 501.160 |   40.283.480 | >= 40.283.480
>
> java.lang.Object[] |
> 625.053 |   21.469.296 | >= 34.984.640
>
> org.h2.jdbc.JdbcConnection |
> 201.404 |   17.723.552 | >= 43.503.144
>
> java.util.ArrayList|
> 616.718 |   14.801.232 | >= 38.616.296
>
> java.util.HashMap$Node |
> 470.953 |   11.302.872 | >= 16.857.176
>
> org.h2.engine.UndoLog  |
> 201.406 |9.667.488 | >= 32.224.960
>
> org.apache.ignite.internal.util.GridCircularBuffer$Item|
> 343.040 |8.232.960 |  >= 8.232.968
>
> java.lang.String   |
> 500.219 |8.003.504 | >= 47.308.152
>
> org.h2.util.CloseWatcher   |
> 201.404 |6.444.928 | >= 13.375.872
>
> java.util.HashMap$Node[]
> |   9.546 |5.074.192 | >= 19.757.480
>
> org.h2.message.Trace   |
> 201.411 |4.833.864 | >= 16.033.024
>
> byte[]
> |   5.104 |1.698.496 |  >= 1.698.496
>
> org.jsr166.ConcurrentHashMap8$Node[]
>|  85 |1.531.792 |  >= 2.455.608
>
> org.apache.ignite.internal.util.GridCircularBuffer$Item[]
> |   4.216 |1.439.616 |  >= 9.672.576
>
> org.apache.ignite.internal.processors.jobmetrics.GridJobMetricsSnapshot|
> 16.384 |1.310.720 |  >= 1.310.720
>
> org.jsr166.ConcurrentLinkedHashMap$HashEntry[]
> |   1.308 |1.225.408 |  >= 1.225.472
>
>
>
>
>
>
>

Re: Failed to wait for initial partition map exchange

[Vladimir]

I have met similar problem too. A node cannot start:

WARN  16:08:36.158 [main]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager:
Still waiting for initial partition map exchange
[fut=GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=fa2d0f11-80cf-4891-9533-7a36392157f6, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=8, intOrder=6,
lastExchangeTime=1496754516085, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], topVer=8, nodeId8=fa2d0f11, msg=null, type=NODE_JOINED,
tstamp=1496754274106], crd=TcpDiscoveryNode
[id=7751c8dc-9bf6-4a21-9ea5-376f2be83913, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1496754273954, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], nodeId=fa2d0f11,
evt=NODE_JOINED], added=false, initFut=GridFutureAdapter
[ignoreInterrupts=false, state=DONE, res=true, hash=1259192920], init=true,
lastVer=null, partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1,
lsnrCalls=4, done=true, cancelled=false, err=null, futs=[true, true, true,
true]], affChangeMsg=null, skipPreload=false, clientOnlyExchange=false,
initTs=1496754276137, centralizedAff=false, changeGlobalStateE=null,
exchangeOnChangeGlobalState=false, forcedRebFut=null, evtLatch=0,
remaining=[bf7a4e48-4867-437b-9cc5-dd598b8795f3,
7751c8dc-9bf6-4a21-9ea5-376f2be83913, 373fd690-4171-4503-ab1b-d622c3ec6fc7],
srvNodes=[TcpDiscoveryNode [id=7751c8dc-9bf6-4a21-9ea5-376f2be83913,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1496754273954, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=bf7a4e48-4867-437b-9cc5-dd598b8795f3, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9461], discPort=9461, order=6, intOrder=4,
lastExchangeTime=1496754273954, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], TcpDiscoveryNode [id=373fd690-4171-4503-ab1b-d622c3ec6fc7,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9462], discPort=9462, order=7,
intOrder=5, lastExchangeTime=1496754273954, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=fa2d0f11-80cf-4891-9533-7a36392157f6, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=8, intOrder=6,
lastExchangeTime=1496754516085, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false]], super=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=1079801970]]]
WARN  16:08:36.499 [exchange-worker-#37%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager:
Failed to wait for partition map exchange [topVer=AffinityTopologyVersion
[topVer=8, minorTopVer=0], node=fa2d0f11-80cf-4891-9533-7a36392157f6].
Dumping pending objects that might be the cause: 

What can it be?



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Failed-to-wait-for-initial-partition-map-exchange-tp6252p13414.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Memory Leak or my Error?

[Vladimir Ozerov]

Hi Devis,

Can you show GC roots of these Session objects?

On Tue, Jun 6, 2017 at 3:52 PM, Devis Balsemin <
devis.balse...@flexvalley.com> wrote:

> Hi,
>
> I’m using Ignite 1.7.
>
> I have a function nativeSQL that is called in many point of my programs.
>
> But after 2/3days I receive my OOM  (this function is called every 0.25ms
> X 10 concurrent users).
>
>
>
> My dubmp show me
>
>
>
> Class Name |
> Objects | Shallow Heap | Retained Heap
>
> 
> 
>
> org.h2.engine.Session  |
> 201.406 |   51.559.936 | >= 95.063.648
>
> char[] |
> 501.160 |   40.283.480 | >= 40.283.480
>
> java.lang.Object[] |
> 625.053 |   21.469.296 | >= 34.984.640
>
> org.h2.jdbc.JdbcConnection |
> 201.404 |   17.723.552 | >= 43.503.144
>
> java.util.ArrayList|
> 616.718 |   14.801.232 | >= 38.616.296
>
> java.util.HashMap$Node |
> 470.953 |   11.302.872 | >= 16.857.176
>
> org.h2.engine.UndoLog  |
> 201.406 |9.667.488 | >= 32.224.960
>
> org.apache.ignite.internal.util.GridCircularBuffer$Item|
> 343.040 |8.232.960 |  >= 8.232.968
>
> java.lang.String   |
> 500.219 |8.003.504 | >= 47.308.152
>
> org.h2.util.CloseWatcher   |
> 201.404 |6.444.928 | >= 13.375.872
>
> java.util.HashMap$Node[]
> |   9.546 |5.074.192 | >= 19.757.480
>
> org.h2.message.Trace   |
> 201.411 |4.833.864 | >= 16.033.024
>
> byte[]
> |   5.104 |1.698.496 |  >= 1.698.496
>
> org.jsr166.ConcurrentHashMap8$Node[]
>|  85 |1.531.792 |  >= 2.455.608
>
> org.apache.ignite.internal.util.GridCircularBuffer$Item[]
> |   4.216 |1.439.616 |  >= 9.672.576
>
> org.apache.ignite.internal.processors.jobmetrics.GridJobMetricsSnapshot|
> 16.384 |1.310.720 |  >= 1.310.720
>
> org.jsr166.ConcurrentLinkedHashMap$HashEntry[]
> |   1.308 |1.225.408 |  >= 1.225.472
>
>
>
>
>
>
>
> Class Name   | Objects | Shallow Heap
>
> --
>
> org.h2.engine.Session| 201.406 |   51.559.936
>
> |- org.h2.jdbc.JdbcConnection| 201.404 |   17.723.552
>
> |- org.h2.util.CloseWatcher  | 201.404 |6.444.928
>
> |- java.util.HashMap$Node| 201.404 |4.833.696
>
> |- org.h2.command.dml.Select |  65 |   11.440
>
> |- org.h2.table.TableFilter  |  65 |6.760
>
> |- org.h2.jdbc.JdbcPreparedStatement |  65 |5.720
>
> |- org.h2.index.IndexCursor  |  64 |4.608
>
> |- org.h2.command.CommandContainer   |  65 |2.600
>
> |- org.h2.engine.Database|   1 |  368
>
> |- java.lang.Thread  |   1 |  120
>
> |- org.jsr166.ConcurrentHashMap8$Node|   4 |   96
>
> |- org.h2.result.LocalResult |   1 |   72
>
> '- Total: 12 entries | |
>
>
>
> Class Name
> | Objects | Shallow Heap
>
> 
> ---
>
> org.h2.jdbc.JdbcConnection
> | 201.404 |   17.723.552
>
> |- org.h2.util.CloseWatcher
> | 201.404 |6.444.928
>
> |- java.lang.Object[]
> |   1 |  960.400
>
> |- org.h2.jdbc.JdbcPreparedStatement
>  |  65 |5.720
>
> |- 
> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$ConnectionWrapper
> |  14 |  224
>
> |- java.lang.Thread
> |   1 |  120
>
> |- org.apache.ignite.internal.processors.query.h2.twostep.
> GridReduceQueryExecutor$QueryRun|   1 |   32
>
>
>
>
>
>
>
>
>
> Class Name |
> Objects | Shallow Heap
>
> 
> 
>
> java.util.ArrayList|
> 616.718 |   14.801.232
>
> |- org.h2.engine.Session   |
> 201.406 |   51.559.936
>
> |- org.h2.engine.UndoLog   |
> 201.406 |9.667.488
>
> |- org.apache.felix.framework.capabilityset.SimpleFilter   |
> 5.798 |  139.152
>
> |- org.apache.felix.framework.wiring.BundleCapabilityImpl  |
> 1.777 |   71.080
>
> |- java.lang.Object[]   

Re: BinaryInvalidTypeException: Unknown pair [platformId=.., typeId=...] when using CacheJdbcPojoStoreFactory and read/write-through

[Vladimir]

>>> Try to add key\value classes in BinaryConfiguration.

Btw any example?



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/BinaryInvalidTypeException-Unknown-pair-platformId-typeId-when-using-CacheJdbcPojoStoreFactory-and-rh-tp13384p13402.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: igfs-meta behavior when node restarts

[Vladimir Ozerov]

Hi,

Ignite performs delete in a "soft" fashion:
1) When "remove" command is executed, we propagate it to the secondary file
system;
2) For IGFS meta cache, we do not remove all records immediately, but
rather execute a single "move" operation and move removed tree to a hidden
"TRASH" folder.
3) "TRASH" folder is cleared periodically - this is what you see in logs.

Removal of trash content should not interfere with normal operations
anyhow, nor should it cause any performance issues. Do you observe some
real slowdown, or you are only concerned with metrics summary?

Vladimir.


On Thu, Jun 1, 2017 at 8:39 PM, joewang  wrote:

> The reads are from a non-IGFS source, but the writes are through IGFS.
> Spark
> uses Hadoop's FileOutputCommitter to write the output to IGFS. I think what
> happens is essentially:
>
> - During processing, temporary files are written by each of n executors
> running on different nodes to some /data/path/output/_temporary/part-n...
> - When the job completes, each of the executor performs the final "commit"
> by renaming the files under /data/path/output/_temporary/part-n... to
> /data/path/output/part-n... and deletes the _temporary directory.
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/igfs-meta-behavior-when-node-restarts-
> tp13155p13322.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: BinaryInvalidTypeException: Unknown pair [platformId=.., typeId=...] when using CacheJdbcPojoStoreFactory and read/write-through

[Vladimir]

Thanks. Looks like so. There is no class data in the file store. A workaround
might putting a value to the cache. That makes it working. But I don't want
to put any junk to my DB :)



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/BinaryInvalidTypeException-Unknown-pair-platformId-typeId-when-using-CacheJdbcPojoStoreFactory-and-rh-tp13384p13389.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

BinaryInvalidTypeException: Unknown pair [platformId=.., typeId=...] when using CacheJdbcPojoStoreFactory and read/write-through

[Vladimir]

Greetings!

Can anyone please help me with cache store feature? Once I add to my cache
config CacheJdbcPojoStoreFactory and enable read/write-through I can't get
any value from the cache because of the deserialization exception:

org.apache.ignite.binary.BinaryInvalidTypeException: Unknown pair
[platformId=0, typeId=-596981488]
at
org.apache.ignite.internal.binary.BinaryContext.descriptorForTypeId(BinaryContext.java:701)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1745)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1704)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.binary.BinaryObjectImpl.deserializeValue(BinaryObjectImpl.java:794)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.binary.BinaryObjectImpl.value(BinaryObjectImpl.java:142)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.processors.cache.CacheObjectContext.unwrapBinary(CacheObjectContext.java:273)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.processors.cache.CacheObjectContext.unwrapBinariesIfNeeded(CacheObjectContext.java:186)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.processors.cache.CacheObjectContext.unwrapBinariesIfNeeded(CacheObjectContext.java:170)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.processors.cache.GridCacheContext.unwrapBinariesIfNeeded(GridCacheContext.java:1719)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.processors.query.GridQueryCacheObjectsIterator.next(GridQueryCacheObjectsIterator.java:64)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.processors.query.GridQueryCacheObjectsIterator.next(GridQueryCacheObjectsIterator.java:29)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$9$1.next(IgniteH2Indexing.java:1542)
~[ignite-indexing-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$9$1.next(IgniteH2Indexing.java:1536)
~[ignite-indexing-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.processors.cache.QueryCursorImpl.getAll(QueryCursorImpl.java:113)
~[ignite-core-2.0.0.jar:2.0.0]

The same happens if I'm trying to get a value by cache.get()

And everything is ok if I do not enable read/write-through and do not set
CacheJdbcPojoStoreFactory in the cache config.

What's wrong?



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/BinaryInvalidTypeException-Unknown-pair-platformId-typeId-when-using-CacheJdbcPojoStoreFactory-and-rh-tp13384.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Apache Ignite - how to get started - simple installation and Hello World

[Zivkovic, Vladimir]

Hi,

My team is trying to evaluate Apache Ignite.
We are first time trying it (started this week) and have issues installing it 
on EC2 (AWS).
Also we will not able to use AMI due to security policies (double-checking / 
reviewing this internally).

Do you have a step by step approach to Download, install Java, MVN, run MVN, 
run script approach for EC2?

We would like to create e.g. 3 node Apache Ignite cluster and test 
storing/retrieving data and other functions.

Can you help post the solution that works please? (we tried several different 
ones but did not work properly – errors on EC2 where there is no errors on Mac 
but could not connect via Python after creating nodes etc).


We are using CentOS (RHEL 7+) as OS for EC2 machine.

Also if you have example with Python (that works on EC2) for connecting and 
working with Ignite (with included libraries that are needed) – that would be 
great.


Thanks, Vladimir
Manager, Data Engineering



The information contained in this e-mail is confidential and/or proprietary to 
Capital One and/or its affiliates and may only be used solely in performance of 
work or services for Capital One. The information transmitted herewith is 
intended only for use by the individual or entity to which it is addressed. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any review, retransmission, dissemination, distribution, copying 
or other use of, or taking of any action in reliance upon this information is 
strictly prohibited. If you have received this communication in error, please 
contact the sender and delete the material from your computer.

Re: How to return ID of Ignite task to stop this task in the future?

[Vladimir Ozerov]

Yakov,

q.1 is answered in separate dev-list topic "IgniteCompute async methods do
not return ComputeTaskFuture".

On Tue, May 30, 2017 at 12:57 PM, Yakov Zhdanov  wrote:

> Couple of questions to Igniters (cross-posting to dev list).
>
> 1. Why IgniteCompute does not return ComputeTaskFuture from all methods?
>
> 2. In GridGain 6.x and earlier there was a method
> GridComputeImpl#cancelTask(GridUuid sesId) - https://github.com/gridgain/
> gridgain/blob/cf56a35cbae442d11181f58004505d6b00fc2f59/modules/core/src/
> main/java/org/gridgain/grid/kernal/GridComputeImpl.java, but I cannot
> find it in Ignite. Who does know anything on this?
>
> Vadim, it seems that you have 2 options now:
> 1. You can get future from IgniteCompute#broadcastAsync() and call
> Future#cancel() - this works if you do not restart client and cancel your
> task from its master node.
> 2. If you want to restart client node and then somehow manage the spawned
> tasks (broadcast() is still a task deep inside Ignite) then first of all
> you need to implement org.apache.ignite.compute.ComputeJobMasterLeaveAware
> on your Runnables so they do not get automatically cancelled on master
> leave. Second, I think you will have to use distributed cache and put some
> job related data to it and periodically poll that cache for changes from
> your jobs and support job's automatic exit if its data changes accordingly.
> Then you can iterate over cache entry set and cancel jobs you want by
> changing their data.
>
> --Yakov
>