Re: Best way to upsetting/delete records in Ignite from Kafka Stream?

[John Smith]

But how does the connector know how to delete records? I see
shallProcessUpdates True/false but there no indictation how to tell it to
remove the record based on a value in the JSON record.

At this point it seems it's easier just to right a consumer and handle it
myself.

On Fri, Mar 22, 2019, 6:58 PM Denis Magda,  wrote:

> Hey John,
>
> Check this integration out. It should support what you are looking for:
> https://docs.gridgain.com/docs/certified-kafka-connector
>
> -
> Denis
>
>
> On Fri, Mar 22, 2019 at 6:17 AM John Smith  wrote:
>
>> Or are you saying I need to write custom streamer?
>>
>> On the streamer how do we get notified of data coming in? The examples
>> don't show that. They only show how to connect... Or is that the only
>> function of streamer?
>>
>> On Fri, Mar 22, 2019, 9:12 AM John Smith,  wrote:
>>
>>> You mean I need to write my own Kafka connect connector using the cache
>>> API and from there decide to do put or remove?
>>>
>>> On Tue, Mar 19, 2019, 8:02 PM aealexsandrov, 
>>> wrote:
>>>
 Hi,

 Yes, looks like the KafkaStreamer doesn't support the DELETE behavior.
 It
 was created to loading data to Ignite.

 However, deleting data from Ignite is possible using Interface
 IgniteDataStreamer API:


 https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteDataStreamer.html#removeData-K-

 So you just require to proceed the Kafka stream and use
 IgniteDataStreamer
 (or IgniteCache API) somewhere in sink function. Possible that you
 should
 take a look at Kafka connector:

 https://kafka.apache.org/documentation.html#quickstart_kafkaconnect

 BR,
 Andrei



 --
 Sent from: http://apache-ignite-users.70518.x6.nabble.com/

>>>

Re: Best way to upsetting/delete records in Ignite from Kafka Stream?

[Denis Magda]

Hey John,

Check this integration out. It should support what you are looking for:
https://docs.gridgain.com/docs/certified-kafka-connector

-
Denis


On Fri, Mar 22, 2019 at 6:17 AM John Smith  wrote:

> Or are you saying I need to write custom streamer?
>
> On the streamer how do we get notified of data coming in? The examples
> don't show that. They only show how to connect... Or is that the only
> function of streamer?
>
> On Fri, Mar 22, 2019, 9:12 AM John Smith,  wrote:
>
>> You mean I need to write my own Kafka connect connector using the cache
>> API and from there decide to do put or remove?
>>
>> On Tue, Mar 19, 2019, 8:02 PM aealexsandrov, 
>> wrote:
>>
>>> Hi,
>>>
>>> Yes, looks like the KafkaStreamer doesn't support the DELETE behavior. It
>>> was created to loading data to Ignite.
>>>
>>> However, deleting data from Ignite is possible using Interface
>>> IgniteDataStreamer API:
>>>
>>>
>>> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteDataStreamer.html#removeData-K-
>>>
>>> So you just require to proceed the Kafka stream and use
>>> IgniteDataStreamer
>>> (or IgniteCache API) somewhere in sink function. Possible that you should
>>> take a look at Kafka connector:
>>>
>>> https://kafka.apache.org/documentation.html#quickstart_kafkaconnect
>>>
>>> BR,
>>> Andrei
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>>

Re: Spark Ignite SaveValues(rdd) not updating cache .

[Harshal Patil]

Can someone please help here ?

On Fri, Mar 22, 2019, 2:06 PM Harshal Patil 
wrote:

> Hi ,
> I am writing spark rdd to Ignie Cache .
>
> If I do ,
>
> *val *df = ic.fromCache("EntitySettingsCache").sql("select * from
> entity_settings”).withColumn("partitionkey",*col*("partitionkey")/2)
>
>
> ic.fromCache("EntitySettingsCache").saveValues(df.*rdd*)
>
>
> *Not able to reflect changes in cache* ,
>
>
> but i do ,
>
>
> val rddPair: RDD[EntitySettingsKey , EntitySettings] = converRddToPair(rdd)
>
>
> ic.fromCache("EntitySettingsCache").savePairs(df.*rdd , true*)
>
>
> above code is relfecting changes in Cache .
>
>
> If I see implementation of  *saveValues* and *savePairs* ,
>
> there is one difference ,
>
>
> *streamer.allowOverwrite(overwrite)*
>
>
> above line is missing in saveValues , can anyone please help me , how do I
> solve the problem , because I dont' want to convert *rdd* to *rddPair* 
> manually
> everytime .
>
>
>
>
>
>
>
>
>
>
>

Re: Finding collocated data in ignite nodes

[Ilya Kasnacheev]

Hello!

I can't see your data so I'm not entirely sure, but:

The first query has expected number of records (500 + 300 + 200, every city
is collocated to its country), hence collocated.
The second query does not have expected number of records (expected (200 +
300) + (500 + 300) + (500 + 200) = 2000 but found just 800), displaying how
uncollocated query fails to join tables in full.

I think you're good now.

Note that your data layout is not very good now. Your cache has 1024
partitions but you're only using 3 at maximum. Our planet does not actually
have enough countries to make such collocation optimal.

Optimal collocation assumes that all your affinity keys are equally
frequent, and total number of affinity keys >> number of partitions.

Maybe you should make Country table REPLICATED instead, and have no
collocation by country_id?

Regards,
-- 
Ilya Kasnacheev


пт, 22 мар. 2019 г. в 19:12, NileshKhaire :

> Hello,
>
> Sorry for late response.
>
> If you will see first message of this third, you will understand that
> initially I had used same schema. But collocation were not working.
>
> By the way, now I have again inserted sample data (sent in previous
> messages) as per your suggestion .
>
> 1000 cities and 3 countries.
>
> 1st Country = 500 cities
> 2nd Country = 300 cities
> 3rd Country = 200 cities
>
> CREATE TABLE Country (
> country_id INT(10),
> country_name CHAR(30),
> Continent  CHAR(30),
> PRIMARY KEY (country_id)
> ) WITH "template=partitioned, backups=1";
>
>
> CREATE TABLE City (
> city_id INT(15),
> country_id INT(10),
> city_name CHAR(50),
> Dist CHAR(20),
> PRIMARY KEY (city_id,country_id)
> ) WITH "template=partitioned, backups=1, affinityKey=country_id";
>
> output :
>
> select count(*) from City c join Country cc on cc.country_id =
> c.country_id
> ;
> => Count = 1000
>
> select count(*) from City c join Country cc on cc.country_id !=
> c.country_id;
> => Count = 800
>
> Do you think data is collocated ?
>
> So what are the logical explanation for count=1000 for first query and
> count
> = 800 for second query.
>
>
> Thanks,
> Nilesh
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Finding collocated data in ignite nodes

[NileshKhaire]

Hello,

Sorry for late response. 

If you will see first message of this third, you will understand that
initially I had used same schema. But collocation were not working. 

By the way, now I have again inserted sample data (sent in previous
messages) as per your suggestion .

1000 cities and 3 countries. 

1st Country = 500 cities
2nd Country = 300 cities
3rd Country = 200 cities

CREATE TABLE Country ( 
country_id INT(10), 
country_name CHAR(30), 
Continent  CHAR(30),
PRIMARY KEY (country_id)
) WITH "template=partitioned, backups=1"; 


CREATE TABLE City ( 
city_id INT(15),
country_id INT(10), 
city_name CHAR(50), 
Dist CHAR(20), 
PRIMARY KEY (city_id,country_id) 
) WITH "template=partitioned, backups=1, affinityKey=country_id"; 

output :

select count(*) from City c join Country cc on cc.country_id = c.country_id 
;
=> Count = 1000

select count(*) from City c join Country cc on cc.country_id !=
c.country_id;
=> Count = 800 

Do you think data is collocated ? 

So what are the logical explanation for count=1000 for first query and count
= 800 for second query.


Thanks,
Nilesh





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite node is down due to full RAM usage

[Ilya Kasnacheev]

Hello!

Unfortunately I would not expect anyone to be debugging your 1.8 cluster
since most people upgraded to 2.x.

Next time this happens, can you capture heap dump from problematic node?
Dominator graph & per-class histogram may help tremendously.

Regards,
-- 
Ilya Kasnacheev


пт, 22 мар. 2019 г. в 15:10, praveeng :

> Hi,
>
> Ignite version : 1.8
> One of the ignite node in 3node cluster is down due to full usage of RAM.
>
> At that point of time i can observe the following logs on this node:
>
> [00:32:02,119][INFO
>
> ][grid-timeout-worker-#7%CasinoApacheIgniteServices%][IgniteKernal%CasinoApacheIgniteServices]
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
> ^-- Node [id=9f8df386, name=CasinoApacheIgniteServices,
> uptime=23:21:45:744]
> ^-- H/N/C [hosts=8, nodes=8, CPUs=44]
> ^-- CPU [cur=8.33%, avg=1.6%, GC=0%]
> ^-- Heap [used=3886MB, free=36.65%, comm=6134MB]
> ^-- Non heap [used=78MB, free=85.96%, comm=529MB]
> ^-- Public thread pool [active=0, idle=0, qSize=0]
> ^-- System thread pool [active=0, idle=16, qSize=0]
> ^-- Outbound messages queue [size=0]
>
> [00:33:24,674][WARN
>
> ][exchange-worker-#23%CasinoApacheIgniteServices%][GridCachePartitionExchangeManager]
> Failed to wait for partition map exchange [topVer=AffinityTopologyVersion
> [topVer=84, minorTopVer=0], node=9f8df386-2886-451f-b1ff-53713878d432].
> Dumping pending objects that might be the cause:
> [00:33:24,674][WARN
>
> ][exchange-worker-#23%CasinoApacheIgniteServices%][GridCachePartitionExchangeManager]
> Failed to wait for partition map exchange [topVer=AffinityTopologyVersion
> [topVer=84, minorTopVer=0], node=9f8df386-2886-451f-b1ff-53713878d432].
> Dumping pending objects that might be the cause:
>
>
> SAR stats for memory usage on this date:
>
> -- mar 6
> 12:00:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit
> %commit  kbactive   kbinact   kbdirty
> 12:10:01 PM170120  16090232 98.95 0   3393384   8222696
> 45.02   9887268   208850460
> 01:50:01 PM168176  16092176 98.97 0   2120848   8224724
> 45.03  10804712   159679248
> 03:10:01 PM199128  16061224 98.78 0991832   8224904
> 45.04  11384652   1241284   436
> 04:10:01 PM153060  16107292 99.06 0229984   8224880
> 45.04  11255628   1627600   208
> 04:20:01 PM165580  16094772 98.98 0 78572   8224828
> 45.03  11338592   156094452
> 04:30:01 PM153508  16106844 99.06 0 29740   8224872
> 45.03  11436544   157946844
> 04:40:01 PM162184  16098168 99.00 0 33152   8224892
> 45.04  11606584   158038824
> 11:10:01 PM370956  15889396 97.72 0 74816   8225312
> 45.04  11927676   161082836
> 11:20:01 PM348576  15911776 97.86 0 69012   8225272
> 45.04  11929820   160274848
> 11:30:01 PM359132  15901220 97.79 0 27060   8225308
> 45.04  11912656   157784836
> 11:40:01 PM340252  15920100 97.91 0 24908   8225272
> 45.04  11910516   157766832
> 11:50:01 PM308340  15952012 98.10 0 39208   8242284
> 45.13  11914564   158920848
> Average:   253568  16006784 98.44 0   2317289   8226063
> 45.04  10368276   1955525   142
>
> Please find the attached file for the cache configuration.
>
>   ignite-clb-cache-config_dev.xml
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t1753/ignite-clb-cache-config_dev.xml>
>
>
> Please find the memory snapshot which is captured by app dynamics tool in
> the attachment.
> memorySnapshot.JPG
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t1753/memorySnapshot.JPG>
>
>
> Following is my analysis.
> When the data is evicting from on heap to off heap, there is not much space
> in off heap.
> Due to that off heap memory usage is full and application has become slow
> and unresponsive.
>
> Even the data in off heap is not expired because of that there is not much
> free memory in RAM.
> After i restarted the application on this node, the RAM usage has become to
> 25% and now it's usage is 45%.
>
> can you please check and suggest once.
>
> Thanks,
> Praveen
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Best way to upsetting/delete records in Ignite from Kafka Stream?

[John Smith]

You mean I need to write my own Kafka connect connector using the cache API
and from there decide to do put or remove?

On Tue, Mar 19, 2019, 8:02 PM aealexsandrov, 
wrote:

> Hi,
>
> Yes, looks like the KafkaStreamer doesn't support the DELETE behavior. It
> was created to loading data to Ignite.
>
> However, deleting data from Ignite is possible using Interface
> IgniteDataStreamer API:
>
>
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteDataStreamer.html#removeData-K-
>
> So you just require to proceed the Kafka stream and use IgniteDataStreamer
> (or IgniteCache API) somewhere in sink function. Possible that you should
> take a look at Kafka connector:
>
> https://kafka.apache.org/documentation.html#quickstart_kafkaconnect
>
> BR,
> Andrei
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Best way to upsetting/delete records in Ignite from Kafka Stream?

[John Smith]

Or are you saying I need to write custom streamer?

On the streamer how do we get notified of data coming in? The examples
don't show that. They only show how to connect... Or is that the only
function of streamer?

On Fri, Mar 22, 2019, 9:12 AM John Smith,  wrote:

> You mean I need to write my own Kafka connect connector using the cache
> API and from there decide to do put or remove?
>
> On Tue, Mar 19, 2019, 8:02 PM aealexsandrov, 
> wrote:
>
>> Hi,
>>
>> Yes, looks like the KafkaStreamer doesn't support the DELETE behavior. It
>> was created to loading data to Ignite.
>>
>> However, deleting data from Ignite is possible using Interface
>> IgniteDataStreamer API:
>>
>>
>> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteDataStreamer.html#removeData-K-
>>
>> So you just require to proceed the Kafka stream and use IgniteDataStreamer
>> (or IgniteCache API) somewhere in sink function. Possible that you should
>> take a look at Kafka connector:
>>
>> https://kafka.apache.org/documentation.html#quickstart_kafkaconnect
>>
>> BR,
>> Andrei
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Re: CacheConfiguration. org.apache.ignite.IgniteException: Default Ignite instance has already been started

[MDmitry_]

Sorry, I use two 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Ignite node is down due to full RAM usage

[praveeng]

Hi,

Ignite version : 1.8
One of the ignite node in 3node cluster is down due to full usage of RAM.

At that point of time i can observe the following logs on this node:

[00:32:02,119][INFO
][grid-timeout-worker-#7%CasinoApacheIgniteServices%][IgniteKernal%CasinoApacheIgniteServices]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=9f8df386, name=CasinoApacheIgniteServices,
uptime=23:21:45:744]
^-- H/N/C [hosts=8, nodes=8, CPUs=44]
^-- CPU [cur=8.33%, avg=1.6%, GC=0%]
^-- Heap [used=3886MB, free=36.65%, comm=6134MB]
^-- Non heap [used=78MB, free=85.96%, comm=529MB]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=16, qSize=0]
^-- Outbound messages queue [size=0]

[00:33:24,674][WARN
][exchange-worker-#23%CasinoApacheIgniteServices%][GridCachePartitionExchangeManager]
Failed to wait for partition map exchange [topVer=AffinityTopologyVersion
[topVer=84, minorTopVer=0], node=9f8df386-2886-451f-b1ff-53713878d432].
Dumping pending objects that might be the cause:
[00:33:24,674][WARN
][exchange-worker-#23%CasinoApacheIgniteServices%][GridCachePartitionExchangeManager]
Failed to wait for partition map exchange [topVer=AffinityTopologyVersion
[topVer=84, minorTopVer=0], node=9f8df386-2886-451f-b1ff-53713878d432].
Dumping pending objects that might be the cause:


SAR stats for memory usage on this date: 

-- mar 6
12:00:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit  
%commit  kbactive   kbinact   kbdirty
12:10:01 PM170120  16090232 98.95 0   3393384   8222696
45.02   9887268   208850460
01:50:01 PM168176  16092176 98.97 0   2120848   8224724
45.03  10804712   159679248
03:10:01 PM199128  16061224 98.78 0991832   8224904
45.04  11384652   1241284   436
04:10:01 PM153060  16107292 99.06 0229984   8224880
45.04  11255628   1627600   208
04:20:01 PM165580  16094772 98.98 0 78572   8224828
45.03  11338592   156094452
04:30:01 PM153508  16106844 99.06 0 29740   8224872
45.03  11436544   157946844
04:40:01 PM162184  16098168 99.00 0 33152   8224892
45.04  11606584   158038824
11:10:01 PM370956  15889396 97.72 0 74816   8225312
45.04  11927676   161082836
11:20:01 PM348576  15911776 97.86 0 69012   8225272
45.04  11929820   160274848
11:30:01 PM359132  15901220 97.79 0 27060   8225308
45.04  11912656   157784836
11:40:01 PM340252  15920100 97.91 0 24908   8225272
45.04  11910516   157766832
11:50:01 PM308340  15952012 98.10 0 39208   8242284
45.13  11914564   158920848
Average:   253568  16006784 98.44 0   2317289   8226063
45.04  10368276   1955525   142

Please find the attached file for the cache configuration.

  ignite-clb-cache-config_dev.xml

  

Please find the memory snapshot which is captured by app dynamics tool in
the attachment.
memorySnapshot.JPG
  

Following is my analysis.
When the data is evicting from on heap to off heap, there is not much space
in off heap.
Due to that off heap memory usage is full and application has become slow
and unresponsive.

Even the data in off heap is not expired because of that there is not much
free memory in RAM.
After i restarted the application on this node, the RAM usage has become to
25% and now it's usage is 45%.

can you please check and suggest once.

Thanks,
Praveen



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Spark Ignite SaveValues(rdd) not updating cache .

[Harshal Patil]

Hi ,
I am writing spark rdd to Ignie Cache .

If I do ,

*val *df = ic.fromCache("EntitySettingsCache").sql("select * from
entity_settings”).withColumn("partitionkey",*col*("partitionkey")/2)


ic.fromCache("EntitySettingsCache").saveValues(df.*rdd*)


*Not able to reflect changes in cache* ,


but i do ,


val rddPair: RDD[EntitySettingsKey , EntitySettings] = converRddToPair(rdd)


ic.fromCache("EntitySettingsCache").savePairs(df.*rdd , true*)


above code is relfecting changes in Cache .


If I see implementation of  *saveValues* and *savePairs* ,

there is one difference ,


*streamer.allowOverwrite(overwrite)*


above line is missing in saveValues , can anyone please help me , how do I
solve the problem , because I dont' want to convert *rdd* to *rddPair* manually
everytime .

Re: Ignite graceful shutdown

[akaptsan]

Hi 

I would like to continue this conversation

What is the best way to perform a rolling restart of Ignite nodes to
guarantee no data loss?

I think Ignite should be able to stop a node in data-safe fashion: wait
until all data are moved to another nodes and then stop

Can I somehow ask a node to move all data to other nodes?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

CacheConfiguration. org.apache.ignite.IgniteException: Default Ignite instance has already been started

[MDmitry_]

When I use bean org.apache.ignite.configuration.CacheConfiguration, at start
of the server I receive an error "CacheConfiguration.
org.apache.ignite.IgniteException: Default Ignite instance has already been
started"