Re: starting ignite in docker

[vitalys]

There is another option you may explore:

1. Create a custom Ignite docker image using Dockerfile and use COPY command
to copy your Server Configuration XML to $IGNITE_HOME/config and then adding
an entry at the end of the Dockerfile :

ENV CONFIG_URI apache-ignite/config/ignite-server-conf.xml

Then start your Ignite Docker container you usually do and it will use your
custom XML



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

RE: Running Ignite Cluster using Docker containers

[vitalys]

I am getting different issue now. I added my custom server configuration file
to the Docker Image build :

ENV CONFIG_URI apache-ignite/config/ignite-server-config.xml

and I defined my network configuration :

/ 
  
   
  
  

  127.0.0.1:47500..47504 
   

  
  
/

When I start my Docker container :
/
 docker run --rm --name myignite1 -p 47500:47500 -p 47501:47501 -p
10800:10800 -it myignite/ignite:1.0/

I am getting this message :

Failed to connect to any address from IP finder (make sure IP finder
addresses are correct and firewalls are disabled on all host machines):
[/127.0.0.1:47500, /127.0.0.1:47501, /127.0.0.1:47502, /127.0.0.1:47503,
/127.0.0.1:47504]

However if I switch to Multicast configuration it seems like I am starting
correctly. 

In both cases my client fails to connect to the Cluster.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

RE: IgniteCache.lock behaviour on a key that doesn't exist in the cache (yet)

[Yohan Fernando]

Yes I did think of that approach but wanted to avoid the risk of the expensive 
load from the DB being called more than once unnecessarily. I suppose that 
should be balanced the cost of the IgniteCache.lock() call.

Thanks for the reply Andei

From: Andrei Aleksandrov 
Sent: 07 August 2019 13:02
To: user@ignite.apache.org
Subject: Re: IgniteCache.lock behaviour on a key that doesn't exist in the 
cache (yet)

Caution: This email originated from outside of Tudor.



Hi,

I tested your code and looks like it worked fine.

However, you also can try to use next method:

https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteCache.html#getAndPutIfAbsent-K-V-

In this case you can:

1)Check cache.containsKey(key)
2)If true then prepare the default value
3)Run getAndPutIfAbsent(key, defaultValue)

If another client was able to put the value during current prepare the default 
then you will get that value, otherwise default value will be returned.

BR,
Andrei
8/7/2019 12:08 PM, Yohan Fernando пишет:
I'm trying to write a transaction-safe way of performing a lazy load of an 
object from the database if it doesnt exist. However as IgniteCache doesn't 
have the equivalent of HashMap.computeIfAbsent, I'm trying to use the 
IgniteCache.lock method to achieve this.

The question is whether IgniteCache.lock will lock the cache for that key even 
if the key does not yet exist in the cache.

Following is the example code,

public GridOrder getOrder(OrderKey key) {
Lock orderKeyLock = cache.lock(key);
try {
orderKeyLock.lock();
if (!cache.containsKey(key)) {
GridOrder order = loadOrderFromDb(key);
if ( order == null) {
throw new IllegalStateException("Key " + key + " not in 
Order Cache or in Database DB Name ");
}
cache.put(key,order);
}
return cache.get(key);
} finally {
orderKeyLock.unlock();
}
}

Alternatively, is there a better way to achieve this?

_

This email, its contents, and any attachments transmitted with it are intended 
only for the addressee(s) and may be confidential and legally privileged. We do 
not waive any confidentiality by misdelivery. If you have received this email 
in error, please notify the sender immediately and delete it. You should not 
copy it, forward it or otherwise use the contents, attachments or information 
in any way. Any liability for viruses is excluded to the fullest extent 
permitted by law.

Tudor Capital Europe LLP (TCE) is authorised and regulated by The Financial 
Conduct Authority (the FCA). TCE is registered as a limited liability 
partnership in England and Wales No: OC340673 with its registered office at 10 
New Burlington Street, London, W1S 3BE, United Kingdom

_

This email, its contents, and any attachments transmitted with it are intended 
only for the addressee(s) and may be confidential and legally privileged. We do 
not waive any confidentiality by misdelivery. If you have received this email 
in error, please notify the sender immediately and delete it. You should not 
copy it, forward it or otherwise use the contents, attachments or information 
in any way. Any liability for viruses is excluded to the fullest extent 
permitted by law.

Tudor Capital Europe LLP (TCE) is authorised and regulated by The Financial 
Conduct Authority (the FCA). TCE is registered as a limited liability 
partnership in England and Wales No: OC340673 with its registered office at 10 
New Burlington Street, London, W1S 3BE, United Kingdom

Re: Regarding Ignite memory going beyond specified memory

[Ilya Kasnacheev]

Hello!

Unfortunately it's hard to reason what happens here. Does it also happen if
you limit heap to 2G? Do you have more than one node on the same machine?

Regards,
-- 
Ilya Kasnacheev


пт, 26 июл. 2019 г. в 22:00, nikhil dhiman :

> Thank Igor,
> But the problem remains the same. Just to give more clarity.
> I am doing load testing on 2 cache keys, with almost 500 Hits per
> second which includes get, put and remove.
> So when I start with 1 server and my clients, it works fine, but when
> I try to add more nodes to my cluster, my initial node fails.
>
> On Fri, Jul 26, 2019 at 5:29 PM Igor Belyakov 
> wrote:
> >
> > Hi Nikhil,
> >
> > You've specified 4gb for heap size and 4gb for data regions in off heap
> memory, it's 8gb in total. Since operation system also consumes some amount
> of memory, 8gb of RAM isn't enough for your case. You should try to reduce
> heap size/data regions size or increase amount of RAM in your system.
> >
> > Regards,
> > Igor
> >
> > On Fri, Jul 26, 2019 at 1:20 PM nikhil dhiman 
> wrote:
> >>
> >> I am running java application having Ignite 2.7.5 as dependency
> >> java -Xms4g -Xmx4g -XX:+AlwaysPreTouch -XX:+UseG1GC
> -XX:+ScavengeBeforeFullGC  -XX:+DisableExplicitGC -server
> -XX:MaxMetaspaceSize=256m -XX:+HeapDumpOnOutOfMemoryError
> -XX:+ExitOnOutOfMemoryError -jar
> target/java_simple-1.0-SNAPSHOT-jar-with-dependencies.jar
> >>
> >> This application does nothing except running Ignite server.
> >>
> >> I have specified max memory as 4g and also internal data region memory
> as ~ 4g with eviction policy defined.
> >> DataRegionConfiguration regionCfg = new DataRegionConfiguration();
> >>   regionCfg.setName("chat");
> >>   regionCfg.setMaxSize(2000L * 1024*1024);
> >>   regionCfg.setInitialSize(1100L * 1024 *1024);
> >>
>  regionCfg.setPageEvictionMode(DataPageEvictionMode.RANDOM_LRU);
> >>   regionCfg.setEvictionThreshold(.7);
> >>   regionCfg.setMetricsEnabled(true);
> >>   DataRegionConfiguration regionCfg2 = new
> DataRegionConfiguration();
> >>   regionCfg2.setName("btc");
> >>   regionCfg2.setMaxSize(2000L * 1024*1024);
> >>   regionCfg2.setInitialSize(1100L * 1024 *1024);
> >>
>  regionCfg2.setPageEvictionMode(DataPageEvictionMode.RANDOM_LRU);
> >>   regionCfg2.setEvictionThreshold(.7);
> >>   regionCfg2.setMetricsEnabled(true);
> >>   storageCfg.setDataRegionConfigurations(regionCfg2, regionCfg);
> >>   cfg.setDataStorageConfiguration(storageCfg);
> >>
> >> I am hitting ignite server with 18 clients with average get/put/remove
> operation of 2k hits per second.
> >> System on which ignite is running has around 8G of ram.
> >>
> >> When I run my clients[From different server] that hit the Ignite server
> for few minutes, it consumes all my system RAM[Ignite server] and it stops
> with  OOM error.
> >>
> >> I don't understand why it goes beyond the memory specified in the data
> region.
> >>
> >> Thanks & Regards,
> >> Nikhil Dhiman
>

Re: Ignite 2.7.0 : thick client has all system threads busy indefinitely

[Ilya Kasnacheev]

Hello!

I can see you have large network timeouts. Sometimes it makes sense, but it
may also cause nodes to have half open connections (one side thinks it's OK
but other already disconnected), which would cause errors like this one.

You should also investigate why connection was closed in the first place.

Regards,
-- 
Ilya Kasnacheev


пн, 22 июл. 2019 г. в 20:51, Mahesh Renduchintala <
mahesh.renduchint...@aline-consulting.com>:

> please see client and server configs.
>
>
>
>

Re: CacheEntryProcessor causes cluster node to stop

[Ilya Kasnacheev]

Hello!

I can see that you had server nodes fail on activation and then there are
"the tree may be corrupted" errors. This points to damaged persistence
files on one or more nodes, probably due to prior running out of disk space
or some kind of internal erorr.

Did it resolve in the end?

Regards,
-- 
Ilya Kasnacheev


ср, 17 июл. 2019 г. в 17:09, ihalilaltun :

> Hi Vladimir,
>
> here is logs from other node
> ignite-3.zip
> 
>
>
>
> -
> İbrahim Halil Altun
> Senior Software Engineer @ Segmentify
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: IgniteCache.lock behaviour on a key that doesn't exist in the cache (yet)

[Andrei Aleksandrov]

Hi,

I tested your code and looks like it worked fine.

However, you also can try to use next method:

https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteCache.html#getAndPutIfAbsent-K-V-

In this case you can:

1)Check cache.containsKey(key)
2)If true then prepare the default value
3)Run getAndPutIfAbsent(key, defaultValue)

If another client was able to put the value during current prepare the 
default then you will get that value, otherwise default value will be 
returned.


BR,
Andrei

8/7/2019 12:08 PM, Yohan Fernando пишет:


I'm trying to write a transaction-safe way of performing a lazy load 
of an object from the database if it doesnt exist. However as 
IgniteCache doesn't have the equivalent of HashMap.computeIfAbsent, 
I'm trying to use the IgniteCache.lock method to achieve this.


The question is whether IgniteCache.lock will lock the cache for that 
key even if the key does not yet exist in the cache.


Following is the example code,

public GridOrder getOrder(OrderKey key) {

    Lock orderKeyLock = cache.lock(key);

    try {

    orderKeyLock.lock();

    if (!cache.containsKey(key)) {

    GridOrder order = loadOrderFromDb(key);

    if ( order == null) {

    throw new IllegalStateException("Key " + key + " 
not in Order Cache or in Database DB Name ");


    }

    cache.put(key,order);

    }

    return cache.get(key);

    } finally {

    orderKeyLock.unlock();

    }

    }

Alternatively, is there a better way to achieve this?

_

This email, its contents, and any attachments transmitted with it are 
intended only for the addressee(s) and may be confidential and legally 
privileged. We do not waive any confidentiality by misdelivery. If you 
have received this email in error, please notify the sender 
immediately and delete it. You should not copy it, forward it or 
otherwise use the contents, attachments or information in any way. Any 
liability for viruses is excluded to the fullest extent permitted by law.


Tudor Capital Europe LLP (TCE) is authorised and regulated by The 
Financial Conduct Authority (the FCA). TCE is registered as a limited 
liability partnership in England and Wales No: OC340673 with its 
registered office at 10 New Burlington Street, London, W1S 3BE, United 
Kingdom

Re: Ignite Partitioned Cache / Use for an in-memory transaction-log

[Ilya Kasnacheev]

Hello!

You can have any number of backups, but you can either update all of them
in sync, or primary only. You can't have a subset of backups updated,
despite such possibility being discussed.

Regards,
-- 
Ilya Kasnacheev


ср, 7 авг. 2019 г. в 13:29, Johannes Lichtenberger <
johannes.lichtenber...@unitedplanet.com>:

> Hi,
>
> if I want to replicate data from one node to N-nodes (that is only a
> subset of nodes in the whole cluster), could I also specify that not only
> the primary is updated in synchronous mode, but some of the backup nodes,
> too?
>
> I also would like to somehow guarantee read-your-own-writes semantics. My
> idea is for replicating a storage / database system to have a single writer
> through which all writes are propagated.
>
> I'd probably use something like oldest node is the master (which can be
> done using Apache Ignite).
>
> Then let's say someone decides to create a database and a first resource.
> I'd have to send an event round robin to create the resource on let's say
> three of 10 cluster nodes. Maybe I could use a distributed semaphore which
> just accepts 3-times acquiring the semaphore (name is database + resource
> concatenated). Once the initial bootstrapping of creating directories and
> files happened on all 3 nodes a write-transaction can be opened, which
> writes into a distributed cache. However as I'm thinking about it the cache
> for this resource must be shared on only the same 3 cluster-nodes. That's
> maybe already where I have no clue how to achieve this.
>
> Maybe I'm looking into how other data stores are replicated and
> partitioned, first ;)
>
> There's also BookKeeper for distributing a transaction-log. But I'm not
> sure if it's overhead if I don't have to persist the changes before
> applying it to the real data-files (to provide consistency -- I don't need
> persisted write-ahead logging).
>
> kind regards
>
> Johannes
> On 06.08.19 15:20, Ilya Kasnacheev wrote:
>
> Hello!
>
> You can use ignite.affinity(cacheName).mapKeyToNode(key): it returns
> ClusterNode which is primary for this key.
>
> Not sure that I understand you about the quorum.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> вт, 6 авг. 2019 г. в 15:47, Johannes Lichtenberger <
> johannes.lichtenber...@unitedplanet.com>:
>
>> Hi,
>>
>> can I somehow query on which cluster-nodes a partitioned cache stores
>> values for specific keys? I might want to use a cache for replicating an
>> in-memory transaction log (which does not have to be persisted) to
>> replicate a document of a NoSQL document store to a few nodes.
>>
>> Thus, instead of a local cache I'd simply write into an ignite cache and
>> then would like to query which cluster-nodes have stored a specific key
>> (for instance the document name). So for instance I could tell a load
>> balancer for reads to read documents from one of the backup replicas.
>>
>> During a transaction commit I would also need this information to know
>> where to send an event to... to commit the transaction. And somehow I'd
>> love to wait for a quorum of nodes if the transaction really has
>> committed or it needs to roll back.
>>
>> kind regards
>>
>> Johannes
>>
>>

Re: Ignite Partitioned Cache / Use for an in-memory transaction-log

[Johannes Lichtenberger]

Hi,

if I want to replicate data from one node to N-nodes (that is only a 
subset of nodes in the whole cluster), could I also specify that not 
only the primary is updated in synchronous mode, but some of the backup 
nodes, too?


I also would like to somehow guarantee read-your-own-writes semantics. 
My idea is for replicating a storage / database system to have a single 
writer through which all writes are propagated.


I'd probably use something like oldest node is the master (which can be 
done using Apache Ignite).


Then let's say someone decides to create a database and a first 
resource. I'd have to send an event round robin to create the resource 
on let's say three of 10 cluster nodes. Maybe I could use a distributed 
semaphore which just accepts 3-times acquiring the semaphore (name is 
database + resource concatenated). Once the initial bootstrapping of 
creating directories and files happened on all 3 nodes a 
write-transaction can be opened, which writes into a distributed cache. 
However as I'm thinking about it the cache for this resource must be 
shared on only the same 3 cluster-nodes. That's maybe already where I 
have no clue how to achieve this.


Maybe I'm looking into how other data stores are replicated and 
partitioned, first ;)


There's also BookKeeper for distributing a transaction-log. But I'm not 
sure if it's overhead if I don't have to persist the changes before 
applying it to the real data-files (to provide consistency -- I don't 
need persisted write-ahead logging).


kind regards

Johannes

On 06.08.19 15:20, Ilya Kasnacheev wrote:

Hello!

You can use ignite.affinity(cacheName).mapKeyToNode(key): it returns 
ClusterNode which is primary for this key.


Not sure that I understand you about the quorum.

Regards,
--
Ilya Kasnacheev


вт, 6 авг. 2019 г. в 15:47, Johannes Lichtenberger 
>:


Hi,

can I somehow query on which cluster-nodes a partitioned cache stores
values for specific keys? I might want to use a cache for
replicating an
in-memory transaction log (which does not have to be persisted) to
replicate a document of a NoSQL document store to a few nodes.

Thus, instead of a local cache I'd simply write into an ignite
cache and
then would like to query which cluster-nodes have stored a
specific key
(for instance the document name). So for instance I could tell a load
balancer for reads to read documents from one of the backup replicas.

During a transaction commit I would also need this information to
know
where to send an event to... to commit the transaction. And
somehow I'd
love to wait for a quorum of nodes if the transaction really has
committed or it needs to roll back.

kind regards

Johannes

IgniteCache.lock behaviour on a key that doesn't exist in the cache (yet)

[Yohan Fernando]

I'm trying to write a transaction-safe way of performing a lazy load of an 
object from the database if it doesnt exist. However as IgniteCache doesn't 
have the equivalent of HashMap.computeIfAbsent, I'm trying to use the 
IgniteCache.lock method to achieve this.

The question is whether IgniteCache.lock will lock the cache for that key even 
if the key does not yet exist in the cache.

Following is the example code,

public GridOrder getOrder(OrderKey key) {
Lock orderKeyLock = cache.lock(key);
try {
orderKeyLock.lock();
if (!cache.containsKey(key)) {
GridOrder order = loadOrderFromDb(key);
if ( order == null) {
throw new IllegalStateException("Key " + key + " not in 
Order Cache or in Database DB Name ");
}
cache.put(key,order);
}
return cache.get(key);
} finally {
orderKeyLock.unlock();
}
}

Alternatively, is there a better way to achieve this?

_

This email, its contents, and any attachments transmitted with it are intended 
only for the addressee(s) and may be confidential and legally privileged. We do 
not waive any confidentiality by misdelivery. If you have received this email 
in error, please notify the sender immediately and delete it. You should not 
copy it, forward it or otherwise use the contents, attachments or information 
in any way. Any liability for viruses is excluded to the fullest extent 
permitted by law.

Tudor Capital Europe LLP (TCE) is authorised and regulated by The Financial 
Conduct Authority (the FCA). TCE is registered as a limited liability 
partnership in England and Wales No: OC340673 with its registered office at 10 
New Burlington Street, London, W1S 3BE, United Kingdom