Cluster endpoints [2.10 c++ thin client]

[jjimeno]

Hi all,

After taking a look to the c++ thin client source code
(cache_client_impl.cpp) it seems partition awareness is only taken into
account for single requests (Get/Put), but neither for batch (GetAll/PutAll)
nor transactions.

But in our tests, adding endpoints to Ignite Client configuration makes a
huge difference in terms of performance in batch requests for transactional
caches... could anyone put some light on how it's possible? 

Thanks in advance for your help.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite 2.10. Performance tests in Azure

[jjimeno]

It's strange, as that's not what is stated in the documentation:

https://ignite.apache.org/docs/latest/key-value-api/basic-cache-operations#atomic-operations

  

otherwise, why would we need specific transactional caches?

Regarding collocated computing:  We have to work in C++ and, since Ignite
nodes use Java, I don't see how it is possible to send a C++ piece of code
(like a lambda) so that it gets executed within the node... 




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite 2.10. Performance tests in Azure

[jjimeno]

Hi,

I have the same feeling, but I think that shouldn't be the case.  Small
number of big batches should decrease the total latency time while would
favor the total throughput. And, as Ilya said:

"In a distributed system, throughput will scale with cluster growth, but
latency will be steady or become slightly worse."

the effects of scaling the cluster should be clearer using a few big batches
rather than a lot of tiny ones, at least in my understanding.

Unfortunately, Data Streamer is not yet supported in the C++ API, afaik.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite 2.10. Performance tests in Azure

[jjimeno]

Hello, and thanks for answering so quick

Because, as you say, I should get a bigger throughput when increasing the
number of nodes.  

Of course I can't get the best of Ignite with this configuration, but I
would expect something similar to what I get writing: time decreasing while
nodes increase until the point that the single thread becomes the
bottleneck.

Also, I wouldn't expect having better writing than reading times.

Sorry, I don't have these values, but I'll try to repeat the tests to get
them.

Josemari.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Ignite 2.10. Performance tests in Azure

[jjimeno]

Hello all,

For our project we need a distributed database with transactional support,
and Ignite is one of the options we are testing.

Scalability is one of our must have, so we created an Ignite Kubernetes
cluster in Azure to test it, but we found that the results were not what we
expected.

To discard the problem was in our code or in using transactional caches, we
created a small test program for writing/reading 1.8M keys of 528 bytes each
(it represents one of our data types). 

As you can see in this graph, reading doesn't seem to scale.  Especially for
the transactional cache, where having 4, 8 or 16 nodes in the cluster
performs worse than having only 2:
 

While writing in atomic caches does... until 8 nodes, then it gets steady
(No transactional times because of  this
  ):
 

Another strange thing is that, for atomic caches, reading seems to be slower
than writing:
 

So, my questions are:
  - Could I been doing something wrong that could lead to this results?
  - How could it be possible to get worse reading timings in a 4/8/16 nodes
cluster than in a 2 nodes cluster for a transactional cache?
  - How could reading be slower than writing in atomic caches?

These are the source code and configuration files we're using:
Test.cpp
  
Order.h   
node-configuration.xml

  

Best regards and thanks in advance!




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: [2.10 branch]cpp thin client transaction :Transaction with id 1 not found.

[jjimeno]

Hello,

I've already tested it and it works like a charm...  Thanks!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: [2.10 branch]cpp thin client transaction :Transaction with id 1 not found.

[jjimeno]

Hello!

In fact, it's very simple:

int main()
   {
   IgniteClientConfiguration cfg;

   cfg.SetEndPoints("10.250.0.10, 10.250.0.4");

   try
  {
  IgniteClient client = IgniteClient::Start(cfg);

  CacheClient cache = client.GetOrCreateCache("vds");

  ClientTransactions transactions = client.ClientTransactions();

  ClientTransaction tx = transactions.TxStart(PESSIMISTIC,
READ_COMMITTED);

  cache.Put(1, 1);

  tx.Commit();
  }
   catch (IgniteError & err)
  {
  std::cout << "An error occurred: " << err.GetText() << std::endl;

  return err.GetCode();
  }

   return 0;
   }

Not always, but sometimes, I get an "stack overflow" error, which makes me
think about a concurrence problem in the code.  

Cluster configuration:
 

Error:
 

Just in case, the C++ version I'm currently using is:
685c1b70ca (HEAD -> master, origin/master, origin/HEAD) IGNITE-13865 Support 
DateTime as a key or value in .NET and Java (#8580)

Le me know if you need anything else



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: [2.10 branch]cpp thin client transaction :Transaction with id 1 not found.

[jjimeno]

Hello

No, both belong to the same cluster.

Waiting for the PutAll fix, we want to benchmark multi-node reading
performance. For that we:

- create a single server node cluster
- create a single cache with 1.8M keys and ~1GB in it
- create a single c++ thin node with a single endpoint (the host containing
the server node)
- read all the entries in batches and store timings

To move to a two-nodes cluster, we 

- create a new single node in a different host
- add it to the cluster's baseline 
- check rebalancing is 100% done (we use gridgain control center for that)
- add the new host to the endpoints in the c++ thin client

... and so on.  We got up to 4 nodes.

We have experienced two things:

- the exception with "transaction with id 1 not found" (this always happens
when we have only 2 nodes)
- no response from the node on a single commit transaction (this happens
when we have 4 nodes)

I hope this helps you.  Let me know if you need further information




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: [2.9.1] Out of memory in data region

[jjimeno]

Hello,

That's what I was thinking of... thanks Ilya



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: [2.10 branch]cpp thin client transaction :Transaction with id 1 not found.

[jjimeno]

Hello,

I'm facing the same problem.  Just let you know my configuration in case it
helps you:

- Two hosts in a local network, each having a server 2.9.1 node.  Different
static IP addresses
- A single c++ thin client. 
- If I add the two addresses in the SetEndPoints function, I get the
error
- If only one is set, it works

Regards



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: [2.9.1] Out of memory in data region

[jjimeno]

Hello!

Ok but, according to the error:

[15:01:23,559][SEVERE][client-connector-#134][] JVM will be halted
immediately due to the failure: [failureCtx=FailureContext
[type=CRITICAL_ERROR, err=class o.a.i.i.mem.IgniteOutOfMemoryException: *Out
of memory in data region* [name=default, initSize=1.0 GiB, maxSize=1.0 GiB,
*persistenceEnabled=true*]

the OOM is occurring in data region, and that's off-heap memory, as you
pointed out.  

On the other hand, on-heap memory in the test is 4GB, more than enough for a
1GB transaction...



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: [2.9.1] Out of memory in data region

[jjimeno]

Hello

Thanks for your help.

I know that fixes the problem, but my question was about why I’m getting
that error when persistence is on.

As far as I know, if persistence is on, off-heap memory holds a subset of
data when they don’t fit in memory, and no OOM error should be thrown...
unless there are some requirements as the one I pointed out in my previous
mail I’m not aware of.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

[2.9.1] Out of memory in data region

[jjimeno]

Hello,

I've got this simple configuration file:



   
  



















 




and, at some point during a transaction of ~1GB, I get this error:

[15:01:23,559][SEVERE][client-connector-#134][] JVM will be halted
immediately due to the failure: [failureCtx=FailureContext
[type=CRITICAL_ERROR, err=class o.a.i.i.mem.IgniteOutOfMemoryException: Out
of memory in data region [name=default, initSize=1.0 GiB, maxSize=1.0 GiB,
*persistenceEnabled=true*] Try the following:
  ^-- Increase maximum off-heap memory size
(DataRegionConfiguration.maxSize)
  ^-- *Enable Ignite persistence*
(DataRegionConfiguration.persistenceEnabled)
  ^-- Enable eviction or expiration policies]]

Maybe the off-heap memory must be, at least, as large as the transaction we
want to commit?

Thanks in advance!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Long transaction suspended

[jjimeno]

Hello!

FYI: https://issues.apache.org/jira/browse/IGNITE-14165




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Long transaction suspended

[jjimeno]

Hello!

Answering to myself:

Should I get a better performance in a multi-node cluster? Of course!  

We have switched to a 4-node cluster and reading times dropped drastically. 
We are really looking forward the PutAll bug to be fixed to perform writing
tests.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Long transaction suspended

[jjimeno]

Hi, thanks for pointing it out

This confirms our tests... moving from a single-node cluster to a two-nodes
one dropped the read timings to less than the half!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Re[2]: Long transaction suspended

[jjimeno]

Great!... I'm really looking forward it :)



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Re[2]: Long transaction suspended

[jjimeno]

Hello!

I'm sorry hearing that.
Would you think it could be fixed to reach these 2GB? Currently it's only
1GB in the C++ Thin Client

Regards



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Long transaction suspended

[jjimeno]

Hello!

That's exactly the reason why we would prefer to choose Ignite over RocksDB. 
Otherwise, we will have to implement scalability by ourselves and, believe
me, that's not something we would like to do.

We also know they're not directly comparable. We would agree to pay the
price for scalability with slightly worse performance but, based on our
tests, it's too big.

For instance:
  - Single node cluster in the same host as the application (no
communication over the wire, trying to get closer to an embedded database)
  - A single user (no multiple users working either on the application or
the database)

A transactional commit with 1.8M keys and 1GB in size takes 97 seconds with
NO persistence, and this time is doubled if persistence is enabled.  RocksDB
takes around 100 seconds to perform a transaction with 4M keys and 4GB in
size, persistence included.  As you can see, there is a huge difference.

On the other hand, limitations like the ones we have found in one month of
research:
  -  PutAll performance in transactional cache
  
  -  Not asynchronous tcp connection
  
  - The maximum transaction size of 1GB we are discussing in this thread

don't really help to go for Ignite, at least in our kind of project.

But we would still like to do more tests to be 100% sure about our decision,
that's why I'd like to ask you: 
  - Should I get a better performance in a multi-node cluster?
Read/Write/Both?
  - Should I do the tests in a different way?

Thanks in advance!




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Long transaction suspended

[jjimeno]

Hi,

Because of the kind of product we have to develop, we currently have a set
of scenarios with this kind of transactions and we're evaluating several
datastores as RocksDB and, sadly, timings there are quite better than the
ones I've got in Ignite... :(

Data streamer is not available in C++ afaik...



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Long transaction suspended

[jjimeno]

I wouldn't mind, but I'm afraid I'm not allowed to... at least, I couldn't
find the option on that page :)



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Long transaction suspended

[jjimeno]

Hello Ilya,

Yes, but it has nothing to do with suspending an active transaction... the
problem is that transaction never reaches ACTIVE state because it takes a
long time creating the tcp message.

Please, take a look to my previous post.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Long transaction suspended

[jjimeno]

Hi, 

First off, thanks for your help.

In the test, I'm using a single server node cluster with the official 2.9.1
version.  Client is a C++ Thin Client with transactions support (commit
685c1b70ca from master branch).

The test is very simple: 

  struct Blob
  {
 int8_t m_blob[512];
  };

  IgniteClient client = IgniteClient::Start(cfg);

  CacheClient cache = client.GetOrCreateCache("vds");

  cache.Clear();

  std::map map;

  for (uint32_t i = 0; i < 200; ++i)
 map.insert (std::make_pair(i, Blob()));

  ClientTransactions transactions = client.ClientTransactions();

  ClientTransaction tx = transactions.TxStart(PESSIMISTIC,
READ_COMMITTED);

  cache.PutAll(map);

  tx.Commit();

As you can see, the total size of the transaction (not taking keys into
account) is 2M * 512B = 1GB.  If we limit the loop up to 1.9M, it works...
and I've found where the problem is:

 

As you can see, as "doubleCap" is an int, trying to double it when "cap" is
big enough makes it negative, therefore, it's not finally doubled... which
leads to a reallocation of 1GB each time a new key-value entry is added to
the tcp message.

Using integers to store capacity in your C++ Thin Client is implicitly
limiting your maximum transaction size up to 1GB.  Maybe you should consider
to use uint64_t instead...






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Long transaction suspended

[jjimeno]

Another test with 2M keys and 1GB also remains in SUSPENDED state after 11
minutes...

I don't understand where the difference between this one and the successful
1.2M keys and 600MB could be.  Any idea is welcomed 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Long transaction suspended

[jjimeno]

Hi again,

For an smaller succeeding transaction 1.2M keys and 600MB in size, I noticed
it changed its state something similar as follows:

SUSPENDED -> ACTIVE -> COMMITTING

... and it takes around 3 min to finish.

For another test with 4M keys and 2GB it is still in SUSPENDED state after
30 min.

There is a maximum number of keys/size for a single transaction?
There is any documentation out there about transaction states?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Long transaction suspended

[jjimeno]

Hi all,

I'm trying to commit a very large transaction (8M keys and ~4GB of data).

After a while, I can see this diagnostics message in node log:
[08:56:31,721][WARNING][sys-#989][diagnostic] >>> Transaction
[startTime=08:55:22.095, curTime=08:56:31.712, ... *state=SUSPENDED* ... 

Does anyone know why it is suspended, and how to avoid it?

Thanks in advance
José





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: putAll() performance big difference, 2.7.5 vs 2.9.1

[jjimeno]

Hi,

Maybe it has to do with this:

Performance of Ignite as key-value datastore. C++ Thin Client

  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Re[4]: Performance of Ignite as key-value datastore. C++ Thin Client.

[jjimeno]

Thanks for updating!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Re[4]: Performance of Ignite as key-value datastore. C++ Thin Client.

[jjimeno]

Ok... thanks!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Re[2]: Performance of Ignite as key-value datastore. C++ Thin Client.

[jjimeno]

First off, thanks to you and Zhenya for your support

I'm afraid that's not easy, as key's data are unrelated. 

As we are in an evaluation phase, we would prefer to wait for the fix and
perform single node benchmarking in the meanwhile... Instead of grouping
keys, we might split the big transaction into several smaller ones, as
Zhenya suggested, but that will not give us the real timings anyway.

According to Zhenya's mail, the problem has been identified and it seems you
can solve it. I know it's difficult, but.. could you give a rough fix date?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Performance of Ignite as key-value datastore. C++ Thin Client.

[jjimeno]

Hi again,

As a test, I just disabled persistence in both nodes.  The already mentioned
transaction of 1.2 million keys and 600MB in size takes 298sec.

Remember that for one single node and persistence enabled it takes 70sec, so
just adding a second node makes the test more than 4 times slower. 

Is this, really, the performance that Ignite can offer? Please, don't take
me wrong, I'm just asking, not criticizing.  I want to be sure I'm not doing
anything wrong and the timings I get are the expected ones...

Thanks!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: [2.7.6] Unsupported protocol version: -76 with thin client.

[jjimeno]

Hello,

Thanks for your answer. 
The server is already in version 2.9.1 and the c++ thin client is from
master branch to get transactions support.

José 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Re[2]: Performance of Ignite as key-value datastore. C++ Thin Client.

[jjimeno]

Hello,

I understand these deadlocks when there are more than one PuAll at the same
time, but in this case there is only one, and it’s always sorted.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Re[2]: Performance of Ignite as key-value datastore. C++ Thin Client.

[jjimeno]

Hello! Ok... thanks Ilya!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Re[2]: Performance of Ignite as key-value datastore. C++ Thin Client.

[jjimeno]

2. I've found this:

[16:02:23,836][WARNING][client-connector-#139][GridDhtColocatedCache] 
Unordered map java.util.LinkedHashMap is used for putAll operation on cache
vds. This can lead to a distribut
ed deadlock. Switch to a sorted map like TreeMap instead.

... but I'm using a c++ std::map.  It's hard to think that it is being
mapped to a unordered map in Java and how a deadlock is possible if keys are
no repeated.  I've modified my code to use Put instead of PutAll and the
transaction finishes... in 312sec

There is a way of fixing this?

3. I think the fix to the problem is not increasing a timeout but in
reducing the commit time. More than 300 sec for a transaction of 600MB is
not what I would expect from Ignite



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Performance of Ignite as key-value datastore. C++ Thin Client.

[jjimeno]

Hi Zhenya,

Thanks for your quick response

1. If it is not set, the following message appears on node's startup:
[13:22:33] Message queue limit is set to 0 which may lead to potential OOMEs
when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to
message queues growth on sender and receiver sides.
2. No improvement.
3. I've got the same question :)  Here are the cluster logs until it
crashes:  ignite-9f92ab96.log
 
 
4. Yes, I'm aware about that since I reported it... but there is only one
transaction in the test
5. Yes, 4Gb is large enough.  There is only one single transaction of 600MB
6. Yes, in fact, that's why I modified the page size



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Performance of Ignite as key-value datastore. C++ Thin Client.

[jjimeno]

Hi everyone,

For our project, we have the next requirements:

- One single cache 
- Ability to lock a list of cache entries.
- Large transactions. A typical one is to commit 1.2 million keys (a single
PutAll call) with a total size of around 600MB.
- Persistence

In our proof of concept, we've got everything implemented and running:

- One server node 2.9.1. Native persistence is enabled for default data
region.
- One client application using one Ignite C++ Thin Client to connect to the
server node.
- Both, server and client, are in the same machine by now.

With this scenario, we're currently evaluating Ignite vs RocksDB.  We would
really like to choose Ignite because of its scalability, but we are facing a
problem related to its performance:

In Ignite, one single transaction commit of 1.2 million keys and 600MB takes
around 70 seconds to complete, while RocksDB takes no more than 12 seconds. 
Moreover, if a second local node is added to the cluster, the application is
not even able of completing the transaction (it stops after 10 minutes)

Default data region's page size has been modified up to 16KB. Persistence
has been enabled.
Cache is PARTITIONED with TRANSACTIONAL atomicity mode.  
Because of the requirement about locking keys, performed transaction is
PESSIMISTIC + READ_COMMITTED.

The rest of the configuration values are the default ones (No backup,
PRIMARY_SYNC, no OnHeapCache, etc)

So, my questions are:

- Taking the requirements into account, is Ignite a good option?
- It's those time values that one might expect?
- If not, any advice to improve them?

Configuration files for both server nodes have been attached.  Thanks
everyone in advance for your help and time,

first-node.xml
  
second-node.xml
  

Josemari



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Multithread transactions in a C++ Thin Client

[jjimeno]

Ok... thanks Igor!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Multithread transactions in a C++ Thin Client

[jjimeno]

Hello all,

We're developing a multithread application using one C++ Thin Client to
connect to a cluster with a single Server Node.  The C++ Thin Client version
is "master" from January 21.

We have implemented a "lock-and-update" system based on the "GetAndPut"
function and PESSIMISTIC+READ_COMMITTED transactions. The idea is to lock a
set of cache entries, update them and commit them atomically.

In our tests we have detected a deadlock when following piece of code is
executed for more than one thread on our application:

...

ClientTransactions transactions = client.ClientTransactions();
ClientTransaction tx = transactions.TxStart(PESSIMISTIC, READ_COMMITTED);

// This call should atomically get the current value for "key" and put
"value" instead, locking the "key" cache entry at the same time
auto oldValue = cache.GetAndPut(key, value);

// Only the thread able of locking "key" should reach this code. Others have
to wait for tx.Commit() to complete
cache.Put (key, newValue);

// After this call, other thread waiting in GetAndPut for "key" to be
released should be able of continuing
tx.Commit ();

...

The thread reaching "cache.Put (key, newValue);" call, gets blocked in
there, concretely in the lockGuard object created at the beginning of
DataChannel::InternalSyncMessage function (data_channel.cpp:108).  After
debugging, we realized that this lockGuard is owned by a different thread,
which is currently waiting on socket while executing GetAndPut function. 
According to this, my guess is that data routing for C++ Thin Clients is not
multithread friendly.  

I did a test creating a C++ Thin Client for each different thread and the
problem disappeared, but this is something I would like to avoid since
threads are created and destroyed on the fly.

So, my questions is: do I have to create a C++ thin client for each
different thread or there is any workaround?

Thanks in advance!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/