Re: Severe performance impact to key-value API on an SQL-enabled cache (even with empty QueryEntity configuration)

[max904]

>> I'm not sure what this test doing

The test is demonstrating significant reduction of the throughput when empty
QueryEntity added to the cache config.

>> how many ops are done per iteration?

Iterations are time based. It uses the default defined by JMH. One iteration
conducted to warm up during 5 seconds and three iterations conducted to take
the measurement using default interval (I think something around 10
seconds).

>> Why do you need to start Ignite for every iteration

No particular reason. Done just in sake of clean benchmark results. What
difference does it make?

>> but I want to say that adding indexing to your cache means having a
>> primary key, in your case a composite primary key, and updating this
>> index is going to take some CPU time, especially since there's nothing
>> else to consume it.

First of all, there are no explicit indexes defined by the test.
The throughput got reduced from 51566.775 ops/s in simple K-V case to
22305.457 ops/s when "empty" QueryEntity added. It's more than 50%!
Do you believe that implicit primary key creation has such severe impact?





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Severe performance impact to key-value API on an SQL-enabled cache (even with empty QueryEntity configuration)

[max904]

Thank you, Anton!

Please find the reproducer code attached.
employee_benchmark.zip

  

I ran the test with Java 11 and Ignite 2.8.1.

Following additional dependencies needed in order to run JMH benchmark:

testImplementation 'org.openjdk.jmh:jmh-core:1.23'
testImplementation 'org.openjdk.jmh:jmh-generator-annprocess:1.23'
testAnnotationProcessor 'org.openjdk.jmh:jmh-generator-annprocess:1.23'




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Severe performance impact to key-value API on an SQL-enabled cache (even with empty QueryEntity configuration)

[max904]

I've come across strange and severe performance impact that Ignite
QueryEntity makes onto the regular cache operations.
I narrowed it down just to adding "empty" QueryEntity to the
CacheConfiguration as below:
final QueryEntity queryEntity = new QueryEntity(EmployeeId.class,
Employee.class);
queryEntity.setTableName("EMPLOYEE");
cacheConfig.setQueryEntities(Collections.singletonList(queryEntity));
Below are the results of JMH run for put/get cache operations

TEST 1: Employee cache put/get with NO QueryEntity added.
Result "EmployeeBenchmark.putGetBinary":
  51566.775 �(99.9%) 2186.773 ops/s [Average]
  (min, avg, max) = (51431.529, 51566.775, 51659.873), stdev = 119.864
  CI (99.9%): [49380.001, 53753.548] (assumes normal distribution)

BenchmarkMode  Cnt  Score   Error  Units
EmployeeBenchmark.putGetBinary  thrpt3  51566.775 �  2186.773  ops/s

--

TEST 2: Employee cache put/get with 'empty' QueryEntity added.
Result "EmployeeBenchmark.putGetBinary":
  22305.457 �(99.9%) 20543.041 ops/s [Average]
  (min, avg, max) = (21156.787, 22305.457, 23407.399), stdev = 1126.033
  CI (99.9%): [1762.416, 42848.498] (assumes normal distribution)

BenchmarkMode  Cnt  Score   Error  Units
EmployeeBenchmark.putGetBinary  thrpt3  22305.457 � 20543.041  ops/s

The difference is very significant (5/2 and in some runs even 5/1) and I
cannot explain it by any indexing (there is none in both cases).
- How can this be explained and mitigated?
- Am I doing something wrong?

I have already checked the article on "indexes not fitting into maximum
inline size" as possible cause of performance degradation on the key-value
API, but it does not apply in my case (there are no indexes defined).

Find more concrete details below...
--

// The cache configuration is very simple as below.
// Where Employee and EmployeeId are very simple Java beans.

final CacheConfiguration cacheConfig = new
CacheConfiguration<>();
cacheConfig.setTypes(EmployeeId.class, Employee.class);
cacheConfig.setName("employee");

// The following fragment disabled in TEST 1 and enabled in TEST 2
final QueryEntity queryEntity = new QueryEntity(EmployeeId.class,
Employee.class);
queryEntity.setTableName("EMPLOYEE");
cacheConfig.setQueryEntities(Collections.singletonList(queryEntity));

IgniteCache employeeCache =
ignite.getOrCreateCache(cacheConfig).withKeepBinary();

// The benchmark code is extremely simple - create new Employee BinaryObject
and perform put/get on it
// Note that I have to "clone" the key due to the other bug I reported
earlier (https://issues.apache.org/jira/browse/IGNITE-12911)

@Benchmark
@OutputTimeUnit(TimeUnit.SECONDS)
@BenchmarkMode(Mode.Throughput)
@Warmup(iterations = 1, time = 5)
@Measurement(iterations = 3)
public void putGetBinary(Blackhole bh) throws Exception {
  final BinaryObject employee = newEmployee(ignite, employeeNumber);
  final BinaryObject employeeId =
  ignite.binary().builder((BinaryObject) employee.field("id")).build();
  employeeCache.put(employeeId, employee);
  bh.consume(employeeCache.get(employeeId));
}

// The Employee BinaryObject instantiated like to following

static BinaryObject newEmployee(Ignite ignite, AtomicInteger employeeNumber)
{
  final BinaryObjectBuilder employeeId =
ignite.binary().builder(EmployeeId.class.getName());
  employeeId.setField("departmentNumber", 123, Integer.class);
  employeeId.setField("employeeNumber", employeeNumber.incrementAndGet(),
Integer.class);

  final BinaryObjectBuilder address =
ignite.binary().builder(Address.class.getName());
  address.setField("street", "101 2nd St", String.class);
  address.setField("city", "Palo Alto", String.class);
  address.setField("state", "CA", String.class);
  address.setField("zip", 94306, Integer.class);
  address.setField("country", "USA", String.class);

  final BinaryObjectBuilder employee =
ignite.binary().builder(Employee.class.getName());
  employee.setField("id", employeeId);
  employee.setField("firstName", "John", String.class);
  employee.setField("lastName", "Smith", String.class);
  employee.setField("birthDate", newDate(1968, 12, 24), Date.class);
  employee.setField("hireDate", newDate(2010, 6, 15), Date.class);
  employee.setField("homeAddress", address);

  return employee.build();
}




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite crashes with CorruptedTreeException: "B+Tree is corrupted" on a composite BinaryObject scenario

[max904]

Thank you, Alex!
Yes, I figured that this line indeed causes this crash. And yes, I'm using
queries and I need this line.
For now, I'm using the workaround I've described (of rebuilding the key
BinaryObject). But it's quite expensive operation and I would like to avoid
it.
This looks like a bug to me, as in any case, Ignite should not crash so
miserably, especially on such common condition.
If you file a ticket, could you please post a reference number for it to
track?

Best regards,
Maxim



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite crashes with CorruptedTreeException: "B+Tree is corrupted" on a composite BinaryObject scenario

[max904]

Thank you, Alex!
Yes, I figured that this line indeed causes this crash. And yes, I'm using
queries and I need this line.
For now, I'm using the workaround I've described (of rebuilding the key
BinaryObject). But it's quite expensive operation and I would like to avoid
it.
This looks like a bug to me, as in any case, Ignite should not crash so
miserably, especially on such common condition.
If you file a ticket, could you please post a reference number for it to
track?

Best regards,
Maxim




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite crashes with CorruptedTreeException: "B+Tree is corrupted" on a composite BinaryObject scenario

[max904]

Yes, of course I'm using "cache.withKeepBinary()". Below is my exact
reproducer:

final URL configUrl =
getClass().getClassLoader().getResource("example-ignite.xml");
final Ignite ignite = Ignition.start(configUrl);

final CacheConfiguration cacheConfig = new
CacheConfiguration<>("Employee");
cacheConfig.setIndexedTypes(EmployeeId.class, Employee.class);

IgniteCache cache =
ignite.getOrCreateCache(cacheConfig);
IgniteCache employeeCache =
cache.withKeepBinary();

try {
  BinaryObjectBuilder key1Builder =
ignite.binary().builder(EmployeeId.class.getName());
  key1Builder.setField("employeeNumber", 65348765, Integer.class);
  key1Builder.setField("departmentNumber", 123, Integer.class);
  BinaryObject key1 = key1Builder.build();
  BinaryObjectBuilder emp1Builder =
ignite.binary().builder(Employee.class.getName());
  emp1Builder.setField("firstName", "John", String.class);
  emp1Builder.setField("lastName", "Smith", String.class);
  emp1Builder.setField("id", key1);
  BinaryObject emp1 = emp1Builder.build();

  employeeCache.put(key1, emp1);
  BinaryObject emp2 = employeeCache.get(key1);
  assertThat(emp2).isNotNull();
  assertThat(emp2).isEqualTo(emp1);

  employeeCache.put(key1, emp1);

  BinaryObject key2 = emp1.field("id");
  employeeCache.put(key2, emp1); // CRASH!!! CorruptedTreeException: B+Tree
is corrupted

  //employeeCache.put(key2.clone(), emp1); // CRASH!!!
CorruptedTreeException: B+Tree is corrupted

  employeeCache.put(key2.toBuilder().build(), emp1); // OK!
} finally {
  Ignition.stop(true);
}



Where the data types are the following:

public interface EmployeeId {
  int getEmployeeNumber();
  void setEmployeeNumber(int employeeNumber);

  int getDepartmentNumber();
  void setDepartmentNumber(int departmentNumber);
}

public interface Employee {

  EmployeeId getId();
  void setId(EmployeeId id);

  String getFirstName();
  void setFirstName(String firstName);

  String getLastName();
  void setLastName(String lastName);

  Date getBirthDate();
  void setBirthDate(Date birthDate);

  ...
}





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Ignite crashes with CorruptedTreeException: "B+Tree is corrupted" on a composite BinaryObject scenario

[max904]

Ignite crashes when I use composite BinaryObject as a key and also include it
into the value object.

Here is the BinaryObject scenario:

// create a new composite key
BinaryObjectBuilder key1Builder =
Ignition.ignite().binary().builder(EmployeeId.class.getName());
key1Builder.setField("employeeNumber", 65348765, Integer.class);
key1Builder.setField("departmentNumber", 123, Integer.class);
BinaryObject key1 = key1Builder.build();

// create a new value
BinaryObjectBuilder emp1Builder =
Ignition.ignite().binary().builder(Employee.class.getName());
emp1Builder.setField("firstName", "John", String.class);
emp1Builder.setField("lastName", "Smith", String.class);
emp1Builder.setField("id", key1); // The composite key is also a part of the
value!
BinaryObject emp1 = emp1Builder.build();

// put the record to the DB - OK
employeeCache.put(key1, emp1);
// read it back - OK
BinaryObject emp2 = employeeCache.get(key1);
assertThat(emp2).isNotNull();
assertThat(emp2).isEqualTo(emp1);

// put the same key and value back to the DB - OK
employeeCache.put(key1, emp1); // OK!

// extract a key from the value
BinaryObject key2 = emp1.field("id");

// try to put a record with the extracted key - CRASH
employeeCache.put(key2, emp1); // CRASH!!! CorruptedTreeException: B+Tree is
corrupted ...

// try to put a record with the extracted key clone - CRASH
employeeCache.put(key2.clone(), emp1); // CRASH!!! CorruptedTreeException:
B+Tree is corrupted ...

// try to put a record with the extracted key rebuilt - OK
employeeCache.put(key2.toBuilder().build(), emp1); // OK!

This is clearly a bug as Ignite node crashes on a such basic use case!
I expect the scenario should work in all three cases, not only when I
explicitly rebuild the extracted key.
I verified it fails in both Ignite 2.8.0 and 2.7.6 (with a different error
though).
I can provide all the stack traces upon request.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Cache#replace(K,V,V) implementation

[max904]

1) How does Ignite implement this Cache#boolean replace(K key, V oldValue, V
newValue) function?JSR107 tells that it's equivalent to: if
(cache.containsKey(key) && *equals*(cache.get(key), oldValue)) { 
cache.put(key, newValue); return true; } else {  return false; }but I assume
this is just logical equivalence.How Ignite really perform
"equals(cache.get(key), oldValue)"? Or it uses some more intelligent way,
like comparing some hash codes or other kind of metadata. Performing
"equals" on the values could be very expensive operation.My question is very
practical. I'm implementing a Web-service that's backed by Ignite, which has
to support "Last-Modified" or "ETag" HTTP headers. These headers would make
sure the data has not been sent over the wire unnecessarily if nothing has
changed, and the data has not been concurrently updated since the last time
it was requested. These headers allow client or proxy to cache the data,
especially if the data is of a substantial size. I'm struggling to find the
most optimal way to map it to my cache operations. Therefore, here is my
second question.2) How would you recommend implementing "Last-Modified" or
"ETag" semantics based on Ignite? (more specifically in the context of
replacing/updating the values)



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Query on nested objects

[max904]

Thank you for the answer! It helped.

I discovered two functions that can help in controlling the way ignite the
attributes onto table aliases.

QueryEntity.addQueryField("homeAddress.zip", "java.lang.String", "ha_zip")
would allow programmatically define the "homeAddress.zip" field in the table
and map it to "ha_zip" alias.

QueryEntity.setAliases(new HashMap(){{put("homeAddress.zip",
"ha_zip");}}) would allow creating "ha_zip" alias for the field
"homeAddress.zip" that has already been defined.

CacheConfiguration.setIndexedTypes() is what I was trying to use before with
@QuerySqlField annotations on the object classes and that what did not work
for the field clashes.
It seems like a defect in the code when the filed name clashes are not being
resolved or even indicated.

The goal of all these is to be able to run a query on a simple model with
multiple nested objects (which could possibly clash on some attributes).
Yes, the RDBMS tables are flat, but Ignite is not RDBMS, but a K-V store
with SQL query capabilities. Ignite permits using complex object structures
as cache values and advertises a capability to query on those.

Again, thanks a lot for your help!




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Query on nested objects

[max904]

I have following "Person" data model:

public class Address implements Binarylizable {
  @QuerySqlField
  private String street;

  @QuerySqlField
  private String city;

  @QuerySqlField
  private String state;

  @QuerySqlField
  private int zip;

  @QuerySqlField
  private String country;
  ...
}

public class Person implements Binarylizable {
  @QuerySqlField(index = true)
  private String name;

  @QuerySqlField
  private Date dateOfBirth;

  @QuerySqlField
  private String email;

  @QuerySqlField
  private String[] telephoneNumber;

  @QuerySqlField
  private *Address homeAddress*;

  @QuerySqlField
  private *Address workAddress*;
  ...
}

public class PersonKey implements Binarylizable {
  @GridToStringInclude
  @QuerySqlField(index = true)
  private String personKey;

  @AffinityKeyMapped
  @QuerySqlField(index = true)
  private String orgKey;
  ...
}

Now, when I create a cache for it and start working with it, I get the
following data model for SQL:

CREATE MEMORY TABLE "example.com:Person".PERSON(
_KEY OTHER INVISIBLE  NOT NULL,
_VAL OTHER INVISIBLE ,
_VER OTHER INVISIBLE ,
PERSONKEY VARCHAR,
ORGKEY VARCHAR,
NAME VARCHAR,
DATEOFBIRTH TIMESTAMP,
EMAIL VARCHAR,
TELEPHONENUMBER ARRAY,
*HOMEADDRESS* OTHER,
STREET VARCHAR,
CITY VARCHAR,
STATE VARCHAR,
ZIP INT,
COUNTRY VARCHAR,
*WORKADDRESS* OTHER
)
ENGINE "org.apache.ignite.internal.processors.query.h2.H2TableEngine"

It's a very basic data model, but I have problems working with it. Hence, I
would like to ask a few questions:
1. I'm able to query a Person based on it's homeAddress zip (with "SELECT *
FROM Person WHERE zip = 123"), but I'm unable to do so based on workAddress
zip. How can that be achieved?

2. How I can control how Ignite flattens the nested objects? Can I control
that?

3. Can I avoid flattening and use nested names in my SQL query "SELECT *
FROM Person WHERE workAddress.zip = 123"? I think I've seen some examples of
that but I could not figure out how to make it work.

4. The table structure indicates that both the "HOMEADDRESS OTHER" and all
of its fields present in the table. Does it mean that Ignite will store and
manage homeAddress data twice?

I'm running Ignite 2.6 and simple example-ignite.xml to run the nodes.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/