Client Connecting Issues

[Alper Tekinalp]

Hi all.


We have two set of applications: server and client applications. Servers
running all the time and with client applications we connect to server, do
things and disconnect, etc. We have some issues with that connections:

1 - We use TcpDiscoveryVmIpFinder to connect cluster. But establishing
connection take some time. We have default 20 sec that clients try to
connect. After that we start client aplications in offline mode. Is it
normal? Is there a way to decrease that time?

2 - Most of the time through the life of client apps, client disconnects
from cluster, goes offline mode and never reconnects without a restart.
After restart it disconnects after some time again. Most of the time we are
connecting with client apps through VPN and we dont have the chance to
improve or edit network configuration or quality.

Our SPI config for client is:

private IgniteConfiguration getIgniteConfiguration() {
TcpDiscoverySpi spi = new TcpDiscoverySpi();
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();

ipFinder.setAddresses("IP of oldest node in cluster");

spi.setIpFinder(ipFinder);

IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
igniteConfiguration.setClientMode(true)
.setPeerClassLoadingEnabled(false)
.setIncludeEventTypes(EventType.EVTS_DISCOVERY)
.setDiscoverySpi(spi)
.setAddressResolver(new AmazonAddressResolver());

return igniteConfiguration;
}

Do you have any suggestions? What can we do about especially for second
case?

Regards.

-- 

*Alper Tekinalp*


*Software Developer*


*Evam ​Stream Analytics*

*Atatürk Mah.Turgut Özal Bulvarı Gardenya 1 Plaza 42/B K:4 Ataşehir /
İSTANBUL*

*Tlf : +90216 688 45 46 <%2B90216%20688%2045%2046> Fax : +90216 688 45 47
<%2B90216%20688%2045%2047> Gsm:+90 536 222 76 01
<%2B90%20553%20489%2044%2099>*
*www.evam.com *

Re: Enums and SQL queries

[Sergi Vladykin]

Sure.

https://issues.apache.org/jira/browse/IGNITE-3595

Sergi

2016-07-27 22:51 GMT+03:00 Dmitriy Setrakyan :

> Sorry, Sergi. What I meant is to file a ticket with the design you
> suggested
>
> Dmitriy
>
>
>
> > On Jul 27, 2016, at 9:18 AM, Sergi Vladykin 
> wrote:
> >
> > I don't see how this can work to file a ticket, could you please give
> more
> > details?
> >
> > Sergi
> >
> > 2016-07-27 15:50 GMT+03:00 Dmitriy Setrakyan :
> >
> >> Sergi, I still thinks my solution works or can be made to work. In the
> >> example you provided, you can choose to return either String or Enum as
> the
> >> query result. However, this is not a clean solution, I agree, but could
> be
> >> a quick one.
> >>
> >> I don't mind breaking compatibility on this issue, given that it never
> >> worked properly anyway. Can you please file a ticket?
> >>
> >> D.
> >>
> >> On Wed, Jul 27, 2016 at 3:01 AM, Sergi Vladykin <
> sergi.vlady...@gmail.com>
> >> wrote:
> >>
> >>> Nope. This will not work. Because take for example the following query:
> >>>
> >>> select my_enum_field from my_table where my_enum_field = 'BLA'
> >>>
> >>> Field well be evaluated only once here, but it must be a Sting and Enum
> >> at
> >>> the same time.
> >>
> >>
> >>> Probably we could contribute to H2 `User defined value types` and
> handle
> >>> this stuff in our own Enum value types, which will support comparison
> >> with
> >>> String. But this requires bug H2 Modification and is a compatibility
> >>> breaking change.
> >>>
> >>> Sergi
> >>>
> >>>
> >>>
> >>> 2016-07-27 9:23 GMT+03:00 Dmitriy Setrakyan :
> >>>
>  Hm... I would prefer to find a way to handle Enums automatically.
> 
>  In case if user expects a String, can we catch an exception and do
>  automatic conversion at that time? In this case, we could catch the
>  exception once and remember the decision. Will something like this
> >> work?
> 
>  D.
> 
>  On Tue, Jul 26, 2016 at 1:49 AM, Sergi Vladykin <
> >>> sergi.vlady...@gmail.com>
>  wrote:
> 
> > GridQueryProcessor contains two types: BinaryProperty and
> >>> ClassProperty.
> > Indexing uses them for fields access. Probably we can change them, so
>  that
> > they will always return String instead of Enum type but this
> >
> > 1. looks like a compatibility breaking change;
> > 2. users doing queries like `select my_enum_field from my_table`
> >>> probably
> > still expect to get a Enum result but not a String.
> >
> > What do you think?
> >
> > Sergi
> >
> >
> > 2016-07-26 3:17 GMT+03:00 Valentin Kulichenko <
> > valentin.kuliche...@gmail.com
> >> :
> >
> >> Sergi,
> >>
> >> Ideally, I would like this query to work:
> >>
> >> select * from MyTable where myEnumField = 'MY_ENUM_NAME'
> >>
> >> The problem is that the field value is not a string, it's a binary
>  object
> >> with the enum value ordinal inside. We can add enum value names
> >> into
>  the
> >> metadata, so that we can always map the field value to its string
> >> representation, but SQL engine will have to know how to use it. Is
> >>> this
> >> possible?
> >>
> >> -Val
> >>
> >> On Fri, Jul 22, 2016 at 10:25 PM, Sergi Vladykin <
> > sergi.vlady...@gmail.com
> >>> wrote:
> >>
> >>> I'm not sure what exactly you are going to patch in SQL engine. We
>  could
> >>> and some SQL function like enum('MY_ENUM_NAME') but probably it
> >> will
>  not
> >>> work if we will not pass enum type there as well, so it have look
> >>> like
> >>> enum('com.mycompany.MyEnum', 'MY_ENUM_NAME') which is ugly.
> >>>
> >>> Sergi
> >>>
> >>> On 23 июля 2016 г., 0:43, Valentin Kulichenko <
> >>> valentin.kuliche...@gmail.com
>  wrote:
> >>>
>  Folks,
> 
>  I noticed that we currently have a very weird limitation for
> >> enum
> > fields
>  when running SQL queries. Basically, you can't use enum value
> >>> names
>  as
> >>> many
>  users would expect.
> 
>  Technically this happens because binary format stores only
> >> ordinal
>  and
> >>> the
>  only way to run such query is to do something like this:
> 
>  SqlFieldsQuery qry = new SqlFieldsQuery("select * from Table
> >> where
>  enumField = ?");
>  qry.setArgs(MyEnum.ENUM_VALUE);
>  cache.query(qry);
> 
>  This means that this query can be executed only if:
>  - IgniteCache API is used. So it doesn't work in JDBC driver,
> > Zeppelin,
>  etc.
>  - The client that executes the query has the enum class on
>  classpath.
> 
>  My first thought about fixing this is to somehow store value
> >> names
>  in
>  binary metadata and patch the SQL engine so that it understands
> >>> how
>  to
> >>> use
> >>>

[jira] [Created] (IGNITE-3595) Improve Enum fields handling in SQL.

[Sergi Vladykin (JIRA)]

Sergi Vladykin created IGNITE-3595:
--

 Summary: Improve Enum fields handling in SQL.
 Key: IGNITE-3595
 URL: https://issues.apache.org/jira/browse/IGNITE-3595
 Project: Ignite
  Issue Type: Improvement
Affects Versions: 1.7
Reporter: Sergi Vladykin
 Fix For: 2.0


Currently queries like this do not work in Ignite if we use enum field on the 
left side and enum constant name or ordinal on the right side:

select * from A where my_enum = 'ENUM_CONST';
select * from A where my_enum = 3;

This is a huge usability issue.

We can try to solve it by contributing `User defined Value types` in H2 and 
implementing the special value type for Enums to handle comparison with String 
(convert to enum by name) and with int (convert to enum by ordinal).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: IGNITE-2294 implementation details

[Sergi Vladykin]

I had a quick look at the PR.

I don't like this @QueryCacheKey and setKeyProp method on public API. They
solve nothing but add complexity and make key to be stored twice in cache,
which is wrong. Please remove this.

If you want to do some public API changes you have to discuss them publicly
before implementing them, ok?

I did not look deeper yet, lets fix the obvious issue first.

Sergi

2016-07-27 21:44 GMT+03:00 Alexander Paschenko <
alexander.a.pasche...@gmail.com>:

> Sergi,
>
> I've made changes to the API according to your valuable
> recommendations, thank you very much for giving them. Please refer to
> PR to see current state of the work.
> Will surely look into ODBC, .NET and Visor. Though they will most
> likely have to support a new feature rather than considerably change
> existing logic.
>
> - Alex
>
> 2016-07-27 14:23 GMT+03:00 Sergi Vladykin :
> > Please don't forget about ODBC, .NET and Visor. They all have to work in
> > the same way.
> >
> > Sergi
> >
> > 2016-07-27 14:15 GMT+03:00 Alexander Paschenko <
> > alexander.a.pasche...@gmail.com>:
> >
> >> OK, I've found that bold cast to QueryCursor in IgniteCacheProxy
> >> and had a look at how SqlFieldsQuery is used in JDBC driver. Thanks.
> >>
> >> - Alex
> >>
> >> 2016-07-27 13:02 GMT+03:00 Sergi Vladykin :
> >> > Where did you see R in SqlFieldsQuery?
> >> >
> >> > Sergi
> >> >
> >> > 2016-07-27 12:59 GMT+03:00 Alexander Paschenko <
> >> > alexander.a.pasche...@gmail.com>:
> >> >
> >> >> Sergi,
> >> >>
> >> >> But current signature of query() method returns not just some
> >> >> iterator, but rather iterator of R which is type param of Query -
> >> >> i.e., we won't be able to return an int inside a QueryCursor. At
> >> >> least without API change (signature of query() method will have to be
> >> >> changed to drop genericness, or in some other weird way). Is this
> what
> >> >> we really want? Or am I missing something in your point?
> >> >>
> >> >> - Alex
> >> >>
> >> >> 2016-07-27 12:51 GMT+03:00 Sergi Vladykin  >:
> >> >> > Exactly. This will allow our Jdbc driver to work transparently.
> >> >> >
> >> >> > Sergi
> >> >> >
> >> >> > 2016-07-27 12:40 GMT+03:00 Alexander Paschenko <
> >> >> > alexander.a.pasche...@gmail.com>:
> >> >> >
> >> >> >> Sergi,
> >> >> >>
> >> >> >> You wrote:
> >> >> >> > I'd prefer to return the same information, so it will not be
> empty
> >> >> >>
> >> >> >> Do you mean return iterator with single element that denotes
> number
> >> of
> >> >> >> rows?
> >> >> >>
> >> >> >> Dmitriy,
> >> >> >>
> >> >> >> You wrote:
> >> >> >> > What is the ticket number for this. Is the new API documented
> >> there?
> >> >> >>
> >> >> >> Overall issue number is 2294. There's no particular issue on API
> >> >> >> changes, but creating one seems to be a good idea, I will do it.
> >> >> >>
> >> >> >> - Alex
> >> >> >>
> >> >> >> 2016-07-27 9:20 GMT+03:00 Dmitriy Setrakyan <
> dsetrak...@apache.org>:
> >> >> >> > What is the ticket number for this. Is the new API documented
> >> there?
> >> >> >> >
> >> >> >> > On Tue, Jul 26, 2016 at 11:36 AM, Sergi Vladykin <
> >> >> >> sergi.vlady...@gmail.com>
> >> >> >> > wrote:
> >> >> >> >
> >> >> >> >> I don't see anything ugly in empty iterator, sorry if I
> insulted
> >> your
> >> >> >> taste
> >> >> >> >> of beauty.
> >> >> >> >>
> >> >> >> >> If you will take a look at Jdbc, you will see that
> >> >> >> Statement.executeUpdate
> >> >> >> >> method returns number of updated rows, I'd prefer to return the
> >> same
> >> >> >> >> information, so it will not be empty (beauty is restored!).
> >> >> >> >>
> >> >> >> >> Sergi
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> 2016-07-26 18:24 GMT+03:00 Alexander Paschenko <
> >> >> >> >> alexander.a.pasche...@gmail.com>:
> >> >> >> >>
> >> >> >> >> > I see your point. But what about my concerns from initial
> post?
> >> >> >> >> > Particularly about signatures of existing methods? I
> personally
> >> >> don't
> >> >> >> >> > like an option of query() method always returning an empty
> >> iterator
> >> >> >> >> > for any non-select query, it seems ugly design wise.
> >> >> >> >> >
> >> >> >> >> > - Alex
> >> >> >> >> >
> >> >> >> >> > 2016-07-26 18:15 GMT+03:00 Sergi Vladykin <
> >> >> sergi.vlady...@gmail.com>:
> >> >> >> >> > > BTW, the simplest way to solve this issue is to allow
> running
> >> SQL
> >> >> >> >> > commands
> >> >> >> >> > > inside of SqlFieldsQuery.
> >> >> >> >> > >
> >> >> >> >> > > We may add some additional convenience API for updates if
> we
> >> >> want,
> >> >> >> but
> >> >> >> >> > JDBC
> >> >> >> >> > > client will always call it like this:
> >> >> >> >> > >
> >> >> >> >> > > cache.query(new SqlFieldsQuery("INSERT INTO MY_TABLE
> >> >> >> >> > > VALUES(?,?)").setArgs(1,2));
> >> >> >> >> > >
> >> >> >> >> > > This will resolve any ambiguity.
> >> >> >> >> > >
> >> >> >> >> > > Sergi
> >> >> >> >> > >
> >> >> >> >> > > 2016-07-26 17:56 GMT+03:00 Sergi Vladykin <
> >> >> sergi.vlady...@gmail.com
> >> >> >> 

[jira] [Created] (IGNITE-3594) ODBC: Add connection properties to support distributed joins.

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-3594:
---

 Summary: ODBC: Add connection properties to support distributed 
joins.
 Key: IGNITE-3594
 URL: https://issues.apache.org/jira/browse/IGNITE-3594
 Project: Ignite
  Issue Type: Task
  Components: odbc
Affects Versions: 1.6
Reporter: Vladimir Ozerov
Assignee: Igor Sapego
Priority: Critical
 Fix For: 1.7






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Apache Ignite - New Release

[Dmitriy Setrakyan]

On Wed, Jul 27, 2016 at 4:00 AM, Semyon Boikov  wrote:

> Dmitry, all Ignite features are always thoroughly tested =)
>

True, completely forgot about it :)


> I created branch ignite-1.7.0.
>
> On Wed, Jul 27, 2016 at 9:32 AM, Dmitriy Setrakyan 
> wrote:
>
> > On Wed, Jul 27, 2016 at 2:29 AM, Semyon Boikov 
> > wrote:
> >
> > > Sure Dmitry, we will run tests with large data set.
> > >
> > > Regarding 1.7. release: If there are no objections I'm going to cut off
> > 1.7
> > > branch and start prepare it for release.
> > >
> >
> > In my opinion this is a great addition to Ignite and is definitely worth
> a
> > release, again, assuming that we thoroughly test it.
> >
> >
> > >
> > >
> > > On Wed, Jul 27, 2016 at 9:15 AM, Dmitriy Setrakyan <
> > dsetrak...@apache.org>
> > > wrote:
> > >
> > > > On Wed, Jul 27, 2016 at 2:09 AM, Semyon Boikov  >
> > > > wrote:
> > > >
> > > > > Regarding distributed join testsing: we added tests verifying
> correct
> > > > join
> > > > > behavior and correct execution plan generation for various SQL
> > queries,
> > > > > tests for joins for various cache types (different number of
> backups,
> > > > > partitioned/replicated), there are tests verifying correct
> > distributed
> > > > > joins results on changing topology with nodes restarts. Also we
> added
> > > > > benchmarks which will be used to verify that there are no
> performance
> > > > > degradation in this functionality between releases. These
> benchmarks
> > > were
> > > > > executed on real clusters, and in next few days we are going to run
> > > more
> > > > > load tests.
> > > > >
> > > >
> > > > Thanks Semyon, sounds great! I would also test it on larger data sets
> > to
> > > > see how a join query will take, say, on 10GB of data. Is it possible?
> > > >
> > > >
> > > > >
> > > > > On Fri, Jul 22, 2016 at 5:39 PM, Dmitriy Setrakyan <
> > > > dsetrak...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > On Fri, Jul 22, 2016 at 7:10 AM, Semyon Boikov <
> > sboi...@gridgain.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > Today I merged into master 'distributed join' implementation -
> > > > > > > https://issues.apache.org/jira/browse/IGNITE-1232 (thanks to
> > > Sergi,
> > > > he
> > > > > > > implemented this feature). I think this together with recent
> > > bugfixes
> > > > > > worth
> > > > > > > 1.7 release. Do you think we can cut off 1.7 release branch
> from
> > > > > master?
> > > > > > >
> > > > > >
> > > > > > Great news. Can you describe that amount of testing we did for
> this
> > > > > > feature?
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jul 21, 2016 at 7:47 PM, Alexandre Boudnik <
> > > > > > > alexander.boud...@gmail.com> wrote:
> > > > > > >
> > > > > > > > Sorry, I missed the typo:
> > > > > > > > To support the point, I would add that PosgreSQL has
> > demonstrated
> > > > the
> > > > > > > > similar behavior (inspired by unix .dot files and ls):
> > > > > > > > - they have "hidden" column: xmin and xmax in any table
> > > > > > > > - they do not appear in select * list unless they are
> specified
> > > > > > > explicitly
> > > > > > > > Take care,
> > > > > > > > Alexandre "Sasha" Boudnik
> > > > > > > >
> > > > > > > > call me via Google Voice:
> > > > > > > > 1(405) BUDNIKA
> > > > > > > > 1(405) 283-6452
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Jul 21, 2016 at 12:46 PM, Alexandre Boudnik
> > > > > > > >  wrote:
> > > > > > > > > To support the point, I would add that PosgreSQL has
> > > demonstrated
> > > > > the
> > > > > > > > > similar behavior (inspired by unix .dot files and ls):
> > > > > > > > > - they have "hidden" column: xmin and xmax in any table
> > > > > > > > > - they appear in select * list unless they are specified
> > > > explicitly
> > > > > > > > >
> > > > > > > > > I'll add some notices to the ticket
> > > > > > > > > Take care,
> > > > > > > > > Alexandre "Sasha" Boudnik
> > > > > > > > >
> > > > > > > > > call me via Google Voice:
> > > > > > > > > 1(405) BUDNIKA
> > > > > > > > > 1(405) 283-6452
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Jul 15, 2016 at 6:37 PM, Valentin Kulichenko
> > > > > > > > >  wrote:
> > > > > > > > >> Agree.
> > > > > > > > >>
> > > > > > > > >> On Fri, Jul 15, 2016 at 12:59 PM, Alexey Goncharuk <
> > > > > > > > >> alexey.goncha...@gmail.com> wrote:
> > > > > > > > >>
> > > > > > > > >>> Looks like the ticket for removing _key and _value from
> > > selct *
> > > > > is
> > > > > > a
> > > > > > > > good
> > > > > > > > >>> candidate for 2.0.
> > > > > > > > >>>
> > > > > > > > >>> 2016-07-15 5:12 GMT-07:00 Sergi Vladykin <
> > > > > sergi.vlady...@gmail.com
> > > > > > >:
> > > > > > > > >>>
> > > > > > > > >>> > We will not be able to just change this, because it
> will
> > > > brake
> > > > > > > > >>> > compatibility. Still I believe that an option to define
> > our
>

Re: Batch support in Cassandra store

[Igor Rudyak]

Hi Luiz,

Logged batches is not the solution to achieve atomic view of your Ignite
transaction changes in Cassandra.

The problem with logged batches(aka atomic) is they guarantees that if any
part of the batch succeeds, all of it will, no other transactional
enforcement is done at the batch level. For example, there is no batch
isolation. Clients are able to read the first updated rows from the batch,
while other rows are still being updated on the server (in RDBMS
terminology it means *READ-UNCOMMITED* isolation level). Thus Cassandra
mean "atomic" in the database sense that if any part of the batch succeeds,
all of it will.

Probably the best way to archive read atomic isolation for Ignite
transaction persisting data into Cassandra, is to implement RAMP
transactions (http://www.bailis.org/papers/ramp-sigmod2014.pdf) on top of
Cassandra.

I may create a ticket for this if community would like it.


Igor Rudyak


On Wed, Jul 27, 2016 at 12:55 PM, Luiz Felipe Trevisan <
luizfelipe.trevi...@gmail.com> wrote:

> Hi Igor,
>
> Does it make sense for you using logged batches to guarantee atomicity in
> Cassandra in cases we are doing a cross cache transaction operation?
>
> Luiz
>
> --
> Luiz Felipe Trevisan
>
> On Wed, Jul 27, 2016 at 2:05 AM, Dmitriy Setrakyan 
> wrote:
>
>> I am very confused still. Ilya, can you please explain what happens in
>> Cassandra if user calls IgniteCache.putAll(...) method?
>>
>> In Ignite, if putAll(...) is called, Ignite will make the best effort to
>> execute the update as a batch, in which case the performance is better.
>> What is the analogy in Cassandra?
>>
>> D.
>>
>> On Tue, Jul 26, 2016 at 9:16 PM, Igor Rudyak  wrote:
>>
>> > Dmitriy,
>> >
>> > There is absolutely same approach for all async read/write/delete
>> > operations - Cassandra session just provides executeAsync(statement)
>> > function
>> > for all type of operations.
>> >
>> > To be more detailed about Cassandra batches, there are actually two
>> types
>> > of batches:
>> >
>> > 1) *Logged batch* (aka atomic) - the main purpose of such batches is to
>> > keep duplicated data in sync while updating multiple tables, but at the
>> > cost of performance.
>> >
>> > 2) *Unlogged batch* - the only specific case for such batch is when all
>> > updates are addressed to only *one* partition key and batch having
>> > "*reasonable
>> > size*". In a such situation there *could be* performance benefits if you
>> > are using Cassandra *TokenAware* load balancing policy. In this
>> particular
>> > case all the updates will go directly without any additional
>> > coordination to the primary node, which is responsible for storing data
>> for
>> > this partition key.
>> >
>> > The *generic rule* is that - *individual updates using async mode*
>> provides
>> > the best performance (
>> > https://docs.datastax.com/en/cql/3.1/cql/cql_using/useBatch.html).
>> That's
>> > because it spread all updates across the whole cluster. In contrast to
>> > this, when you are using batches, what this is actually doing is
>> putting a
>> > huge amount of pressure on a single coordinator node. This is because
>> the
>> > coordinator needs to forward each individual insert/update/delete to the
>> > correct replicas. In general you're just losing all the benefit of
>> > Cassandra TokenAware load balancing policy when you're updating
>> different
>> > partitions in a single round trip to the database.
>> >
>> > Probably the only enhancement which could be done is to separate our
>> batch
>> > to smaller batches, each of which is updating records having the same
>> > partition key. In this case it could provide some performance benefits
>> when
>> > used in combination with Cassandra TokenAware policy. But there are
>> several
>> > concerns:
>> >
>> > 1) It looks like rather rare case
>> > 2) Makes error handling more complex - you just don't know what
>> operations
>> > in a batch succeed and what failed and need to retry all batch
>> > 3) Retry logic could produce more load on the cluster - in case of
>> > individual updates you just need to retry the only mutations which are
>> > failed, in case of batches you need to retry the whole batch
>> > 4)* Unlogged batch is deprecated in Cassandra 3.0* (
>> > https://docs.datastax.com/en/cql/3.3/cql/cql_reference/batch_r.html),
>> > which
>> > we are currently using for Ignite Cassandra module.
>> >
>> >
>> > Igor Rudyak
>> >
>> >
>> >
>> > On Tue, Jul 26, 2016 at 4:45 PM, Dmitriy Setrakyan <
>> dsetrak...@apache.org>
>> > wrote:
>> >
>> > >
>> > >
>> > > On Tue, Jul 26, 2016 at 5:53 PM, Igor Rudyak 
>> wrote:
>> > >
>> > >> Hi Valentin,
>> > >>
>> > >> For writeAll/readAll Cassandra cache store implementation uses async
>> > >> operations (
>> http://www.datastax.com/dev/blog/java-driver-async-queries)
>> > >> and
>> > >> futures, which has the best characteristics in terms of performance.
>> > >>
>> > >>
>> > > Thanks, Igor. This link describes the query operations, but I could
>> not
>> > > find the me

Re: Batch support in Cassandra store

[Luiz Felipe Trevisan]

Hi Igor,

Does it make sense for you using logged batches to guarantee atomicity in
Cassandra in cases we are doing a cross cache transaction operation?

Luiz

--
Luiz Felipe Trevisan

On Wed, Jul 27, 2016 at 2:05 AM, Dmitriy Setrakyan 
wrote:

> I am very confused still. Ilya, can you please explain what happens in
> Cassandra if user calls IgniteCache.putAll(...) method?
>
> In Ignite, if putAll(...) is called, Ignite will make the best effort to
> execute the update as a batch, in which case the performance is better.
> What is the analogy in Cassandra?
>
> D.
>
> On Tue, Jul 26, 2016 at 9:16 PM, Igor Rudyak  wrote:
>
> > Dmitriy,
> >
> > There is absolutely same approach for all async read/write/delete
> > operations - Cassandra session just provides executeAsync(statement)
> > function
> > for all type of operations.
> >
> > To be more detailed about Cassandra batches, there are actually two types
> > of batches:
> >
> > 1) *Logged batch* (aka atomic) - the main purpose of such batches is to
> > keep duplicated data in sync while updating multiple tables, but at the
> > cost of performance.
> >
> > 2) *Unlogged batch* - the only specific case for such batch is when all
> > updates are addressed to only *one* partition key and batch having
> > "*reasonable
> > size*". In a such situation there *could be* performance benefits if you
> > are using Cassandra *TokenAware* load balancing policy. In this
> particular
> > case all the updates will go directly without any additional
> > coordination to the primary node, which is responsible for storing data
> for
> > this partition key.
> >
> > The *generic rule* is that - *individual updates using async mode*
> provides
> > the best performance (
> > https://docs.datastax.com/en/cql/3.1/cql/cql_using/useBatch.html).
> That's
> > because it spread all updates across the whole cluster. In contrast to
> > this, when you are using batches, what this is actually doing is putting
> a
> > huge amount of pressure on a single coordinator node. This is because the
> > coordinator needs to forward each individual insert/update/delete to the
> > correct replicas. In general you're just losing all the benefit of
> > Cassandra TokenAware load balancing policy when you're updating different
> > partitions in a single round trip to the database.
> >
> > Probably the only enhancement which could be done is to separate our
> batch
> > to smaller batches, each of which is updating records having the same
> > partition key. In this case it could provide some performance benefits
> when
> > used in combination with Cassandra TokenAware policy. But there are
> several
> > concerns:
> >
> > 1) It looks like rather rare case
> > 2) Makes error handling more complex - you just don't know what
> operations
> > in a batch succeed and what failed and need to retry all batch
> > 3) Retry logic could produce more load on the cluster - in case of
> > individual updates you just need to retry the only mutations which are
> > failed, in case of batches you need to retry the whole batch
> > 4)* Unlogged batch is deprecated in Cassandra 3.0* (
> > https://docs.datastax.com/en/cql/3.3/cql/cql_reference/batch_r.html),
> > which
> > we are currently using for Ignite Cassandra module.
> >
> >
> > Igor Rudyak
> >
> >
> >
> > On Tue, Jul 26, 2016 at 4:45 PM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
> > wrote:
> >
> > >
> > >
> > > On Tue, Jul 26, 2016 at 5:53 PM, Igor Rudyak 
> wrote:
> > >
> > >> Hi Valentin,
> > >>
> > >> For writeAll/readAll Cassandra cache store implementation uses async
> > >> operations (
> http://www.datastax.com/dev/blog/java-driver-async-queries)
> > >> and
> > >> futures, which has the best characteristics in terms of performance.
> > >>
> > >>
> > > Thanks, Igor. This link describes the query operations, but I could not
> > > find the mention of writes.
> > >
> > >
> > >> Cassandra BATCH statement is actually quite often anti-pattern for
> those
> > >> who come from relational world. BATCH statement concept in Cassandra
> is
> > >> totally different from relational world and is not for optimizing
> > >> batch/bulk operations. The main purpose of Cassandra BATCH is to keep
> > >> denormalized data in sync. For example when you duplicating the same
> > data
> > >> into several tables. All other cases are not recommended for Cassandra
> > >> batches:
> > >>  -
> > >>
> > >>
> >
> https://medium.com/@foundev/cassandra-batch-loading-without-the-batch-keyword-40f00e35e23e#.k4xfir8ij
> > >>  -
> > >>
> > >>
> >
> http://christopher-batey.blogspot.com/2015/02/cassandra-anti-pattern-misuse-of.html
> > >>  -
> https://inoio.de/blog/2016/01/13/cassandra-to-batch-or-not-to-batch/
> > >>
> > >> It's also good to mention that in CassandraCacheStore implementation
> > >> (actually in CassandraSessionImpl) all operation with Cassandra is
> > wrapped
> > >> in a loop. The reason is in a case of failure it will be performed 20
> > >> attempts to retry the operation with incrementally i

Re: Enums and SQL queries

[Dmitriy Setrakyan]

Sorry, Sergi. What I meant is to file a ticket with the design you suggested

Dmitriy



> On Jul 27, 2016, at 9:18 AM, Sergi Vladykin  wrote:
> 
> I don't see how this can work to file a ticket, could you please give more
> details?
> 
> Sergi
> 
> 2016-07-27 15:50 GMT+03:00 Dmitriy Setrakyan :
> 
>> Sergi, I still thinks my solution works or can be made to work. In the
>> example you provided, you can choose to return either String or Enum as the
>> query result. However, this is not a clean solution, I agree, but could be
>> a quick one.
>> 
>> I don't mind breaking compatibility on this issue, given that it never
>> worked properly anyway. Can you please file a ticket?
>> 
>> D.
>> 
>> On Wed, Jul 27, 2016 at 3:01 AM, Sergi Vladykin 
>> wrote:
>> 
>>> Nope. This will not work. Because take for example the following query:
>>> 
>>> select my_enum_field from my_table where my_enum_field = 'BLA'
>>> 
>>> Field well be evaluated only once here, but it must be a Sting and Enum
>> at
>>> the same time.
>> 
>> 
>>> Probably we could contribute to H2 `User defined value types` and handle
>>> this stuff in our own Enum value types, which will support comparison
>> with
>>> String. But this requires bug H2 Modification and is a compatibility
>>> breaking change.
>>> 
>>> Sergi
>>> 
>>> 
>>> 
>>> 2016-07-27 9:23 GMT+03:00 Dmitriy Setrakyan :
>>> 
 Hm... I would prefer to find a way to handle Enums automatically.
 
 In case if user expects a String, can we catch an exception and do
 automatic conversion at that time? In this case, we could catch the
 exception once and remember the decision. Will something like this
>> work?
 
 D.
 
 On Tue, Jul 26, 2016 at 1:49 AM, Sergi Vladykin <
>>> sergi.vlady...@gmail.com>
 wrote:
 
> GridQueryProcessor contains two types: BinaryProperty and
>>> ClassProperty.
> Indexing uses them for fields access. Probably we can change them, so
 that
> they will always return String instead of Enum type but this
> 
> 1. looks like a compatibility breaking change;
> 2. users doing queries like `select my_enum_field from my_table`
>>> probably
> still expect to get a Enum result but not a String.
> 
> What do you think?
> 
> Sergi
> 
> 
> 2016-07-26 3:17 GMT+03:00 Valentin Kulichenko <
> valentin.kuliche...@gmail.com
>> :
> 
>> Sergi,
>> 
>> Ideally, I would like this query to work:
>> 
>> select * from MyTable where myEnumField = 'MY_ENUM_NAME'
>> 
>> The problem is that the field value is not a string, it's a binary
 object
>> with the enum value ordinal inside. We can add enum value names
>> into
 the
>> metadata, so that we can always map the field value to its string
>> representation, but SQL engine will have to know how to use it. Is
>>> this
>> possible?
>> 
>> -Val
>> 
>> On Fri, Jul 22, 2016 at 10:25 PM, Sergi Vladykin <
> sergi.vlady...@gmail.com
>>> wrote:
>> 
>>> I'm not sure what exactly you are going to patch in SQL engine. We
 could
>>> and some SQL function like enum('MY_ENUM_NAME') but probably it
>> will
 not
>>> work if we will not pass enum type there as well, so it have look
>>> like
>>> enum('com.mycompany.MyEnum', 'MY_ENUM_NAME') which is ugly.
>>> 
>>> Sergi
>>> 
>>> On 23 июля 2016 г., 0:43, Valentin Kulichenko <
>>> valentin.kuliche...@gmail.com
 wrote:
>>> 
 Folks,
 
 I noticed that we currently have a very weird limitation for
>> enum
> fields
 when running SQL queries. Basically, you can't use enum value
>>> names
 as
>>> many
 users would expect.
 
 Technically this happens because binary format stores only
>> ordinal
 and
>>> the
 only way to run such query is to do something like this:
 
 SqlFieldsQuery qry = new SqlFieldsQuery("select * from Table
>> where
 enumField = ?");
 qry.setArgs(MyEnum.ENUM_VALUE);
 cache.query(qry);
 
 This means that this query can be executed only if:
 - IgniteCache API is used. So it doesn't work in JDBC driver,
> Zeppelin,
 etc.
 - The client that executes the query has the enum class on
 classpath.
 
 My first thought about fixing this is to somehow store value
>> names
 in
 binary metadata and patch the SQL engine so that it understands
>>> how
 to
>>> use
 this information.
 
 Thoughts?
 
 -Val
>> 

Re: IGNITE-2294 implementation details

[Alexander Paschenko]

Sergi,

I've made changes to the API according to your valuable
recommendations, thank you very much for giving them. Please refer to
PR to see current state of the work.
Will surely look into ODBC, .NET and Visor. Though they will most
likely have to support a new feature rather than considerably change
existing logic.

- Alex

2016-07-27 14:23 GMT+03:00 Sergi Vladykin :
> Please don't forget about ODBC, .NET and Visor. They all have to work in
> the same way.
>
> Sergi
>
> 2016-07-27 14:15 GMT+03:00 Alexander Paschenko <
> alexander.a.pasche...@gmail.com>:
>
>> OK, I've found that bold cast to QueryCursor in IgniteCacheProxy
>> and had a look at how SqlFieldsQuery is used in JDBC driver. Thanks.
>>
>> - Alex
>>
>> 2016-07-27 13:02 GMT+03:00 Sergi Vladykin :
>> > Where did you see R in SqlFieldsQuery?
>> >
>> > Sergi
>> >
>> > 2016-07-27 12:59 GMT+03:00 Alexander Paschenko <
>> > alexander.a.pasche...@gmail.com>:
>> >
>> >> Sergi,
>> >>
>> >> But current signature of query() method returns not just some
>> >> iterator, but rather iterator of R which is type param of Query -
>> >> i.e., we won't be able to return an int inside a QueryCursor. At
>> >> least without API change (signature of query() method will have to be
>> >> changed to drop genericness, or in some other weird way). Is this what
>> >> we really want? Or am I missing something in your point?
>> >>
>> >> - Alex
>> >>
>> >> 2016-07-27 12:51 GMT+03:00 Sergi Vladykin :
>> >> > Exactly. This will allow our Jdbc driver to work transparently.
>> >> >
>> >> > Sergi
>> >> >
>> >> > 2016-07-27 12:40 GMT+03:00 Alexander Paschenko <
>> >> > alexander.a.pasche...@gmail.com>:
>> >> >
>> >> >> Sergi,
>> >> >>
>> >> >> You wrote:
>> >> >> > I'd prefer to return the same information, so it will not be empty
>> >> >>
>> >> >> Do you mean return iterator with single element that denotes number
>> of
>> >> >> rows?
>> >> >>
>> >> >> Dmitriy,
>> >> >>
>> >> >> You wrote:
>> >> >> > What is the ticket number for this. Is the new API documented
>> there?
>> >> >>
>> >> >> Overall issue number is 2294. There's no particular issue on API
>> >> >> changes, but creating one seems to be a good idea, I will do it.
>> >> >>
>> >> >> - Alex
>> >> >>
>> >> >> 2016-07-27 9:20 GMT+03:00 Dmitriy Setrakyan :
>> >> >> > What is the ticket number for this. Is the new API documented
>> there?
>> >> >> >
>> >> >> > On Tue, Jul 26, 2016 at 11:36 AM, Sergi Vladykin <
>> >> >> sergi.vlady...@gmail.com>
>> >> >> > wrote:
>> >> >> >
>> >> >> >> I don't see anything ugly in empty iterator, sorry if I insulted
>> your
>> >> >> taste
>> >> >> >> of beauty.
>> >> >> >>
>> >> >> >> If you will take a look at Jdbc, you will see that
>> >> >> Statement.executeUpdate
>> >> >> >> method returns number of updated rows, I'd prefer to return the
>> same
>> >> >> >> information, so it will not be empty (beauty is restored!).
>> >> >> >>
>> >> >> >> Sergi
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> 2016-07-26 18:24 GMT+03:00 Alexander Paschenko <
>> >> >> >> alexander.a.pasche...@gmail.com>:
>> >> >> >>
>> >> >> >> > I see your point. But what about my concerns from initial post?
>> >> >> >> > Particularly about signatures of existing methods? I personally
>> >> don't
>> >> >> >> > like an option of query() method always returning an empty
>> iterator
>> >> >> >> > for any non-select query, it seems ugly design wise.
>> >> >> >> >
>> >> >> >> > - Alex
>> >> >> >> >
>> >> >> >> > 2016-07-26 18:15 GMT+03:00 Sergi Vladykin <
>> >> sergi.vlady...@gmail.com>:
>> >> >> >> > > BTW, the simplest way to solve this issue is to allow running
>> SQL
>> >> >> >> > commands
>> >> >> >> > > inside of SqlFieldsQuery.
>> >> >> >> > >
>> >> >> >> > > We may add some additional convenience API for updates if we
>> >> want,
>> >> >> but
>> >> >> >> > JDBC
>> >> >> >> > > client will always call it like this:
>> >> >> >> > >
>> >> >> >> > > cache.query(new SqlFieldsQuery("INSERT INTO MY_TABLE
>> >> >> >> > > VALUES(?,?)").setArgs(1,2));
>> >> >> >> > >
>> >> >> >> > > This will resolve any ambiguity.
>> >> >> >> > >
>> >> >> >> > > Sergi
>> >> >> >> > >
>> >> >> >> > > 2016-07-26 17:56 GMT+03:00 Sergi Vladykin <
>> >> sergi.vlady...@gmail.com
>> >> >> >:
>> >> >> >> > >
>> >> >> >> > >> I don't like any pre-parsing, especially with some libraries
>> >> other
>> >> >> >> than
>> >> >> >> > >> H2. H2 itself has enough quirks to multiply it on quirks of
>> >> another
>> >> >> >> > library.
>> >> >> >> > >>
>> >> >> >> > >> This is exactly what I was talking about - we need some
>> single
>> >> >> entry
>> >> >> >> > point
>> >> >> >> > >> on API for all the SQL commands and queries. Thats why I
>> >> suggested
>> >> >> >> > >> SqlUpdate to extend Query. To me its is the cleanest
>> approach.
>> >> May
>> >> >> be
>> >> >> >> we
>> >> >> >> > >> need to change in some backward compatible way this Query
>> >> >> hierarchy to
>> >> >> >> > get
>> >> >> >> > >> rid of extra methods but the idea is still the same.
>> >> >> 

Re: Ignite 2.0 tasks/roadmap

[Dmitriy Setrakyan]

On Wed, Jul 27, 2016 at 11:36 AM, Yakov Zhdanov  wrote:

> Guys, I think we can also split event notification for user listeners and
> internal system listeners. I have been seeing a lot of issues caused by
> some heavy or blocking operations in user-defined listeners. This may block
> internal component notification (e.g. on discovery event) causing topology
> hangings.
>

Sure. There are a lot of features being added. Would be nice to assign a
release manager for Ignite 2.0 and document all the discussed features on
the Wiki.


>
> --Yakov
>
> 2016-06-25 2:42 GMT+03:00 Alexey Goncharuk :
>
> > Folks,
> >
> > Recently I have seen a couple of emails suggesting tasks/improvements
> that
> > we cannot do in 1.x releases due to API compatibility reasons, so they
> are
> > postponed to 2.0. I would like to keep track of these tasks in some way
> in
> > our Jira to make sure we do not have anything obsolete when it comes to
> the
> > next major version release.
> >
> > My question for now is how should we track such tasks? Should it be a
> > label, a parent task with subtasks, something else?
> >
> > I would go with a label + release version.
> >
> > --AG
> >
>

[GitHub] ignite pull request #903: IGNITE-3593 .NET: Fix IgniteConfiguration.WorkDire...

[ptupitsyn]

GitHub user ptupitsyn opened a pull request:

https://github.com/apache/ignite/pull/903

IGNITE-3593 .NET: Fix IgniteConfiguration.WorkDirectory having no effect



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ptupitsyn/ignite ignite-3593

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/903.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #903






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Ignite 2.0 tasks/roadmap

[Yakov Zhdanov]

Guys, I think we can also split event notification for user listeners and
internal system listeners. I have been seeing a lot of issues caused by
some heavy or blocking operations in user-defined listeners. This may block
internal component notification (e.g. on discovery event) causing topology
hangings.

--Yakov

2016-06-25 2:42 GMT+03:00 Alexey Goncharuk :

> Folks,
>
> Recently I have seen a couple of emails suggesting tasks/improvements that
> we cannot do in 1.x releases due to API compatibility reasons, so they are
> postponed to 2.0. I would like to keep track of these tasks in some way in
> our Jira to make sure we do not have anything obsolete when it comes to the
> next major version release.
>
> My question for now is how should we track such tasks? Should it be a
> label, a parent task with subtasks, something else?
>
> I would go with a label + release version.
>
> --AG
>

[GitHub] ignite pull request #902: GG-11360 Merged IGNITE-2680 to ignite-1.6.3.

[ascherbakoff]

GitHub user ascherbakoff opened a pull request:

https://github.com/apache/ignite/pull/902

GG-11360 Merged IGNITE-2680 to ignite-1.6.3.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ascherbakoff/ignite ignite-gg-11360

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/902.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #902


commit 8c21f27827ab193b1d06b31419401be87fb0a11b
Author: Aleksei Scherbakov 
Date:   2016-07-26T17:11:01Z

GG-11360 Merged IGNITE-2680 to ignite-1.6.3.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Enums and SQL queries

[Sergi Vladykin]

I don't see how this can work to file a ticket, could you please give more
details?

Sergi

2016-07-27 15:50 GMT+03:00 Dmitriy Setrakyan :

> Sergi, I still thinks my solution works or can be made to work. In the
> example you provided, you can choose to return either String or Enum as the
> query result. However, this is not a clean solution, I agree, but could be
> a quick one.
>
> I don't mind breaking compatibility on this issue, given that it never
> worked properly anyway. Can you please file a ticket?
>
> D.
>
> On Wed, Jul 27, 2016 at 3:01 AM, Sergi Vladykin 
> wrote:
>
> > Nope. This will not work. Because take for example the following query:
> >
> > select my_enum_field from my_table where my_enum_field = 'BLA'
> >
> > Field well be evaluated only once here, but it must be a Sting and Enum
> at
> > the same time.
>
>
> > Probably we could contribute to H2 `User defined value types` and handle
> > this stuff in our own Enum value types, which will support comparison
> with
> > String. But this requires bug H2 Modification and is a compatibility
> > breaking change.
> >
> > Sergi
> >
> >
> >
> > 2016-07-27 9:23 GMT+03:00 Dmitriy Setrakyan :
> >
> > > Hm... I would prefer to find a way to handle Enums automatically.
> > >
> > > In case if user expects a String, can we catch an exception and do
> > > automatic conversion at that time? In this case, we could catch the
> > > exception once and remember the decision. Will something like this
> work?
> > >
> > > D.
> > >
> > > On Tue, Jul 26, 2016 at 1:49 AM, Sergi Vladykin <
> > sergi.vlady...@gmail.com>
> > > wrote:
> > >
> > > > GridQueryProcessor contains two types: BinaryProperty and
> > ClassProperty.
> > > > Indexing uses them for fields access. Probably we can change them, so
> > > that
> > > > they will always return String instead of Enum type but this
> > > >
> > > > 1. looks like a compatibility breaking change;
> > > > 2. users doing queries like `select my_enum_field from my_table`
> > probably
> > > > still expect to get a Enum result but not a String.
> > > >
> > > > What do you think?
> > > >
> > > > Sergi
> > > >
> > > >
> > > > 2016-07-26 3:17 GMT+03:00 Valentin Kulichenko <
> > > > valentin.kuliche...@gmail.com
> > > > >:
> > > >
> > > > > Sergi,
> > > > >
> > > > > Ideally, I would like this query to work:
> > > > >
> > > > > select * from MyTable where myEnumField = 'MY_ENUM_NAME'
> > > > >
> > > > > The problem is that the field value is not a string, it's a binary
> > > object
> > > > > with the enum value ordinal inside. We can add enum value names
> into
> > > the
> > > > > metadata, so that we can always map the field value to its string
> > > > > representation, but SQL engine will have to know how to use it. Is
> > this
> > > > > possible?
> > > > >
> > > > > -Val
> > > > >
> > > > > On Fri, Jul 22, 2016 at 10:25 PM, Sergi Vladykin <
> > > > sergi.vlady...@gmail.com
> > > > > > wrote:
> > > > >
> > > > >> I'm not sure what exactly you are going to patch in SQL engine. We
> > > could
> > > > >> and some SQL function like enum('MY_ENUM_NAME') but probably it
> will
> > > not
> > > > >> work if we will not pass enum type there as well, so it have look
> > like
> > > > >> enum('com.mycompany.MyEnum', 'MY_ENUM_NAME') which is ugly.
> > > > >>
> > > > >> Sergi
> > > > >>
> > > > >> On 23 июля 2016 г., 0:43, Valentin Kulichenko <
> > > > >> valentin.kuliche...@gmail.com
> > > > >> > wrote:
> > > > >>
> > > > >> > Folks,
> > > > >> >
> > > > >> > I noticed that we currently have a very weird limitation for
> enum
> > > > fields
> > > > >> > when running SQL queries. Basically, you can't use enum value
> > names
> > > as
> > > > >> many
> > > > >> > users would expect.
> > > > >> >
> > > > >> > Technically this happens because binary format stores only
> ordinal
> > > and
> > > > >> the
> > > > >> > only way to run such query is to do something like this:
> > > > >> >
> > > > >> > SqlFieldsQuery qry = new SqlFieldsQuery("select * from Table
> where
> > > > >> > enumField = ?");
> > > > >> > qry.setArgs(MyEnum.ENUM_VALUE);
> > > > >> > cache.query(qry);
> > > > >> >
> > > > >> > This means that this query can be executed only if:
> > > > >> > - IgniteCache API is used. So it doesn't work in JDBC driver,
> > > > Zeppelin,
> > > > >> > etc.
> > > > >> > - The client that executes the query has the enum class on
> > > classpath.
> > > > >> >
> > > > >> > My first thought about fixing this is to somehow store value
> names
> > > in
> > > > >> > binary metadata and patch the SQL engine so that it understands
> > how
> > > to
> > > > >> use
> > > > >> > this information.
> > > > >> >
> > > > >> > Thoughts?
> > > > >> >
> > > > >> > -Val
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: Enums and SQL queries

[Dmitriy Setrakyan]

Sergi, I still thinks my solution works or can be made to work. In the
example you provided, you can choose to return either String or Enum as the
query result. However, this is not a clean solution, I agree, but could be
a quick one.

I don't mind breaking compatibility on this issue, given that it never
worked properly anyway. Can you please file a ticket?

D.

On Wed, Jul 27, 2016 at 3:01 AM, Sergi Vladykin 
wrote:

> Nope. This will not work. Because take for example the following query:
>
> select my_enum_field from my_table where my_enum_field = 'BLA'
>
> Field well be evaluated only once here, but it must be a Sting and Enum at
> the same time.


> Probably we could contribute to H2 `User defined value types` and handle
> this stuff in our own Enum value types, which will support comparison with
> String. But this requires bug H2 Modification and is a compatibility
> breaking change.
>
> Sergi
>
>
>
> 2016-07-27 9:23 GMT+03:00 Dmitriy Setrakyan :
>
> > Hm... I would prefer to find a way to handle Enums automatically.
> >
> > In case if user expects a String, can we catch an exception and do
> > automatic conversion at that time? In this case, we could catch the
> > exception once and remember the decision. Will something like this work?
> >
> > D.
> >
> > On Tue, Jul 26, 2016 at 1:49 AM, Sergi Vladykin <
> sergi.vlady...@gmail.com>
> > wrote:
> >
> > > GridQueryProcessor contains two types: BinaryProperty and
> ClassProperty.
> > > Indexing uses them for fields access. Probably we can change them, so
> > that
> > > they will always return String instead of Enum type but this
> > >
> > > 1. looks like a compatibility breaking change;
> > > 2. users doing queries like `select my_enum_field from my_table`
> probably
> > > still expect to get a Enum result but not a String.
> > >
> > > What do you think?
> > >
> > > Sergi
> > >
> > >
> > > 2016-07-26 3:17 GMT+03:00 Valentin Kulichenko <
> > > valentin.kuliche...@gmail.com
> > > >:
> > >
> > > > Sergi,
> > > >
> > > > Ideally, I would like this query to work:
> > > >
> > > > select * from MyTable where myEnumField = 'MY_ENUM_NAME'
> > > >
> > > > The problem is that the field value is not a string, it's a binary
> > object
> > > > with the enum value ordinal inside. We can add enum value names into
> > the
> > > > metadata, so that we can always map the field value to its string
> > > > representation, but SQL engine will have to know how to use it. Is
> this
> > > > possible?
> > > >
> > > > -Val
> > > >
> > > > On Fri, Jul 22, 2016 at 10:25 PM, Sergi Vladykin <
> > > sergi.vlady...@gmail.com
> > > > > wrote:
> > > >
> > > >> I'm not sure what exactly you are going to patch in SQL engine. We
> > could
> > > >> and some SQL function like enum('MY_ENUM_NAME') but probably it will
> > not
> > > >> work if we will not pass enum type there as well, so it have look
> like
> > > >> enum('com.mycompany.MyEnum', 'MY_ENUM_NAME') which is ugly.
> > > >>
> > > >> Sergi
> > > >>
> > > >> On 23 июля 2016 г., 0:43, Valentin Kulichenko <
> > > >> valentin.kuliche...@gmail.com
> > > >> > wrote:
> > > >>
> > > >> > Folks,
> > > >> >
> > > >> > I noticed that we currently have a very weird limitation for enum
> > > fields
> > > >> > when running SQL queries. Basically, you can't use enum value
> names
> > as
> > > >> many
> > > >> > users would expect.
> > > >> >
> > > >> > Technically this happens because binary format stores only ordinal
> > and
> > > >> the
> > > >> > only way to run such query is to do something like this:
> > > >> >
> > > >> > SqlFieldsQuery qry = new SqlFieldsQuery("select * from Table where
> > > >> > enumField = ?");
> > > >> > qry.setArgs(MyEnum.ENUM_VALUE);
> > > >> > cache.query(qry);
> > > >> >
> > > >> > This means that this query can be executed only if:
> > > >> > - IgniteCache API is used. So it doesn't work in JDBC driver,
> > > Zeppelin,
> > > >> > etc.
> > > >> > - The client that executes the query has the enum class on
> > classpath.
> > > >> >
> > > >> > My first thought about fixing this is to somehow store value names
> > in
> > > >> > binary metadata and patch the SQL engine so that it understands
> how
> > to
> > > >> use
> > > >> > this information.
> > > >> >
> > > >> > Thoughts?
> > > >> >
> > > >> > -Val
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

[jira] [Created] (IGNITE-3593) .NET: IgniteConfiguration.WorkDirectory has no effect

[Pavel Tupitsyn (JIRA)]

Pavel Tupitsyn created IGNITE-3593:
--

 Summary: .NET: IgniteConfiguration.WorkDirectory has no effect
 Key: IGNITE-3593
 URL: https://issues.apache.org/jira/browse/IGNITE-3593
 Project: Ignite
  Issue Type: Bug
  Components: platforms
Affects Versions: 1.6
Reporter: Pavel Tupitsyn
Assignee: Pavel Tupitsyn
 Fix For: 1.7


Temporary marshaller is created during startup, it sets the work dir, and it 
can't be changed after that (static state).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Data compression in Ignite 2.0

[Alexey Kuznetsov]

FYI, I created issue for Ignite 2.0:
https://issues.apache.org/jira/browse/IGNITE-3592

Thanks!

On Wed, Jul 27, 2016 at 2:36 PM, Sergi Vladykin 
wrote:

> Nikita,
>
> I agree with Andrey, HANA is a bad comparison to Ignite in this respect. I
> did not find any evidence on the internet that their row store is very
> efficient with compression. It was always about column store.
>
> Alexey,
>
> As for DB2, can you check what exactly, when and how it compresses and does
> it give any decent results before suggesting it as an example to follow? I
> don't think it is good idea to repeat every bad idea after other products.
>
> And even if there are good results in DB2, will this all be applicable to
> Ignite? PostgreSql for example provides TOAST compression and this can be
> useful when used in a smart way, but this is a very different architecture
> from what we have.
>
> All in all I agree that may be we should provide some kind of pluggable
> compression SPI support, but do not expect much from it, usually it will be
> just useless.
>
> Sergi
>
>
>
> 2016-07-27 10:16 GMT+03:00 Sebastien DIAZ :
>
> > Hi
> >
> > I add Redis as a sample of memory compression strategy
> >
> > http://labs.octivi.com/how-we-cut-down-memory-usage-by-82/
> >
> > http://redis.io/topics/memory-optimization
> >
> > Regards
> >
> > S DIAZ
> >
> >
> >
> > 2016-07-27 8:24 GMT+02:00 Alexey Kuznetsov :
> >
> > > Nikita,
> > >
> > > That was my intention: "we may need to provide a better facility to
> > inject
> > > user's logic here..."
> > >
> > > Andrey,
> > > About compression, once again - DB2 is a row-based DB and they can
> > compress
> > > :)
> > >
> > > On Wed, Jul 27, 2016 at 12:56 PM, Nikita Ivanov 
> > > wrote:
> > >
> > > > Very good points indeed. I get the compression in Ignite question
> quite
> > > > often and Hana reference is a typical lead in.
> > > >
> > > > My personal opinion is still that in Ignite *specifically* the
> > > compression
> > > > is best left to the end-user. But we may need to provide a better
> > > facility
> > > > to inject user's logic here...
> > > >
> > > > --
> > > > Nikita Ivanov
> > > >
> > > >
> > > > On Tue, Jul 26, 2016 at 9:53 PM, Andrey Kornev <
> > andrewkor...@hotmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Dictionary compression requires some knowledge about data being
> > > > > compressed. For example, for numeric types a range of values must
> be
> > > > known
> > > > > so that the dictionary can be generated. For strings, the number of
> > > > unique
> > > > > values of the column is the key piece of input into the dictionary
> > > > > generation.
> > > > > SAP HANA is a column-based database system: it stores the fields of
> > the
> > > > > data tuple individually using the best compression for the given
> data
> > > > type
> > > > > and the particular set of values. HANA has been specifically built
> > as a
> > > > > general purpose database, rather than as an afterthought layer on
> top
> > > of
> > > > an
> > > > > already existing distributed cache.
> > > > > On the other hand, Ignite is a distributed cache implementation (a
> > > pretty
> > > > > good one!) that in general requires no schema and stores its data
> in
> > > the
> > > > > row-based fashion. Its current design doesn't land itself readily
> to
> > > the
> > > > > kind of optimizations HANA provides out of the box.
> > > > > For the curios types among us, the implementation details of HANA
> are
> > > > well
> > > > > documented in "In-memory Data Management", by Hasso Plattner &
> > > Alexander
> > > > > Zeier.
> > > > > Cheers
> > > > > Andrey
> > > > > _
> > > > > From: Alexey Kuznetsov  > > > > akuznet...@gridgain.com>>
> > > > > Sent: Tuesday, July 26, 2016 5:36 AM
> > > > > Subject: Re: Data compression in Ignite 2.0
> > > > > To: mailto:dev@ignite.apache.org>>
> > > > >
> > > > >
> > > > > Sergey Kozlov wrote:
> > > > > >> For approach 1: Put a large object into a partition cache will
> > > > > force to update
> > > > > the dictionary placed on replication cache. It may be time-expense
> > > > > operation.
> > > > > The dictionary will be built only once. And we could control what
> > > should
> > > > be
> > > > > put into dictionary, for example, we could check min and max size
> and
> > > > > decide - put value to dictionary or not.
> > > > >
> > > > > >> Approach 2-3 are make sense for rare cases as Sergi commented.
> > > > > But it is better at least have a possibility to plug user code for
> > > > > compression than not to have it at all.
> > > > >
> > > > > >> Also I see a danger of OOM if we've got high compression level
> and
> > > try
> > > > > to restore original value in memory.
> > > > > We could easily get OOM with many other operations right now
> without
> > > > > compression, I think it is not an issue, we could add a NOTE to
> > > > > documentation about such possibility.
> > > > >
> > > > > Andrey Kornev wrote:
> > > > > >> ... in general I think compression is

[jira] [Created] (IGNITE-3592) Provide some kind of pluggable compression SPI support

[Alexey Kuznetsov (JIRA)]

Alexey Kuznetsov created IGNITE-3592:


 Summary: Provide some kind of pluggable compression SPI support
 Key: IGNITE-3592
 URL: https://issues.apache.org/jira/browse/IGNITE-3592
 Project: Ignite
  Issue Type: Task
  Components: cache
Reporter: Alexey Kuznetsov
 Fix For: 2.0


It may be useful in some cases to compress data stored in cache.
And in order to give access to compressed data from SQL engine this support 
should be implemented in ignite-core level.

See discussion on dev-list: 
http://apache-ignite-developers.2346864.n4.nabble.com/Data-compression-in-Ignite-2-0-td10099.html




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: IGNITE-2294 implementation details

[Sergi Vladykin]

Please don't forget about ODBC, .NET and Visor. They all have to work in
the same way.

Sergi

2016-07-27 14:15 GMT+03:00 Alexander Paschenko <
alexander.a.pasche...@gmail.com>:

> OK, I've found that bold cast to QueryCursor in IgniteCacheProxy
> and had a look at how SqlFieldsQuery is used in JDBC driver. Thanks.
>
> - Alex
>
> 2016-07-27 13:02 GMT+03:00 Sergi Vladykin :
> > Where did you see R in SqlFieldsQuery?
> >
> > Sergi
> >
> > 2016-07-27 12:59 GMT+03:00 Alexander Paschenko <
> > alexander.a.pasche...@gmail.com>:
> >
> >> Sergi,
> >>
> >> But current signature of query() method returns not just some
> >> iterator, but rather iterator of R which is type param of Query -
> >> i.e., we won't be able to return an int inside a QueryCursor. At
> >> least without API change (signature of query() method will have to be
> >> changed to drop genericness, or in some other weird way). Is this what
> >> we really want? Or am I missing something in your point?
> >>
> >> - Alex
> >>
> >> 2016-07-27 12:51 GMT+03:00 Sergi Vladykin :
> >> > Exactly. This will allow our Jdbc driver to work transparently.
> >> >
> >> > Sergi
> >> >
> >> > 2016-07-27 12:40 GMT+03:00 Alexander Paschenko <
> >> > alexander.a.pasche...@gmail.com>:
> >> >
> >> >> Sergi,
> >> >>
> >> >> You wrote:
> >> >> > I'd prefer to return the same information, so it will not be empty
> >> >>
> >> >> Do you mean return iterator with single element that denotes number
> of
> >> >> rows?
> >> >>
> >> >> Dmitriy,
> >> >>
> >> >> You wrote:
> >> >> > What is the ticket number for this. Is the new API documented
> there?
> >> >>
> >> >> Overall issue number is 2294. There's no particular issue on API
> >> >> changes, but creating one seems to be a good idea, I will do it.
> >> >>
> >> >> - Alex
> >> >>
> >> >> 2016-07-27 9:20 GMT+03:00 Dmitriy Setrakyan :
> >> >> > What is the ticket number for this. Is the new API documented
> there?
> >> >> >
> >> >> > On Tue, Jul 26, 2016 at 11:36 AM, Sergi Vladykin <
> >> >> sergi.vlady...@gmail.com>
> >> >> > wrote:
> >> >> >
> >> >> >> I don't see anything ugly in empty iterator, sorry if I insulted
> your
> >> >> taste
> >> >> >> of beauty.
> >> >> >>
> >> >> >> If you will take a look at Jdbc, you will see that
> >> >> Statement.executeUpdate
> >> >> >> method returns number of updated rows, I'd prefer to return the
> same
> >> >> >> information, so it will not be empty (beauty is restored!).
> >> >> >>
> >> >> >> Sergi
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> 2016-07-26 18:24 GMT+03:00 Alexander Paschenko <
> >> >> >> alexander.a.pasche...@gmail.com>:
> >> >> >>
> >> >> >> > I see your point. But what about my concerns from initial post?
> >> >> >> > Particularly about signatures of existing methods? I personally
> >> don't
> >> >> >> > like an option of query() method always returning an empty
> iterator
> >> >> >> > for any non-select query, it seems ugly design wise.
> >> >> >> >
> >> >> >> > - Alex
> >> >> >> >
> >> >> >> > 2016-07-26 18:15 GMT+03:00 Sergi Vladykin <
> >> sergi.vlady...@gmail.com>:
> >> >> >> > > BTW, the simplest way to solve this issue is to allow running
> SQL
> >> >> >> > commands
> >> >> >> > > inside of SqlFieldsQuery.
> >> >> >> > >
> >> >> >> > > We may add some additional convenience API for updates if we
> >> want,
> >> >> but
> >> >> >> > JDBC
> >> >> >> > > client will always call it like this:
> >> >> >> > >
> >> >> >> > > cache.query(new SqlFieldsQuery("INSERT INTO MY_TABLE
> >> >> >> > > VALUES(?,?)").setArgs(1,2));
> >> >> >> > >
> >> >> >> > > This will resolve any ambiguity.
> >> >> >> > >
> >> >> >> > > Sergi
> >> >> >> > >
> >> >> >> > > 2016-07-26 17:56 GMT+03:00 Sergi Vladykin <
> >> sergi.vlady...@gmail.com
> >> >> >:
> >> >> >> > >
> >> >> >> > >> I don't like any pre-parsing, especially with some libraries
> >> other
> >> >> >> than
> >> >> >> > >> H2. H2 itself has enough quirks to multiply it on quirks of
> >> another
> >> >> >> > library.
> >> >> >> > >>
> >> >> >> > >> This is exactly what I was talking about - we need some
> single
> >> >> entry
> >> >> >> > point
> >> >> >> > >> on API for all the SQL commands and queries. Thats why I
> >> suggested
> >> >> >> > >> SqlUpdate to extend Query. To me its is the cleanest
> approach.
> >> May
> >> >> be
> >> >> >> we
> >> >> >> > >> need to change in some backward compatible way this Query
> >> >> hierarchy to
> >> >> >> > get
> >> >> >> > >> rid of extra methods but the idea is still the same.
> >> >> >> > >>
> >> >> >> > >> Sergi
> >> >> >> > >>
> >> >> >> > >> 2016-07-26 14:34 GMT+03:00 Alexander Paschenko <
> >> >> >> > >> alexander.a.pasche...@gmail.com>:
> >> >> >> > >>
> >> >> >> > >>> Guys,
> >> >> >> > >>>
> >> >> >> > >>> I would like to advance the discussion further. There's one
> >> quite
> >> >> >> > >>> important question that arose based on current state of
> work on
> >> >> this
> >> >> >> > >>> issue. If we use some kind of interactive console, like
> Visor,
> >> >> then
> >> >> >> > >>> how shoul

Re: IGNITE-2294 implementation details

[Alexander Paschenko]

OK, I've found that bold cast to QueryCursor in IgniteCacheProxy
and had a look at how SqlFieldsQuery is used in JDBC driver. Thanks.

- Alex

2016-07-27 13:02 GMT+03:00 Sergi Vladykin :
> Where did you see R in SqlFieldsQuery?
>
> Sergi
>
> 2016-07-27 12:59 GMT+03:00 Alexander Paschenko <
> alexander.a.pasche...@gmail.com>:
>
>> Sergi,
>>
>> But current signature of query() method returns not just some
>> iterator, but rather iterator of R which is type param of Query -
>> i.e., we won't be able to return an int inside a QueryCursor. At
>> least without API change (signature of query() method will have to be
>> changed to drop genericness, or in some other weird way). Is this what
>> we really want? Or am I missing something in your point?
>>
>> - Alex
>>
>> 2016-07-27 12:51 GMT+03:00 Sergi Vladykin :
>> > Exactly. This will allow our Jdbc driver to work transparently.
>> >
>> > Sergi
>> >
>> > 2016-07-27 12:40 GMT+03:00 Alexander Paschenko <
>> > alexander.a.pasche...@gmail.com>:
>> >
>> >> Sergi,
>> >>
>> >> You wrote:
>> >> > I'd prefer to return the same information, so it will not be empty
>> >>
>> >> Do you mean return iterator with single element that denotes number of
>> >> rows?
>> >>
>> >> Dmitriy,
>> >>
>> >> You wrote:
>> >> > What is the ticket number for this. Is the new API documented there?
>> >>
>> >> Overall issue number is 2294. There's no particular issue on API
>> >> changes, but creating one seems to be a good idea, I will do it.
>> >>
>> >> - Alex
>> >>
>> >> 2016-07-27 9:20 GMT+03:00 Dmitriy Setrakyan :
>> >> > What is the ticket number for this. Is the new API documented there?
>> >> >
>> >> > On Tue, Jul 26, 2016 at 11:36 AM, Sergi Vladykin <
>> >> sergi.vlady...@gmail.com>
>> >> > wrote:
>> >> >
>> >> >> I don't see anything ugly in empty iterator, sorry if I insulted your
>> >> taste
>> >> >> of beauty.
>> >> >>
>> >> >> If you will take a look at Jdbc, you will see that
>> >> Statement.executeUpdate
>> >> >> method returns number of updated rows, I'd prefer to return the same
>> >> >> information, so it will not be empty (beauty is restored!).
>> >> >>
>> >> >> Sergi
>> >> >>
>> >> >>
>> >> >>
>> >> >> 2016-07-26 18:24 GMT+03:00 Alexander Paschenko <
>> >> >> alexander.a.pasche...@gmail.com>:
>> >> >>
>> >> >> > I see your point. But what about my concerns from initial post?
>> >> >> > Particularly about signatures of existing methods? I personally
>> don't
>> >> >> > like an option of query() method always returning an empty iterator
>> >> >> > for any non-select query, it seems ugly design wise.
>> >> >> >
>> >> >> > - Alex
>> >> >> >
>> >> >> > 2016-07-26 18:15 GMT+03:00 Sergi Vladykin <
>> sergi.vlady...@gmail.com>:
>> >> >> > > BTW, the simplest way to solve this issue is to allow running SQL
>> >> >> > commands
>> >> >> > > inside of SqlFieldsQuery.
>> >> >> > >
>> >> >> > > We may add some additional convenience API for updates if we
>> want,
>> >> but
>> >> >> > JDBC
>> >> >> > > client will always call it like this:
>> >> >> > >
>> >> >> > > cache.query(new SqlFieldsQuery("INSERT INTO MY_TABLE
>> >> >> > > VALUES(?,?)").setArgs(1,2));
>> >> >> > >
>> >> >> > > This will resolve any ambiguity.
>> >> >> > >
>> >> >> > > Sergi
>> >> >> > >
>> >> >> > > 2016-07-26 17:56 GMT+03:00 Sergi Vladykin <
>> sergi.vlady...@gmail.com
>> >> >:
>> >> >> > >
>> >> >> > >> I don't like any pre-parsing, especially with some libraries
>> other
>> >> >> than
>> >> >> > >> H2. H2 itself has enough quirks to multiply it on quirks of
>> another
>> >> >> > library.
>> >> >> > >>
>> >> >> > >> This is exactly what I was talking about - we need some single
>> >> entry
>> >> >> > point
>> >> >> > >> on API for all the SQL commands and queries. Thats why I
>> suggested
>> >> >> > >> SqlUpdate to extend Query. To me its is the cleanest approach.
>> May
>> >> be
>> >> >> we
>> >> >> > >> need to change in some backward compatible way this Query
>> >> hierarchy to
>> >> >> > get
>> >> >> > >> rid of extra methods but the idea is still the same.
>> >> >> > >>
>> >> >> > >> Sergi
>> >> >> > >>
>> >> >> > >> 2016-07-26 14:34 GMT+03:00 Alexander Paschenko <
>> >> >> > >> alexander.a.pasche...@gmail.com>:
>> >> >> > >>
>> >> >> > >>> Guys,
>> >> >> > >>>
>> >> >> > >>> I would like to advance the discussion further. There's one
>> quite
>> >> >> > >>> important question that arose based on current state of work on
>> >> this
>> >> >> > >>> issue. If we use some kind of interactive console, like Visor,
>> >> then
>> >> >> > >>> how should it know whether SQL query it is requested to execute
>> >> >> > >>> returns a result set or not? In JDBC world, solution is quite
>> >> simple
>> >> >> -
>> >> >> > >>> there's base interface called Statement that all commands
>> >> implement,
>> >> >> > >>> and it has magic isResultSet method that tells whether
>> statement
>> >> is a
>> >> >> > >>> query or an update command. The API proposed now has separate
>> >> Query
>> >> >> > >>> and Update operations which I bel

[GitHub] ignite pull request #901: IGNITE-1084 fixed issue broken HibernateL2CacheSel...

[mwadhwa]

GitHub user mwadhwa opened a pull request:

https://github.com/apache/ignite/pull/901

IGNITE-1084 fixed issue  broken HibernateL2CacheSelfTest#testNaturalI…

…dCache()

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mwadhwa/ignite master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/901.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #901


commit 2a1be7d6925390081d8aa674b516381e4c5a0b42
Author: milap.wadhwa 
Date:   2016-07-27T11:11:55Z

IGNITE-1084 fixed issue  broken 
HibernateL2CacheSelfTest#testNaturalIdCache()




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #900: IGNITE-3561 .NET: DistributedJoins property in Sql...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/900


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: IGNITE-2294 implementation details

[Sergi Vladykin]

Where did you see R in SqlFieldsQuery?

Sergi

2016-07-27 12:59 GMT+03:00 Alexander Paschenko <
alexander.a.pasche...@gmail.com>:

> Sergi,
>
> But current signature of query() method returns not just some
> iterator, but rather iterator of R which is type param of Query -
> i.e., we won't be able to return an int inside a QueryCursor. At
> least without API change (signature of query() method will have to be
> changed to drop genericness, or in some other weird way). Is this what
> we really want? Or am I missing something in your point?
>
> - Alex
>
> 2016-07-27 12:51 GMT+03:00 Sergi Vladykin :
> > Exactly. This will allow our Jdbc driver to work transparently.
> >
> > Sergi
> >
> > 2016-07-27 12:40 GMT+03:00 Alexander Paschenko <
> > alexander.a.pasche...@gmail.com>:
> >
> >> Sergi,
> >>
> >> You wrote:
> >> > I'd prefer to return the same information, so it will not be empty
> >>
> >> Do you mean return iterator with single element that denotes number of
> >> rows?
> >>
> >> Dmitriy,
> >>
> >> You wrote:
> >> > What is the ticket number for this. Is the new API documented there?
> >>
> >> Overall issue number is 2294. There's no particular issue on API
> >> changes, but creating one seems to be a good idea, I will do it.
> >>
> >> - Alex
> >>
> >> 2016-07-27 9:20 GMT+03:00 Dmitriy Setrakyan :
> >> > What is the ticket number for this. Is the new API documented there?
> >> >
> >> > On Tue, Jul 26, 2016 at 11:36 AM, Sergi Vladykin <
> >> sergi.vlady...@gmail.com>
> >> > wrote:
> >> >
> >> >> I don't see anything ugly in empty iterator, sorry if I insulted your
> >> taste
> >> >> of beauty.
> >> >>
> >> >> If you will take a look at Jdbc, you will see that
> >> Statement.executeUpdate
> >> >> method returns number of updated rows, I'd prefer to return the same
> >> >> information, so it will not be empty (beauty is restored!).
> >> >>
> >> >> Sergi
> >> >>
> >> >>
> >> >>
> >> >> 2016-07-26 18:24 GMT+03:00 Alexander Paschenko <
> >> >> alexander.a.pasche...@gmail.com>:
> >> >>
> >> >> > I see your point. But what about my concerns from initial post?
> >> >> > Particularly about signatures of existing methods? I personally
> don't
> >> >> > like an option of query() method always returning an empty iterator
> >> >> > for any non-select query, it seems ugly design wise.
> >> >> >
> >> >> > - Alex
> >> >> >
> >> >> > 2016-07-26 18:15 GMT+03:00 Sergi Vladykin <
> sergi.vlady...@gmail.com>:
> >> >> > > BTW, the simplest way to solve this issue is to allow running SQL
> >> >> > commands
> >> >> > > inside of SqlFieldsQuery.
> >> >> > >
> >> >> > > We may add some additional convenience API for updates if we
> want,
> >> but
> >> >> > JDBC
> >> >> > > client will always call it like this:
> >> >> > >
> >> >> > > cache.query(new SqlFieldsQuery("INSERT INTO MY_TABLE
> >> >> > > VALUES(?,?)").setArgs(1,2));
> >> >> > >
> >> >> > > This will resolve any ambiguity.
> >> >> > >
> >> >> > > Sergi
> >> >> > >
> >> >> > > 2016-07-26 17:56 GMT+03:00 Sergi Vladykin <
> sergi.vlady...@gmail.com
> >> >:
> >> >> > >
> >> >> > >> I don't like any pre-parsing, especially with some libraries
> other
> >> >> than
> >> >> > >> H2. H2 itself has enough quirks to multiply it on quirks of
> another
> >> >> > library.
> >> >> > >>
> >> >> > >> This is exactly what I was talking about - we need some single
> >> entry
> >> >> > point
> >> >> > >> on API for all the SQL commands and queries. Thats why I
> suggested
> >> >> > >> SqlUpdate to extend Query. To me its is the cleanest approach.
> May
> >> be
> >> >> we
> >> >> > >> need to change in some backward compatible way this Query
> >> hierarchy to
> >> >> > get
> >> >> > >> rid of extra methods but the idea is still the same.
> >> >> > >>
> >> >> > >> Sergi
> >> >> > >>
> >> >> > >> 2016-07-26 14:34 GMT+03:00 Alexander Paschenko <
> >> >> > >> alexander.a.pasche...@gmail.com>:
> >> >> > >>
> >> >> > >>> Guys,
> >> >> > >>>
> >> >> > >>> I would like to advance the discussion further. There's one
> quite
> >> >> > >>> important question that arose based on current state of work on
> >> this
> >> >> > >>> issue. If we use some kind of interactive console, like Visor,
> >> then
> >> >> > >>> how should it know whether SQL query it is requested to execute
> >> >> > >>> returns a result set or not? In JDBC world, solution is quite
> >> simple
> >> >> -
> >> >> > >>> there's base interface called Statement that all commands
> >> implement,
> >> >> > >>> and it has magic isResultSet method that tells whether
> statement
> >> is a
> >> >> > >>> query or an update command. The API proposed now has separate
> >> Query
> >> >> > >>> and Update operations which I believe to be a right thing by
> the
> >> >> > >>> reasons I outlined in the beginning of this thread. However,
> their
> >> >> > >>> lack of common ancestor prevents possible console clients from
> >> >> running
> >> >> > >>> text SQL commands in a fully transparent manner - like
> >> >> > >>> IgniteCache.execute(String sql). Therefore

Re: IGNITE-2294 implementation details

[Alexander Paschenko]

Sergi,

But current signature of query() method returns not just some
iterator, but rather iterator of R which is type param of Query -
i.e., we won't be able to return an int inside a QueryCursor. At
least without API change (signature of query() method will have to be
changed to drop genericness, or in some other weird way). Is this what
we really want? Or am I missing something in your point?

- Alex

2016-07-27 12:51 GMT+03:00 Sergi Vladykin :
> Exactly. This will allow our Jdbc driver to work transparently.
>
> Sergi
>
> 2016-07-27 12:40 GMT+03:00 Alexander Paschenko <
> alexander.a.pasche...@gmail.com>:
>
>> Sergi,
>>
>> You wrote:
>> > I'd prefer to return the same information, so it will not be empty
>>
>> Do you mean return iterator with single element that denotes number of
>> rows?
>>
>> Dmitriy,
>>
>> You wrote:
>> > What is the ticket number for this. Is the new API documented there?
>>
>> Overall issue number is 2294. There's no particular issue on API
>> changes, but creating one seems to be a good idea, I will do it.
>>
>> - Alex
>>
>> 2016-07-27 9:20 GMT+03:00 Dmitriy Setrakyan :
>> > What is the ticket number for this. Is the new API documented there?
>> >
>> > On Tue, Jul 26, 2016 at 11:36 AM, Sergi Vladykin <
>> sergi.vlady...@gmail.com>
>> > wrote:
>> >
>> >> I don't see anything ugly in empty iterator, sorry if I insulted your
>> taste
>> >> of beauty.
>> >>
>> >> If you will take a look at Jdbc, you will see that
>> Statement.executeUpdate
>> >> method returns number of updated rows, I'd prefer to return the same
>> >> information, so it will not be empty (beauty is restored!).
>> >>
>> >> Sergi
>> >>
>> >>
>> >>
>> >> 2016-07-26 18:24 GMT+03:00 Alexander Paschenko <
>> >> alexander.a.pasche...@gmail.com>:
>> >>
>> >> > I see your point. But what about my concerns from initial post?
>> >> > Particularly about signatures of existing methods? I personally don't
>> >> > like an option of query() method always returning an empty iterator
>> >> > for any non-select query, it seems ugly design wise.
>> >> >
>> >> > - Alex
>> >> >
>> >> > 2016-07-26 18:15 GMT+03:00 Sergi Vladykin :
>> >> > > BTW, the simplest way to solve this issue is to allow running SQL
>> >> > commands
>> >> > > inside of SqlFieldsQuery.
>> >> > >
>> >> > > We may add some additional convenience API for updates if we want,
>> but
>> >> > JDBC
>> >> > > client will always call it like this:
>> >> > >
>> >> > > cache.query(new SqlFieldsQuery("INSERT INTO MY_TABLE
>> >> > > VALUES(?,?)").setArgs(1,2));
>> >> > >
>> >> > > This will resolve any ambiguity.
>> >> > >
>> >> > > Sergi
>> >> > >
>> >> > > 2016-07-26 17:56 GMT+03:00 Sergi Vladykin > >:
>> >> > >
>> >> > >> I don't like any pre-parsing, especially with some libraries other
>> >> than
>> >> > >> H2. H2 itself has enough quirks to multiply it on quirks of another
>> >> > library.
>> >> > >>
>> >> > >> This is exactly what I was talking about - we need some single
>> entry
>> >> > point
>> >> > >> on API for all the SQL commands and queries. Thats why I suggested
>> >> > >> SqlUpdate to extend Query. To me its is the cleanest approach. May
>> be
>> >> we
>> >> > >> need to change in some backward compatible way this Query
>> hierarchy to
>> >> > get
>> >> > >> rid of extra methods but the idea is still the same.
>> >> > >>
>> >> > >> Sergi
>> >> > >>
>> >> > >> 2016-07-26 14:34 GMT+03:00 Alexander Paschenko <
>> >> > >> alexander.a.pasche...@gmail.com>:
>> >> > >>
>> >> > >>> Guys,
>> >> > >>>
>> >> > >>> I would like to advance the discussion further. There's one quite
>> >> > >>> important question that arose based on current state of work on
>> this
>> >> > >>> issue. If we use some kind of interactive console, like Visor,
>> then
>> >> > >>> how should it know whether SQL query it is requested to execute
>> >> > >>> returns a result set or not? In JDBC world, solution is quite
>> simple
>> >> -
>> >> > >>> there's base interface called Statement that all commands
>> implement,
>> >> > >>> and it has magic isResultSet method that tells whether statement
>> is a
>> >> > >>> query or an update command. The API proposed now has separate
>> Query
>> >> > >>> and Update operations which I believe to be a right thing by the
>> >> > >>> reasons I outlined in the beginning of this thread. However, their
>> >> > >>> lack of common ancestor prevents possible console clients from
>> >> running
>> >> > >>> text SQL commands in a fully transparent manner - like
>> >> > >>> IgniteCache.execute(String sql). Therefore I see two possible
>> ways of
>> >> > >>> solving this:
>> >> > >>>
>> >> > >>> - we change API so that it includes new class or interface
>> parenting
>> >> > >>> both Query and Update, and clients use it to communicate with
>> cache
>> >> > >>> - we let (or make :) ) the client determine command type
>> >> independently
>> >> > >>> and behave accordingly - for it to work it will have some kind of
>> >> > >>> command parsing by itself just to determine its type. Vi

Re: IGNITE-2294 implementation details

[Sergi Vladykin]

Exactly. This will allow our Jdbc driver to work transparently.

Sergi

2016-07-27 12:40 GMT+03:00 Alexander Paschenko <
alexander.a.pasche...@gmail.com>:

> Sergi,
>
> You wrote:
> > I'd prefer to return the same information, so it will not be empty
>
> Do you mean return iterator with single element that denotes number of
> rows?
>
> Dmitriy,
>
> You wrote:
> > What is the ticket number for this. Is the new API documented there?
>
> Overall issue number is 2294. There's no particular issue on API
> changes, but creating one seems to be a good idea, I will do it.
>
> - Alex
>
> 2016-07-27 9:20 GMT+03:00 Dmitriy Setrakyan :
> > What is the ticket number for this. Is the new API documented there?
> >
> > On Tue, Jul 26, 2016 at 11:36 AM, Sergi Vladykin <
> sergi.vlady...@gmail.com>
> > wrote:
> >
> >> I don't see anything ugly in empty iterator, sorry if I insulted your
> taste
> >> of beauty.
> >>
> >> If you will take a look at Jdbc, you will see that
> Statement.executeUpdate
> >> method returns number of updated rows, I'd prefer to return the same
> >> information, so it will not be empty (beauty is restored!).
> >>
> >> Sergi
> >>
> >>
> >>
> >> 2016-07-26 18:24 GMT+03:00 Alexander Paschenko <
> >> alexander.a.pasche...@gmail.com>:
> >>
> >> > I see your point. But what about my concerns from initial post?
> >> > Particularly about signatures of existing methods? I personally don't
> >> > like an option of query() method always returning an empty iterator
> >> > for any non-select query, it seems ugly design wise.
> >> >
> >> > - Alex
> >> >
> >> > 2016-07-26 18:15 GMT+03:00 Sergi Vladykin :
> >> > > BTW, the simplest way to solve this issue is to allow running SQL
> >> > commands
> >> > > inside of SqlFieldsQuery.
> >> > >
> >> > > We may add some additional convenience API for updates if we want,
> but
> >> > JDBC
> >> > > client will always call it like this:
> >> > >
> >> > > cache.query(new SqlFieldsQuery("INSERT INTO MY_TABLE
> >> > > VALUES(?,?)").setArgs(1,2));
> >> > >
> >> > > This will resolve any ambiguity.
> >> > >
> >> > > Sergi
> >> > >
> >> > > 2016-07-26 17:56 GMT+03:00 Sergi Vladykin  >:
> >> > >
> >> > >> I don't like any pre-parsing, especially with some libraries other
> >> than
> >> > >> H2. H2 itself has enough quirks to multiply it on quirks of another
> >> > library.
> >> > >>
> >> > >> This is exactly what I was talking about - we need some single
> entry
> >> > point
> >> > >> on API for all the SQL commands and queries. Thats why I suggested
> >> > >> SqlUpdate to extend Query. To me its is the cleanest approach. May
> be
> >> we
> >> > >> need to change in some backward compatible way this Query
> hierarchy to
> >> > get
> >> > >> rid of extra methods but the idea is still the same.
> >> > >>
> >> > >> Sergi
> >> > >>
> >> > >> 2016-07-26 14:34 GMT+03:00 Alexander Paschenko <
> >> > >> alexander.a.pasche...@gmail.com>:
> >> > >>
> >> > >>> Guys,
> >> > >>>
> >> > >>> I would like to advance the discussion further. There's one quite
> >> > >>> important question that arose based on current state of work on
> this
> >> > >>> issue. If we use some kind of interactive console, like Visor,
> then
> >> > >>> how should it know whether SQL query it is requested to execute
> >> > >>> returns a result set or not? In JDBC world, solution is quite
> simple
> >> -
> >> > >>> there's base interface called Statement that all commands
> implement,
> >> > >>> and it has magic isResultSet method that tells whether statement
> is a
> >> > >>> query or an update command. The API proposed now has separate
> Query
> >> > >>> and Update operations which I believe to be a right thing by the
> >> > >>> reasons I outlined in the beginning of this thread. However, their
> >> > >>> lack of common ancestor prevents possible console clients from
> >> running
> >> > >>> text SQL commands in a fully transparent manner - like
> >> > >>> IgniteCache.execute(String sql). Therefore I see two possible
> ways of
> >> > >>> solving this:
> >> > >>>
> >> > >>> - we change API so that it includes new class or interface
> parenting
> >> > >>> both Query and Update, and clients use it to communicate with
> cache
> >> > >>> - we let (or make :) ) the client determine command type
> >> independently
> >> > >>> and behave accordingly - for it to work it will have some kind of
> >> > >>> command parsing by itself just to determine its type. Visor
> console
> >> > >>> may use simple library like JSqlParser
> >> > >>> (https://github.com/JSQLParser/JSqlParser; dual LGPL 2.1/ASF 2.0
> >> > >>> licensed) to determine request type in terms of JDBC, and behave
> >> > >>> accordingly.
> >> > >>>
> >> > >>> Personally, I think that the second approach is better - and
> here's
> >> > why.
> >> > >>>
> >> > >>> First, it does not seem wise to change API simply to make console
> (or
> >> > >>> any other) clients simpler. Programmatic APIs should be concise
> and
> >> > >>> short for programmatic use, console clients should be eas

Re: IGNITE-2294 implementation details

[Alexander Paschenko]

Sergi,

You wrote:
> I'd prefer to return the same information, so it will not be empty

Do you mean return iterator with single element that denotes number of rows?

Dmitriy,

You wrote:
> What is the ticket number for this. Is the new API documented there?

Overall issue number is 2294. There's no particular issue on API
changes, but creating one seems to be a good idea, I will do it.

- Alex

2016-07-27 9:20 GMT+03:00 Dmitriy Setrakyan :
> What is the ticket number for this. Is the new API documented there?
>
> On Tue, Jul 26, 2016 at 11:36 AM, Sergi Vladykin 
> wrote:
>
>> I don't see anything ugly in empty iterator, sorry if I insulted your taste
>> of beauty.
>>
>> If you will take a look at Jdbc, you will see that Statement.executeUpdate
>> method returns number of updated rows, I'd prefer to return the same
>> information, so it will not be empty (beauty is restored!).
>>
>> Sergi
>>
>>
>>
>> 2016-07-26 18:24 GMT+03:00 Alexander Paschenko <
>> alexander.a.pasche...@gmail.com>:
>>
>> > I see your point. But what about my concerns from initial post?
>> > Particularly about signatures of existing methods? I personally don't
>> > like an option of query() method always returning an empty iterator
>> > for any non-select query, it seems ugly design wise.
>> >
>> > - Alex
>> >
>> > 2016-07-26 18:15 GMT+03:00 Sergi Vladykin :
>> > > BTW, the simplest way to solve this issue is to allow running SQL
>> > commands
>> > > inside of SqlFieldsQuery.
>> > >
>> > > We may add some additional convenience API for updates if we want, but
>> > JDBC
>> > > client will always call it like this:
>> > >
>> > > cache.query(new SqlFieldsQuery("INSERT INTO MY_TABLE
>> > > VALUES(?,?)").setArgs(1,2));
>> > >
>> > > This will resolve any ambiguity.
>> > >
>> > > Sergi
>> > >
>> > > 2016-07-26 17:56 GMT+03:00 Sergi Vladykin :
>> > >
>> > >> I don't like any pre-parsing, especially with some libraries other
>> than
>> > >> H2. H2 itself has enough quirks to multiply it on quirks of another
>> > library.
>> > >>
>> > >> This is exactly what I was talking about - we need some single entry
>> > point
>> > >> on API for all the SQL commands and queries. Thats why I suggested
>> > >> SqlUpdate to extend Query. To me its is the cleanest approach. May be
>> we
>> > >> need to change in some backward compatible way this Query hierarchy to
>> > get
>> > >> rid of extra methods but the idea is still the same.
>> > >>
>> > >> Sergi
>> > >>
>> > >> 2016-07-26 14:34 GMT+03:00 Alexander Paschenko <
>> > >> alexander.a.pasche...@gmail.com>:
>> > >>
>> > >>> Guys,
>> > >>>
>> > >>> I would like to advance the discussion further. There's one quite
>> > >>> important question that arose based on current state of work on this
>> > >>> issue. If we use some kind of interactive console, like Visor, then
>> > >>> how should it know whether SQL query it is requested to execute
>> > >>> returns a result set or not? In JDBC world, solution is quite simple
>> -
>> > >>> there's base interface called Statement that all commands implement,
>> > >>> and it has magic isResultSet method that tells whether statement is a
>> > >>> query or an update command. The API proposed now has separate Query
>> > >>> and Update operations which I believe to be a right thing by the
>> > >>> reasons I outlined in the beginning of this thread. However, their
>> > >>> lack of common ancestor prevents possible console clients from
>> running
>> > >>> text SQL commands in a fully transparent manner - like
>> > >>> IgniteCache.execute(String sql). Therefore I see two possible ways of
>> > >>> solving this:
>> > >>>
>> > >>> - we change API so that it includes new class or interface parenting
>> > >>> both Query and Update, and clients use it to communicate with cache
>> > >>> - we let (or make :) ) the client determine command type
>> independently
>> > >>> and behave accordingly - for it to work it will have some kind of
>> > >>> command parsing by itself just to determine its type. Visor console
>> > >>> may use simple library like JSqlParser
>> > >>> (https://github.com/JSQLParser/JSqlParser; dual LGPL 2.1/ASF 2.0
>> > >>> licensed) to determine request type in terms of JDBC, and behave
>> > >>> accordingly.
>> > >>>
>> > >>> Personally, I think that the second approach is better - and here's
>> > why.
>> > >>>
>> > >>> First, it does not seem wise to change API simply to make console (or
>> > >>> any other) clients simpler. Programmatic APIs should be concise and
>> > >>> short for programmatic use, console clients should be easy to use
>> from
>> > >>> console - and that's it: after all, console client exists to free a
>> > >>> user from burden of doing things programmatically, so its aim is to
>> > >>> adapt API to console or whatever UI.
>> > >>> Second, possible complications in client implied by such approach
>> > >>> certainly won't be dramatic - I don't think that additional single
>> > >>> query parsing operation in client code will make it much harder to
>> > >>> d

[GitHub] ignite pull request #895: IGNITE-3323 Use maven dependencies for Jetbrains a...

[jayho]

Github user jayho closed the pull request at:

https://github.com/apache/ignite/pull/895


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #900: IGNITE-3561 .NET: DistributedJoins property in Sql...

[ptupitsyn]

GitHub user ptupitsyn opened a pull request:

https://github.com/apache/ignite/pull/900

IGNITE-3561 .NET: DistributedJoins property in SqlQuery & SqlFieldsQuery - 
improve API and tests



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ptupitsyn/ignite ignite-3561-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/900.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #900






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-3591) .NET: Self-joins do not work in LINQ

[Pavel Tupitsyn (JIRA)]

Pavel Tupitsyn created IGNITE-3591:
--

 Summary: .NET: Self-joins do not work in LINQ
 Key: IGNITE-3591
 URL: https://issues.apache.org/jira/browse/IGNITE-3591
 Project: Ignite
  Issue Type: Bug
  Components: platforms
Affects Versions: 1.6
Reporter: Pavel Tupitsyn
Assignee: Pavel Tupitsyn
 Fix For: 1.7


When joining the table on itself, the same table alias is used, leading to 
ambiguous column name error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Apache Ignite - New Release

[Semyon Boikov]

Dmitry, all Ignite features are always thoroughly tested =)

I created branch ignite-1.7.0.

On Wed, Jul 27, 2016 at 9:32 AM, Dmitriy Setrakyan 
wrote:

> On Wed, Jul 27, 2016 at 2:29 AM, Semyon Boikov 
> wrote:
>
> > Sure Dmitry, we will run tests with large data set.
> >
> > Regarding 1.7. release: If there are no objections I'm going to cut off
> 1.7
> > branch and start prepare it for release.
> >
>
> In my opinion this is a great addition to Ignite and is definitely worth a
> release, again, assuming that we thoroughly test it.
>
>
> >
> >
> > On Wed, Jul 27, 2016 at 9:15 AM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
> > wrote:
> >
> > > On Wed, Jul 27, 2016 at 2:09 AM, Semyon Boikov 
> > > wrote:
> > >
> > > > Regarding distributed join testsing: we added tests verifying correct
> > > join
> > > > behavior and correct execution plan generation for various SQL
> queries,
> > > > tests for joins for various cache types (different number of backups,
> > > > partitioned/replicated), there are tests verifying correct
> distributed
> > > > joins results on changing topology with nodes restarts. Also we added
> > > > benchmarks which will be used to verify that there are no performance
> > > > degradation in this functionality between releases. These benchmarks
> > were
> > > > executed on real clusters, and in next few days we are going to run
> > more
> > > > load tests.
> > > >
> > >
> > > Thanks Semyon, sounds great! I would also test it on larger data sets
> to
> > > see how a join query will take, say, on 10GB of data. Is it possible?
> > >
> > >
> > > >
> > > > On Fri, Jul 22, 2016 at 5:39 PM, Dmitriy Setrakyan <
> > > dsetrak...@apache.org>
> > > > wrote:
> > > >
> > > > > On Fri, Jul 22, 2016 at 7:10 AM, Semyon Boikov <
> sboi...@gridgain.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Today I merged into master 'distributed join' implementation -
> > > > > > https://issues.apache.org/jira/browse/IGNITE-1232 (thanks to
> > Sergi,
> > > he
> > > > > > implemented this feature). I think this together with recent
> > bugfixes
> > > > > worth
> > > > > > 1.7 release. Do you think we can cut off 1.7 release branch from
> > > > master?
> > > > > >
> > > > >
> > > > > Great news. Can you describe that amount of testing we did for this
> > > > > feature?
> > > > >
> > > > >
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > >
> > > > > > On Thu, Jul 21, 2016 at 7:47 PM, Alexandre Boudnik <
> > > > > > alexander.boud...@gmail.com> wrote:
> > > > > >
> > > > > > > Sorry, I missed the typo:
> > > > > > > To support the point, I would add that PosgreSQL has
> demonstrated
> > > the
> > > > > > > similar behavior (inspired by unix .dot files and ls):
> > > > > > > - they have "hidden" column: xmin and xmax in any table
> > > > > > > - they do not appear in select * list unless they are specified
> > > > > > explicitly
> > > > > > > Take care,
> > > > > > > Alexandre "Sasha" Boudnik
> > > > > > >
> > > > > > > call me via Google Voice:
> > > > > > > 1(405) BUDNIKA
> > > > > > > 1(405) 283-6452
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jul 21, 2016 at 12:46 PM, Alexandre Boudnik
> > > > > > >  wrote:
> > > > > > > > To support the point, I would add that PosgreSQL has
> > demonstrated
> > > > the
> > > > > > > > similar behavior (inspired by unix .dot files and ls):
> > > > > > > > - they have "hidden" column: xmin and xmax in any table
> > > > > > > > - they appear in select * list unless they are specified
> > > explicitly
> > > > > > > >
> > > > > > > > I'll add some notices to the ticket
> > > > > > > > Take care,
> > > > > > > > Alexandre "Sasha" Boudnik
> > > > > > > >
> > > > > > > > call me via Google Voice:
> > > > > > > > 1(405) BUDNIKA
> > > > > > > > 1(405) 283-6452
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Jul 15, 2016 at 6:37 PM, Valentin Kulichenko
> > > > > > > >  wrote:
> > > > > > > >> Agree.
> > > > > > > >>
> > > > > > > >> On Fri, Jul 15, 2016 at 12:59 PM, Alexey Goncharuk <
> > > > > > > >> alexey.goncha...@gmail.com> wrote:
> > > > > > > >>
> > > > > > > >>> Looks like the ticket for removing _key and _value from
> > selct *
> > > > is
> > > > > a
> > > > > > > good
> > > > > > > >>> candidate for 2.0.
> > > > > > > >>>
> > > > > > > >>> 2016-07-15 5:12 GMT-07:00 Sergi Vladykin <
> > > > sergi.vlady...@gmail.com
> > > > > >:
> > > > > > > >>>
> > > > > > > >>> > We will not be able to just change this, because it will
> > > brake
> > > > > > > >>> > compatibility. Still I believe that an option to define
> our
> > > SQL
> > > > > > > tables
> > > > > > > >>> > without _key and _value fields. But this is another story
> > you
> > > > can
> > > > > > > file a
> > > > > > > >>> > ticket for it, can we fix somehow our JDBC for now? Like
> > > > > returning
> > > > > > > >>> > BinaryObject instance or something?
> > > > > > > >>> >
> > > > > > > >>> > Sergi
> > > > > > > >>> >
> > > > > > > >>> >
> > > > > > > 

Re: Data compression in Ignite 2.0

[Sergi Vladykin]

Nikita,

I agree with Andrey, HANA is a bad comparison to Ignite in this respect. I
did not find any evidence on the internet that their row store is very
efficient with compression. It was always about column store.

Alexey,

As for DB2, can you check what exactly, when and how it compresses and does
it give any decent results before suggesting it as an example to follow? I
don't think it is good idea to repeat every bad idea after other products.

And even if there are good results in DB2, will this all be applicable to
Ignite? PostgreSql for example provides TOAST compression and this can be
useful when used in a smart way, but this is a very different architecture
from what we have.

All in all I agree that may be we should provide some kind of pluggable
compression SPI support, but do not expect much from it, usually it will be
just useless.

Sergi



2016-07-27 10:16 GMT+03:00 Sebastien DIAZ :

> Hi
>
> I add Redis as a sample of memory compression strategy
>
> http://labs.octivi.com/how-we-cut-down-memory-usage-by-82/
>
> http://redis.io/topics/memory-optimization
>
> Regards
>
> S DIAZ
>
>
>
> 2016-07-27 8:24 GMT+02:00 Alexey Kuznetsov :
>
> > Nikita,
> >
> > That was my intention: "we may need to provide a better facility to
> inject
> > user's logic here..."
> >
> > Andrey,
> > About compression, once again - DB2 is a row-based DB and they can
> compress
> > :)
> >
> > On Wed, Jul 27, 2016 at 12:56 PM, Nikita Ivanov 
> > wrote:
> >
> > > Very good points indeed. I get the compression in Ignite question quite
> > > often and Hana reference is a typical lead in.
> > >
> > > My personal opinion is still that in Ignite *specifically* the
> > compression
> > > is best left to the end-user. But we may need to provide a better
> > facility
> > > to inject user's logic here...
> > >
> > > --
> > > Nikita Ivanov
> > >
> > >
> > > On Tue, Jul 26, 2016 at 9:53 PM, Andrey Kornev <
> andrewkor...@hotmail.com
> > >
> > > wrote:
> > >
> > > > Dictionary compression requires some knowledge about data being
> > > > compressed. For example, for numeric types a range of values must be
> > > known
> > > > so that the dictionary can be generated. For strings, the number of
> > > unique
> > > > values of the column is the key piece of input into the dictionary
> > > > generation.
> > > > SAP HANA is a column-based database system: it stores the fields of
> the
> > > > data tuple individually using the best compression for the given data
> > > type
> > > > and the particular set of values. HANA has been specifically built
> as a
> > > > general purpose database, rather than as an afterthought layer on top
> > of
> > > an
> > > > already existing distributed cache.
> > > > On the other hand, Ignite is a distributed cache implementation (a
> > pretty
> > > > good one!) that in general requires no schema and stores its data in
> > the
> > > > row-based fashion. Its current design doesn't land itself readily to
> > the
> > > > kind of optimizations HANA provides out of the box.
> > > > For the curios types among us, the implementation details of HANA are
> > > well
> > > > documented in "In-memory Data Management", by Hasso Plattner &
> > Alexander
> > > > Zeier.
> > > > Cheers
> > > > Andrey
> > > > _
> > > > From: Alexey Kuznetsov  > > > akuznet...@gridgain.com>>
> > > > Sent: Tuesday, July 26, 2016 5:36 AM
> > > > Subject: Re: Data compression in Ignite 2.0
> > > > To: mailto:dev@ignite.apache.org>>
> > > >
> > > >
> > > > Sergey Kozlov wrote:
> > > > >> For approach 1: Put a large object into a partition cache will
> > > > force to update
> > > > the dictionary placed on replication cache. It may be time-expense
> > > > operation.
> > > > The dictionary will be built only once. And we could control what
> > should
> > > be
> > > > put into dictionary, for example, we could check min and max size and
> > > > decide - put value to dictionary or not.
> > > >
> > > > >> Approach 2-3 are make sense for rare cases as Sergi commented.
> > > > But it is better at least have a possibility to plug user code for
> > > > compression than not to have it at all.
> > > >
> > > > >> Also I see a danger of OOM if we've got high compression level and
> > try
> > > > to restore original value in memory.
> > > > We could easily get OOM with many other operations right now without
> > > > compression, I think it is not an issue, we could add a NOTE to
> > > > documentation about such possibility.
> > > >
> > > > Andrey Kornev wrote:
> > > > >> ... in general I think compression is a great data. The cleanest
> way
> > > to
> > > > achieve that would be to just make it possible to chain the
> > > marshallers...
> > > > I think it is also good idea. And looks like it could be used for
> > > > compression with some sort of ZIP algorithm, but how to deal with
> > > > compression by dictionary substitution?
> > > > We need to build dictionary first. Any ideas?
> > > >
> > > > Nikita Ivanov wrote:
> > > > >> SAP Hana d

Re: IgniteConfiguration.toXml()

[Pavel Tupitsyn]

So, no objections for adding this in .NET?

Another important use case is migrating from Spring XML (the only option in
1.5) to .NET XML (1.6+):
instead of rewriting config by hand, users can call
ignite.GetConfigurarion().ToXml() and copy the result.

On Tue, Jul 26, 2016 at 11:17 AM, Alexey Kuznetsov 
wrote:

> Pavel,
>
> In Web Console [1] we have generation of Spring XML and appropriate Java
> code generation.
> And from my experience it is not trivial thing to generate Spring XML in
> some cases (if we will write code by ourselves).
>
> If we will use Spring marshaling - that will bring a dependency from Spring
>  to ignite-core and we cannot do this.
> We could add this as utility method to ignite-spring module.
>
> [1]. https://ignite.apache.org/addons.html#web-console
>
>
> On Tue, Jul 26, 2016 at 2:19 PM, Pavel Tupitsyn 
> wrote:
>
> > Igniters,
> >
> > This applies equally to Java and .NET:
> >
> > Writing XML configuration in not easy or fun. I've seen lots of questions
> > on how to configure things in XML.
> > Spring syntax is cumbersome even when you get used to it.
> >
> > On the other hand, setting IgniteConfiguration properties directly in
> Java
> > or C# is much easier because of the IDE support.
> > Sometimes users will start with a code-only app and decide to move
> > configuration to XML.
> >
> > So I propose to add IgniteConfiguration.toXml() method in Ignite.NET.
> > Not sure how easy it is in Java, but .NET already has it, just hidden
> from
> > public API.
> >
> > Thoughts? Objections?
> >
> > Pavel.
> >
>
>
>
> --
> Alexey Kuznetsov
> GridGain Systems
> www.gridgain.com
>

[jira] [Created] (IGNITE-3590) Create test and first simple implementation of secondary file system.

[Dmitry Karachentsev (JIRA)]

Dmitry Karachentsev created IGNITE-3590:
---

 Summary: Create test and first simple implementation of secondary 
file system.
 Key: IGNITE-3590
 URL: https://issues.apache.org/jira/browse/IGNITE-3590
 Project: Ignite
  Issue Type: Sub-task
  Components: IGFS
Affects Versions: 1.6
Reporter: Dmitry Karachentsev
Assignee: Dmitry Karachentsev
 Fix For: 1.7


Create new simple (copy paste from IgniteHadoopIgfsSecondaryFileSystem) 
inplementation and tests for it as start point.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Data compression in Ignite 2.0

[Sebastien DIAZ]

Hi

I add Redis as a sample of memory compression strategy

http://labs.octivi.com/how-we-cut-down-memory-usage-by-82/

http://redis.io/topics/memory-optimization

Regards

S DIAZ



2016-07-27 8:24 GMT+02:00 Alexey Kuznetsov :

> Nikita,
>
> That was my intention: "we may need to provide a better facility to inject
> user's logic here..."
>
> Andrey,
> About compression, once again - DB2 is a row-based DB and they can compress
> :)
>
> On Wed, Jul 27, 2016 at 12:56 PM, Nikita Ivanov 
> wrote:
>
> > Very good points indeed. I get the compression in Ignite question quite
> > often and Hana reference is a typical lead in.
> >
> > My personal opinion is still that in Ignite *specifically* the
> compression
> > is best left to the end-user. But we may need to provide a better
> facility
> > to inject user's logic here...
> >
> > --
> > Nikita Ivanov
> >
> >
> > On Tue, Jul 26, 2016 at 9:53 PM, Andrey Kornev  >
> > wrote:
> >
> > > Dictionary compression requires some knowledge about data being
> > > compressed. For example, for numeric types a range of values must be
> > known
> > > so that the dictionary can be generated. For strings, the number of
> > unique
> > > values of the column is the key piece of input into the dictionary
> > > generation.
> > > SAP HANA is a column-based database system: it stores the fields of the
> > > data tuple individually using the best compression for the given data
> > type
> > > and the particular set of values. HANA has been specifically built as a
> > > general purpose database, rather than as an afterthought layer on top
> of
> > an
> > > already existing distributed cache.
> > > On the other hand, Ignite is a distributed cache implementation (a
> pretty
> > > good one!) that in general requires no schema and stores its data in
> the
> > > row-based fashion. Its current design doesn't land itself readily to
> the
> > > kind of optimizations HANA provides out of the box.
> > > For the curios types among us, the implementation details of HANA are
> > well
> > > documented in "In-memory Data Management", by Hasso Plattner &
> Alexander
> > > Zeier.
> > > Cheers
> > > Andrey
> > > _
> > > From: Alexey Kuznetsov  > > akuznet...@gridgain.com>>
> > > Sent: Tuesday, July 26, 2016 5:36 AM
> > > Subject: Re: Data compression in Ignite 2.0
> > > To: mailto:dev@ignite.apache.org>>
> > >
> > >
> > > Sergey Kozlov wrote:
> > > >> For approach 1: Put a large object into a partition cache will
> > > force to update
> > > the dictionary placed on replication cache. It may be time-expense
> > > operation.
> > > The dictionary will be built only once. And we could control what
> should
> > be
> > > put into dictionary, for example, we could check min and max size and
> > > decide - put value to dictionary or not.
> > >
> > > >> Approach 2-3 are make sense for rare cases as Sergi commented.
> > > But it is better at least have a possibility to plug user code for
> > > compression than not to have it at all.
> > >
> > > >> Also I see a danger of OOM if we've got high compression level and
> try
> > > to restore original value in memory.
> > > We could easily get OOM with many other operations right now without
> > > compression, I think it is not an issue, we could add a NOTE to
> > > documentation about such possibility.
> > >
> > > Andrey Kornev wrote:
> > > >> ... in general I think compression is a great data. The cleanest way
> > to
> > > achieve that would be to just make it possible to chain the
> > marshallers...
> > > I think it is also good idea. And looks like it could be used for
> > > compression with some sort of ZIP algorithm, but how to deal with
> > > compression by dictionary substitution?
> > > We need to build dictionary first. Any ideas?
> > >
> > > Nikita Ivanov wrote:
> > > >> SAP Hana does the compression by 1) compressing SQL parameters
> before
> > > execution...
> > > Looks interesting, but my initial point was about compression of cache
> > > data, not SQL queries.
> > > My idea was to make compression transparent for SQL engine when it will
> > > lookup for data.
> > >
> > > But idea of compressing SQL queries result looks very interesting,
> > because
> > > it is known fact, that SQL engine could consume quite a lot of heap for
> > > storing result sets.
> > > I think this should be discussed in separate thread.
> > >
> > > Just for you information, in first message I mentioned that DB2 has
> > > compression by dictionary and according to them it is possible to
> > > compress usual data to 50-80%.
> > > I have some experience with DB2 and can confirm this.
> > >
> > > --
> > > Alexey Kuznetsov
> >
>
>
> --
> Alexey Kuznetsov
>

Re: Enums and SQL queries

[Sergi Vladykin]

Nope. This will not work. Because take for example the following query:

select my_enum_field from my_table where my_enum_field = 'BLA'

Field well be evaluated only once here, but it must be a Sting and Enum at
the same time.

Probably we could contribute to H2 `User defined value types` and handle
this stuff in our own Enum value types, which will support comparison with
String. But this requires bug H2 Modification and is a compatibility
breaking change.

Sergi



2016-07-27 9:23 GMT+03:00 Dmitriy Setrakyan :

> Hm... I would prefer to find a way to handle Enums automatically.
>
> In case if user expects a String, can we catch an exception and do
> automatic conversion at that time? In this case, we could catch the
> exception once and remember the decision. Will something like this work?
>
> D.
>
> On Tue, Jul 26, 2016 at 1:49 AM, Sergi Vladykin 
> wrote:
>
> > GridQueryProcessor contains two types: BinaryProperty and ClassProperty.
> > Indexing uses them for fields access. Probably we can change them, so
> that
> > they will always return String instead of Enum type but this
> >
> > 1. looks like a compatibility breaking change;
> > 2. users doing queries like `select my_enum_field from my_table` probably
> > still expect to get a Enum result but not a String.
> >
> > What do you think?
> >
> > Sergi
> >
> >
> > 2016-07-26 3:17 GMT+03:00 Valentin Kulichenko <
> > valentin.kuliche...@gmail.com
> > >:
> >
> > > Sergi,
> > >
> > > Ideally, I would like this query to work:
> > >
> > > select * from MyTable where myEnumField = 'MY_ENUM_NAME'
> > >
> > > The problem is that the field value is not a string, it's a binary
> object
> > > with the enum value ordinal inside. We can add enum value names into
> the
> > > metadata, so that we can always map the field value to its string
> > > representation, but SQL engine will have to know how to use it. Is this
> > > possible?
> > >
> > > -Val
> > >
> > > On Fri, Jul 22, 2016 at 10:25 PM, Sergi Vladykin <
> > sergi.vlady...@gmail.com
> > > > wrote:
> > >
> > >> I'm not sure what exactly you are going to patch in SQL engine. We
> could
> > >> and some SQL function like enum('MY_ENUM_NAME') but probably it will
> not
> > >> work if we will not pass enum type there as well, so it have look like
> > >> enum('com.mycompany.MyEnum', 'MY_ENUM_NAME') which is ugly.
> > >>
> > >> Sergi
> > >>
> > >> On 23 июля 2016 г., 0:43, Valentin Kulichenko <
> > >> valentin.kuliche...@gmail.com
> > >> > wrote:
> > >>
> > >> > Folks,
> > >> >
> > >> > I noticed that we currently have a very weird limitation for enum
> > fields
> > >> > when running SQL queries. Basically, you can't use enum value names
> as
> > >> many
> > >> > users would expect.
> > >> >
> > >> > Technically this happens because binary format stores only ordinal
> and
> > >> the
> > >> > only way to run such query is to do something like this:
> > >> >
> > >> > SqlFieldsQuery qry = new SqlFieldsQuery("select * from Table where
> > >> > enumField = ?");
> > >> > qry.setArgs(MyEnum.ENUM_VALUE);
> > >> > cache.query(qry);
> > >> >
> > >> > This means that this query can be executed only if:
> > >> > - IgniteCache API is used. So it doesn't work in JDBC driver,
> > Zeppelin,
> > >> > etc.
> > >> > - The client that executes the query has the enum class on
> classpath.
> > >> >
> > >> > My first thought about fixing this is to somehow store value names
> in
> > >> > binary metadata and patch the SQL engine so that it understands how
> to
> > >> use
> > >> > this information.
> > >> >
> > >> > Thoughts?
> > >> >
> > >> > -Val
> > >> >
> > >>
> > >
> > >
> >
>