Re: Enums and SQL queries

[Sergi Vladykin]

GridQueryProcessor contains two types: BinaryProperty and ClassProperty.
Indexing uses them for fields access. Probably we can change them, so that
they will always return String instead of Enum type but this

1. looks like a compatibility breaking change;
2. users doing queries like `select my_enum_field from my_table` probably
still expect to get a Enum result but not a String.

What do you think?

Sergi


2016-07-26 3:17 GMT+03:00 Valentin Kulichenko :

> Sergi,
>
> Ideally, I would like this query to work:
>
> select * from MyTable where myEnumField = 'MY_ENUM_NAME'
>
> The problem is that the field value is not a string, it's a binary object
> with the enum value ordinal inside. We can add enum value names into the
> metadata, so that we can always map the field value to its string
> representation, but SQL engine will have to know how to use it. Is this
> possible?
>
> -Val
>
> On Fri, Jul 22, 2016 at 10:25 PM, Sergi Vladykin  > wrote:
>
>> I'm not sure what exactly you are going to patch in SQL engine. We could
>> and some SQL function like enum('MY_ENUM_NAME') but probably it will not
>> work if we will not pass enum type there as well, so it have look like
>> enum('com.mycompany.MyEnum', 'MY_ENUM_NAME') which is ugly.
>>
>> Sergi
>>
>> On 23 июля 2016 г., 0:43, Valentin Kulichenko <
>> valentin.kuliche...@gmail.com
>> > wrote:
>>
>> > Folks,
>> >
>> > I noticed that we currently have a very weird limitation for enum fields
>> > when running SQL queries. Basically, you can't use enum value names as
>> many
>> > users would expect.
>> >
>> > Technically this happens because binary format stores only ordinal and
>> the
>> > only way to run such query is to do something like this:
>> >
>> > SqlFieldsQuery qry = new SqlFieldsQuery("select * from Table where
>> > enumField = ?");
>> > qry.setArgs(MyEnum.ENUM_VALUE);
>> > cache.query(qry);
>> >
>> > This means that this query can be executed only if:
>> > - IgniteCache API is used. So it doesn't work in JDBC driver, Zeppelin,
>> > etc.
>> > - The client that executes the query has the enum class on classpath.
>> >
>> > My first thought about fixing this is to somehow store value names in
>> > binary metadata and patch the SQL engine so that it understands how to
>> use
>> > this information.
>> >
>> > Thoughts?
>> >
>> > -Val
>> >
>>
>
>

Re: "ArrayIndexOutOfBoundsException" happened when doing qurey in version 1.6.0

[Semyon Boikov]

We already have similar issue reproduced in our benchmarks -
https://issues.apache.org/jira/browse/IGNITE-3300. I think this is related
to optimziation done in 1.6 to store key partition in the cache key. I
believe fix will be included in 1.7 release.

On Tue, Jul 26, 2016 at 2:13 AM, Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Crossposting to dev@
>
> Folks,
>
> I noticed the exception below in couple of user threads already. This
> actually means that partition number for the key is -1, which doesn't make
> much sense to me. Does anyone has any ideas?
>
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
>   at java.util.ArrayList.elementData(ArrayList.java:400)
>   at java.util.ArrayList.get(ArrayList.java:413)
>   at 
> org.apache.ignite.internal.processors.affinity.GridAffinityAssignment.get(GridAffinityAssignment.java:152)
>   at 
> org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.nodes(GridAffinityAssignmentCache.java:387)
>   at 
> org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.nodes(GridCacheAffinityManager.java:251)
>   at 
> org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.primary(GridCacheAffinityManager.java:287)
>   at 
> org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.primary(GridCacheAffinityManager.java:278)
>   at 
> org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.primary(GridCacheAffinityManager.java:302)
>   at 
> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$7$3.apply(IgniteH2Indexing.java:1730)
>   at 
> org.apache.ignite.internal.processors.query.h2.opt.GridH2IndexBase$FilteringIterator.accept(GridH2IndexBase.java:251)
>   at 
> org.apache.ignite.internal.processors.query.h2.opt.GridH2IndexBase$FilteringIterator.accept(GridH2IndexBase.java:200)
>   at 
> org.apache.ignite.internal.util.lang.GridFilteredIterator.hasNext(GridFilteredIterator.java:59)
>   at 
> org.apache.ignite.internal.processors.query.h2.opt.GridH2Cursor.next(GridH2Cursor.java:59)
>   at org.h2.index.IndexCursor.next(IndexCursor.java:274)
>   at org.h2.table.TableFilter.next(TableFilter.java:359)
>   at org.h2.command.dml.Select.queryFlat(Select.java:527)
>   at org.h2.command.dml.Select.queryWithoutCache(Select.java:632)
>   at org.h2.command.dml.Query.query(Query.java:297)
>   at org.h2.command.dml.Query.query(Query.java:284)
>   at org.h2.command.dml.Query.query(Query.java:36)
>   at org.h2.command.CommandContainer.query(CommandContainer.java:91)
>   at org.h2.command.Command.executeQuery(Command.java:196)
>   ... 26 more
>
>
> -Val
>
>
> On Mon, Jul 25, 2016 at 5:07 AM, ght230  wrote:
>
>> ignite-related.xml
>> <
>> http://apache-ignite-users.70518.x6.nabble.com/file/n6516/ignite-related.xml
>> >
>> Please refer to the attachment.
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-ignite-users.70518.x6.nabble.com/ArrayIndexOutOfBoundsException-happened-when-doing-qurey-in-version-1-6-0-tp6245p6516.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>

[GitHub] ignite pull request #889: ignite-3563 Support distributedJoins flag in JDBC ...

[agura]

Github user agura closed the pull request at:

https://github.com/apache/ignite/pull/889


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #891: ignite-3573 Support of Distributed joins queries i...

[agura]

Github user agura closed the pull request at:

https://github.com/apache/ignite/pull/891


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Data compression in Ignite 2.0

[Andrey Kornev]

I'm guessing the suggestion here is to use the compressed form directly for 
WHERE clause evaluation. If that's the case I think there are a couple of 
issues:

1) the LIKE predicate.

2) predicates other than equality (for example, <, >, etc.)


But since Ignite isn't just about SQL queries (surprisingly some people still 
use it just as distributed cache!), in general I think compression is a great 
data. The cleanest way to achieve that would be to just make it possible to 
chain the marshallers. It is possible to do it already without any Ignite code 
changes, but unfortunately it would force people to use the non-public 
BinaryMarshaller class directly (as the first element of the chain).


Cheers

Andrey


From: Dmitriy Setrakyan 
Sent: Monday, July 25, 2016 1:53 PM
To: dev@ignite.apache.org
Subject: Re: Data compression in Ignite 2.0

Nikita, this sounds like a pretty elegant approach.

Does anyone in the community see a problem with this design?

On Mon, Jul 25, 2016 at 4:49 PM, Nikita Ivanov  wrote:

> SAP Hana does the compression by 1) compressing SQL parameters before
> execution, and 2) storing only compressed data in memory. This way all SQL
> queries work as normal with zero modifications or performance overhead.
> Only results of the query can be (optionally) decompressed back before
> returning to the user.
>
> --
> Nikita Ivanov
>
>
> On Mon, Jul 25, 2016 at 1:40 PM, Sergey Kozlov 
> wrote:
>
> > Hi
> >
> > For approach 1: Put a large object into a partiton cache will force to
> > update the dictionary placed on replication cache. It seeis it may be
> > time-expense operation.
> > Appoach 2-3 are make sense for rare cases as Sergi commented.
> > Aslo I see a danger of OOM if we've got high compression level and try to
> > restore orginal value in memory.
> >
> > On Mon, Jul 25, 2016 at 10:39 AM, Alexey Kuznetsov <
> > akuznet...@gridgain.com>
> > wrote:
> >
> > > Sergi,
> > >
> > > Of course it will introduce some slowdown, but with compression more
> data
> > > could be stored in memory
> > > and not will be evicted to disk. In case of compress by dictionary
> > > substitution it will be only one more lookup
> > > and should be fast.
> > >
> > > In general we could provide only API for compression out of the box,
> and
> > > users that really need some sort of compression
> > > will implement it by them self. This will not require much effort I
> > think.
> > >
> > >
> > >
> > > On Mon, Jul 25, 2016 at 2:18 PM, Sergi Vladykin <
> > sergi.vlady...@gmail.com>
> > > wrote:
> > >
> > > > This will make sense only for rare cases when you have very large
> > objects
> > > > stored, which can be effectively compressed. And even then it will
> > > > introduce slowdown on all the operations, which often will not be
> > > > acceptable. I guess only few users will find this feature useful,
> thus
> > I
> > > > think it does not worth the effort.
> > > >
> > > > Sergi
> > > >
> > > >
> > > >
> > > > 2016-07-25 9:28 GMT+03:00 Alexey Kuznetsov  >:
> > > >
> > > > > Hi, All!
> > > > >
> > > > > I would like to propose one more feature for Ignite 2.0.
> > > > >
> > > > > Data compression for data in binary format.
> > > > >
> > > > > Binary format is stored as field name + field data.
> > > > > So we have a description.
> > > > > How about to add one more byte to binary data descriptor:
> > > > >
> > > > > *Compressed*:
> > > > >  0 - Data stored as is (no compression).
> > > > >  1 - Data compressed by dictionary (something like DB2 row
> > compression
> > > > [1],
> > > > >  but for all binary types). We could have system or user defined
> > > > replicated
> > > > > cache for such dictionary and *cache.compact()* method that will
> scan
> > > > > cache, build dictionary and compact data.
> > > > >  2 - Data compressed by Java built in ZIP.
> > > > >  3 - Data compressed by some user custom algorithm.
> > > > >
> > > > > Of course it is possible to compress data in current Ignite 1.x but
> > in
> > > > this
> > > > > case compressed data cannot be accessed from SQL engine, if we
> > > implement
> > > > > support for compression on Ignite core level SQL engine will be
> able
> > to
> > > > > detect that data is compressed and properly handle such data.
> > > > >
> > > > > What do you think?
> > > > > If community consider this feature useful I will create issue in
> > JIRA.
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> http://www.ibm.com/developerworks/data/library/techarticle/dm-1205db210compression/
Optimize storage with deep compression in DB2 
10
www.ibm.com
Thomas Fanghaenel has been with IBM for nine years and is a Senior Software 
Engineer with the DB2 Data Warehouse Storage and Indexing ...


> > > > >
> > > > > --
> > > > > Alexey Kuznetsov
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Alexey Kuznetsov
> > >
> >
> >
> >
> > --
> > Sergey Kozlov
> > GridGain

Re: Enums and SQL queries

[Valentin Kulichenko]

Sergi,

Ideally, I would like this query to work:

select * from MyTable where myEnumField = 'MY_ENUM_NAME'

The problem is that the field value is not a string, it's a binary object
with the enum value ordinal inside. We can add enum value names into the
metadata, so that we can always map the field value to its string
representation, but SQL engine will have to know how to use it. Is this
possible?

-Val

On Fri, Jul 22, 2016 at 10:25 PM, Sergi Vladykin 
wrote:

> I'm not sure what exactly you are going to patch in SQL engine. We could
> and some SQL function like enum('MY_ENUM_NAME') but probably it will not
> work if we will not pass enum type there as well, so it have look like
> enum('com.mycompany.MyEnum', 'MY_ENUM_NAME') which is ugly.
>
> Sergi
>
> On 23 июля 2016 г., 0:43, Valentin Kulichenko <
> valentin.kuliche...@gmail.com
> > wrote:
>
> > Folks,
> >
> > I noticed that we currently have a very weird limitation for enum fields
> > when running SQL queries. Basically, you can't use enum value names as
> many
> > users would expect.
> >
> > Technically this happens because binary format stores only ordinal and
> the
> > only way to run such query is to do something like this:
> >
> > SqlFieldsQuery qry = new SqlFieldsQuery("select * from Table where
> > enumField = ?");
> > qry.setArgs(MyEnum.ENUM_VALUE);
> > cache.query(qry);
> >
> > This means that this query can be executed only if:
> > - IgniteCache API is used. So it doesn't work in JDBC driver, Zeppelin,
> > etc.
> > - The client that executes the query has the enum class on classpath.
> >
> > My first thought about fixing this is to somehow store value names in
> > binary metadata and patch the SQL engine so that it understands how to
> use
> > this information.
> >
> > Thoughts?
> >
> > -Val
> >
>

Re: "ArrayIndexOutOfBoundsException" happened when doing qurey in version 1.6.0

[Valentin Kulichenko]

Crossposting to dev@

Folks,

I noticed the exception below in couple of user threads already. This
actually means that partition number for the key is -1, which doesn't make
much sense to me. Does anyone has any ideas?

Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
at java.util.ArrayList.elementData(ArrayList.java:400)
at java.util.ArrayList.get(ArrayList.java:413)
at 
org.apache.ignite.internal.processors.affinity.GridAffinityAssignment.get(GridAffinityAssignment.java:152)
at 
org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.nodes(GridAffinityAssignmentCache.java:387)
at 
org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.nodes(GridCacheAffinityManager.java:251)
at 
org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.primary(GridCacheAffinityManager.java:287)
at 
org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.primary(GridCacheAffinityManager.java:278)
at 
org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.primary(GridCacheAffinityManager.java:302)
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$7$3.apply(IgniteH2Indexing.java:1730)
at 
org.apache.ignite.internal.processors.query.h2.opt.GridH2IndexBase$FilteringIterator.accept(GridH2IndexBase.java:251)
at 
org.apache.ignite.internal.processors.query.h2.opt.GridH2IndexBase$FilteringIterator.accept(GridH2IndexBase.java:200)
at 
org.apache.ignite.internal.util.lang.GridFilteredIterator.hasNext(GridFilteredIterator.java:59)
at 
org.apache.ignite.internal.processors.query.h2.opt.GridH2Cursor.next(GridH2Cursor.java:59)
at org.h2.index.IndexCursor.next(IndexCursor.java:274)
at org.h2.table.TableFilter.next(TableFilter.java:359)
at org.h2.command.dml.Select.queryFlat(Select.java:527)
at org.h2.command.dml.Select.queryWithoutCache(Select.java:632)
at org.h2.command.dml.Query.query(Query.java:297)
at org.h2.command.dml.Query.query(Query.java:284)
at org.h2.command.dml.Query.query(Query.java:36)
at org.h2.command.CommandContainer.query(CommandContainer.java:91)
at org.h2.command.Command.executeQuery(Command.java:196)
... 26 more


-Val


On Mon, Jul 25, 2016 at 5:07 AM, ght230  wrote:

> ignite-related.xml
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/n6516/ignite-related.xml
> >
> Please refer to the attachment.
>
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/ArrayIndexOutOfBoundsException-happened-when-doing-qurey-in-version-1-6-0-tp6245p6516.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Implement collecting what SQL statements executed on cluster and their metrics.

[Valentin Kulichenko]

SET query is for IgniteSet and it's internal only, not exposed to the
public API.

-Val

On Mon, Jul 25, 2016 at 3:37 PM, Dmitriy Setrakyan 
wrote:

> My comments are below...
>
> On Thu, Jul 7, 2016 at 9:41 AM, Alexey Kuznetsov 
> wrote:
>
> > Hi, All!
> >
> > I'm working on issue "Implement collecting what SQL statements executed
> on
> > cluster and their metrics" [1]
> >
> > For now I'm investigating code and have following questions:
> >
> > 1) We have different types of queries:  GridCacheQueryType: SPI, SCAN,
> SQL,
> > SQL_FIELDS, TEXT, SET
> > Should I implement collecting metrics for ALL of them?
> > Or for example I could collect:
> >SCAN: cache name
> >SQL: type name + SQL clauseю
> >SQL_FIELDS: SQL statement text
> >TEXT: type + searched string
> >
> > What I could collect for SPI and SET query types? Or I may ignore them?
> >
>
> I would implement stat gathering for all the known ones. SPI is user
> defined type, so  I am not sure if we can provide useful stats there.
>
> Can anyone chime in and describe what SET is for? Sergi, do you know?
>
>
> > 2) What metrics I could collect? For now I could collect per query:
> >   Number of executions.
> >   Number of failures.
> >   Duration.
> > Is there something else I could collect?
> >
>
> The metrics you selected look like a good starting point. I would also add
> execution plan to it, if possible.
>
>
> >
> >
> > 3) Configuration.
> >   Should we have this feature configured per cache or globally on
> > IgniteConfiguration?
> >   Should this feature be ON or OFF by default?
> >   How many items I could collect by default? 100? 1000?
> >
>
> How about both? There should be a global setting which can be overridden on
> per-cache basis.
>
>
> >
> > Any comments are welcome.
> >
> > --
> > https://issues.apache.org/jira/browse/IGNITE-3443
> >
> > --
> > Alexey Kuznetsov
> >
>

[GitHub] ignite pull request #895: IGNITE-3323 Use maven dependencies for Jetbrains a...

[jayho]

GitHub user jayho opened a pull request:

https://github.com/apache/ignite/pull/895

IGNITE-3323 Use maven dependencies for Jetbrains annotations



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jayho/ignite ignite-3323

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/895.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #895


commit 4fe63750a0130d2c8b854a82ec6547a75d37d14b
Author: Jens Hoffmann 
Date:   2016-07-25T22:38:13Z

IGNITE-3323 Use maven dependencies for Jetbrains annotations




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Implement collecting what SQL statements executed on cluster and their metrics.

[Dmitriy Setrakyan]

My comments are below...

On Thu, Jul 7, 2016 at 9:41 AM, Alexey Kuznetsov 
wrote:

> Hi, All!
>
> I'm working on issue "Implement collecting what SQL statements executed on
> cluster and their metrics" [1]
>
> For now I'm investigating code and have following questions:
>
> 1) We have different types of queries:  GridCacheQueryType: SPI, SCAN, SQL,
> SQL_FIELDS, TEXT, SET
> Should I implement collecting metrics for ALL of them?
> Or for example I could collect:
>SCAN: cache name
>SQL: type name + SQL clauseю
>SQL_FIELDS: SQL statement text
>TEXT: type + searched string
>
> What I could collect for SPI and SET query types? Or I may ignore them?
>

I would implement stat gathering for all the known ones. SPI is user
defined type, so  I am not sure if we can provide useful stats there.

Can anyone chime in and describe what SET is for? Sergi, do you know?


> 2) What metrics I could collect? For now I could collect per query:
>   Number of executions.
>   Number of failures.
>   Duration.
> Is there something else I could collect?
>

The metrics you selected look like a good starting point. I would also add
execution plan to it, if possible.


>
>
> 3) Configuration.
>   Should we have this feature configured per cache or globally on
> IgniteConfiguration?
>   Should this feature be ON or OFF by default?
>   How many items I could collect by default? 100? 1000?
>

How about both? There should be a global setting which can be overridden on
per-cache basis.


>
> Any comments are welcome.
>
> --
> https://issues.apache.org/jira/browse/IGNITE-3443
>
> --
> Alexey Kuznetsov
>

Re: Data compression in Ignite 2.0

[Dmitriy Setrakyan]

Nikita, this sounds like a pretty elegant approach.

Does anyone in the community see a problem with this design?

On Mon, Jul 25, 2016 at 4:49 PM, Nikita Ivanov  wrote:

> SAP Hana does the compression by 1) compressing SQL parameters before
> execution, and 2) storing only compressed data in memory. This way all SQL
> queries work as normal with zero modifications or performance overhead.
> Only results of the query can be (optionally) decompressed back before
> returning to the user.
>
> --
> Nikita Ivanov
>
>
> On Mon, Jul 25, 2016 at 1:40 PM, Sergey Kozlov 
> wrote:
>
> > Hi
> >
> > For approach 1: Put a large object into a partiton cache will force to
> > update the dictionary placed on replication cache. It seeis it may be
> > time-expense operation.
> > Appoach 2-3 are make sense for rare cases as Sergi commented.
> > Aslo I see a danger of OOM if we've got high compression level and try to
> > restore orginal value in memory.
> >
> > On Mon, Jul 25, 2016 at 10:39 AM, Alexey Kuznetsov <
> > akuznet...@gridgain.com>
> > wrote:
> >
> > > Sergi,
> > >
> > > Of course it will introduce some slowdown, but with compression more
> data
> > > could be stored in memory
> > > and not will be evicted to disk. In case of compress by dictionary
> > > substitution it will be only one more lookup
> > > and should be fast.
> > >
> > > In general we could provide only API for compression out of the box,
> and
> > > users that really need some sort of compression
> > > will implement it by them self. This will not require much effort I
> > think.
> > >
> > >
> > >
> > > On Mon, Jul 25, 2016 at 2:18 PM, Sergi Vladykin <
> > sergi.vlady...@gmail.com>
> > > wrote:
> > >
> > > > This will make sense only for rare cases when you have very large
> > objects
> > > > stored, which can be effectively compressed. And even then it will
> > > > introduce slowdown on all the operations, which often will not be
> > > > acceptable. I guess only few users will find this feature useful,
> thus
> > I
> > > > think it does not worth the effort.
> > > >
> > > > Sergi
> > > >
> > > >
> > > >
> > > > 2016-07-25 9:28 GMT+03:00 Alexey Kuznetsov  >:
> > > >
> > > > > Hi, All!
> > > > >
> > > > > I would like to propose one more feature for Ignite 2.0.
> > > > >
> > > > > Data compression for data in binary format.
> > > > >
> > > > > Binary format is stored as field name + field data.
> > > > > So we have a description.
> > > > > How about to add one more byte to binary data descriptor:
> > > > >
> > > > > *Compressed*:
> > > > >  0 - Data stored as is (no compression).
> > > > >  1 - Data compressed by dictionary (something like DB2 row
> > compression
> > > > [1],
> > > > >  but for all binary types). We could have system or user defined
> > > > replicated
> > > > > cache for such dictionary and *cache.compact()* method that will
> scan
> > > > > cache, build dictionary and compact data.
> > > > >  2 - Data compressed by Java built in ZIP.
> > > > >  3 - Data compressed by some user custom algorithm.
> > > > >
> > > > > Of course it is possible to compress data in current Ignite 1.x but
> > in
> > > > this
> > > > > case compressed data cannot be accessed from SQL engine, if we
> > > implement
> > > > > support for compression on Ignite core level SQL engine will be
> able
> > to
> > > > > detect that data is compressed and properly handle such data.
> > > > >
> > > > > What do you think?
> > > > > If community consider this feature useful I will create issue in
> > JIRA.
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> http://www.ibm.com/developerworks/data/library/techarticle/dm-1205db210compression/
> > > > >
> > > > > --
> > > > > Alexey Kuznetsov
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Alexey Kuznetsov
> > >
> >
> >
> >
> > --
> > Sergey Kozlov
> > GridGain Systems
> > www.gridgain.com
> >
>

Re: Data compression in Ignite 2.0

[Nikita Ivanov]

SAP Hana does the compression by 1) compressing SQL parameters before
execution, and 2) storing only compressed data in memory. This way all SQL
queries work as normal with zero modifications or performance overhead.
Only results of the query can be (optionally) decompressed back before
returning to the user.

--
Nikita Ivanov


On Mon, Jul 25, 2016 at 1:40 PM, Sergey Kozlov  wrote:

> Hi
>
> For approach 1: Put a large object into a partiton cache will force to
> update the dictionary placed on replication cache. It seeis it may be
> time-expense operation.
> Appoach 2-3 are make sense for rare cases as Sergi commented.
> Aslo I see a danger of OOM if we've got high compression level and try to
> restore orginal value in memory.
>
> On Mon, Jul 25, 2016 at 10:39 AM, Alexey Kuznetsov <
> akuznet...@gridgain.com>
> wrote:
>
> > Sergi,
> >
> > Of course it will introduce some slowdown, but with compression more data
> > could be stored in memory
> > and not will be evicted to disk. In case of compress by dictionary
> > substitution it will be only one more lookup
> > and should be fast.
> >
> > In general we could provide only API for compression out of the box, and
> > users that really need some sort of compression
> > will implement it by them self. This will not require much effort I
> think.
> >
> >
> >
> > On Mon, Jul 25, 2016 at 2:18 PM, Sergi Vladykin <
> sergi.vlady...@gmail.com>
> > wrote:
> >
> > > This will make sense only for rare cases when you have very large
> objects
> > > stored, which can be effectively compressed. And even then it will
> > > introduce slowdown on all the operations, which often will not be
> > > acceptable. I guess only few users will find this feature useful, thus
> I
> > > think it does not worth the effort.
> > >
> > > Sergi
> > >
> > >
> > >
> > > 2016-07-25 9:28 GMT+03:00 Alexey Kuznetsov :
> > >
> > > > Hi, All!
> > > >
> > > > I would like to propose one more feature for Ignite 2.0.
> > > >
> > > > Data compression for data in binary format.
> > > >
> > > > Binary format is stored as field name + field data.
> > > > So we have a description.
> > > > How about to add one more byte to binary data descriptor:
> > > >
> > > > *Compressed*:
> > > >  0 - Data stored as is (no compression).
> > > >  1 - Data compressed by dictionary (something like DB2 row
> compression
> > > [1],
> > > >  but for all binary types). We could have system or user defined
> > > replicated
> > > > cache for such dictionary and *cache.compact()* method that will scan
> > > > cache, build dictionary and compact data.
> > > >  2 - Data compressed by Java built in ZIP.
> > > >  3 - Data compressed by some user custom algorithm.
> > > >
> > > > Of course it is possible to compress data in current Ignite 1.x but
> in
> > > this
> > > > case compressed data cannot be accessed from SQL engine, if we
> > implement
> > > > support for compression on Ignite core level SQL engine will be able
> to
> > > > detect that data is compressed and properly handle such data.
> > > >
> > > > What do you think?
> > > > If community consider this feature useful I will create issue in
> JIRA.
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> http://www.ibm.com/developerworks/data/library/techarticle/dm-1205db210compression/
> > > >
> > > > --
> > > > Alexey Kuznetsov
> > > >
> > >
> >
> >
> >
> > --
> > Alexey Kuznetsov
> >
>
>
>
> --
> Sergey Kozlov
> GridGain Systems
> www.gridgain.com
>

Re: Data compression in Ignite 2.0

[Sergey Kozlov]

Hi

For approach 1: Put a large object into a partiton cache will force to
update the dictionary placed on replication cache. It seeis it may be
time-expense operation.
Appoach 2-3 are make sense for rare cases as Sergi commented.
Aslo I see a danger of OOM if we've got high compression level and try to
restore orginal value in memory.

On Mon, Jul 25, 2016 at 10:39 AM, Alexey Kuznetsov 
wrote:

> Sergi,
>
> Of course it will introduce some slowdown, but with compression more data
> could be stored in memory
> and not will be evicted to disk. In case of compress by dictionary
> substitution it will be only one more lookup
> and should be fast.
>
> In general we could provide only API for compression out of the box, and
> users that really need some sort of compression
> will implement it by them self. This will not require much effort I think.
>
>
>
> On Mon, Jul 25, 2016 at 2:18 PM, Sergi Vladykin 
> wrote:
>
> > This will make sense only for rare cases when you have very large objects
> > stored, which can be effectively compressed. And even then it will
> > introduce slowdown on all the operations, which often will not be
> > acceptable. I guess only few users will find this feature useful, thus I
> > think it does not worth the effort.
> >
> > Sergi
> >
> >
> >
> > 2016-07-25 9:28 GMT+03:00 Alexey Kuznetsov :
> >
> > > Hi, All!
> > >
> > > I would like to propose one more feature for Ignite 2.0.
> > >
> > > Data compression for data in binary format.
> > >
> > > Binary format is stored as field name + field data.
> > > So we have a description.
> > > How about to add one more byte to binary data descriptor:
> > >
> > > *Compressed*:
> > >  0 - Data stored as is (no compression).
> > >  1 - Data compressed by dictionary (something like DB2 row compression
> > [1],
> > >  but for all binary types). We could have system or user defined
> > replicated
> > > cache for such dictionary and *cache.compact()* method that will scan
> > > cache, build dictionary and compact data.
> > >  2 - Data compressed by Java built in ZIP.
> > >  3 - Data compressed by some user custom algorithm.
> > >
> > > Of course it is possible to compress data in current Ignite 1.x but in
> > this
> > > case compressed data cannot be accessed from SQL engine, if we
> implement
> > > support for compression on Ignite core level SQL engine will be able to
> > > detect that data is compressed and properly handle such data.
> > >
> > > What do you think?
> > > If community consider this feature useful I will create issue in JIRA.
> > >
> > > [1]
> > >
> > >
> >
> http://www.ibm.com/developerworks/data/library/techarticle/dm-1205db210compression/
> > >
> > > --
> > > Alexey Kuznetsov
> > >
> >
>
>
>
> --
> Alexey Kuznetsov
>



-- 
Sergey Kozlov
GridGain Systems
www.gridgain.com

[GitHub] ignite pull request #894: IGNITE-3578: Added support of DistributedJoins to ...

[isapego]

GitHub user isapego opened a pull request:

https://github.com/apache/ignite/pull/894

IGNITE-3578: Added support of DistributedJoins to SqlQuery and

SqlFieldsQuery.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/isapego/ignite ignite-3578

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/894.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #894


commit a38a15f2026cea1a796cd02ef11e4cd8b11869bd
Author: isapego 
Date:   2016-07-25T17:22:19Z

IGNITE-3578: Added support of DistributedJoins to SqlQuery and
SqlFieldsQuery.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-3578) CPP: Add support of distributed joins in SqlQuery & SqlFieldsQuery

[Igor Sapego (JIRA)]

Igor Sapego created IGNITE-3578:
---

 Summary: CPP: Add support of distributed joins in SqlQuery & 
SqlFieldsQuery
 Key: IGNITE-3578
 URL: https://issues.apache.org/jira/browse/IGNITE-3578
 Project: Ignite
  Issue Type: Task
  Components: platforms
Affects Versions: 1.6
Reporter: Igor Sapego
Assignee: Igor Sapego
 Fix For: 1.7


IGNITE-1232 added new properties to {{SqlQuery}} and {{SqlFieldsQuery}} 
classes. Propagate them to C++.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Distributed joins for JDBC

[Pavel Tupitsyn]

Added properties to .NET (https://issues.apache.org/jira/browse/IGNITE-3561
).

On Mon, Jul 25, 2016 at 1:23 PM, Andrey Gura  wrote:

> Implemented. See corresponding ticket.
>
> Not supported by Java client based driver because it is deprecated and does
> not support any SqlFieldQuery properties. If we want support this property
> (and may be other, like local and collocated) in Java client based driver
> then we should implement it as separate task.
>
> On Mon, Jul 25, 2016 at 10:32 AM, Andrey Gura  wrote:
>
> > Our JDBC drivers already have connection properties that correspond to
> > SqlFieldsQuery properties. So we can just add support of this parameter
> to
> > connection string parser.
> >
> > Corresponding ticket created IGNITE-3563 (
> > https://issues.apache.org/jira/browse/IGNITE-3563 ).
> >
> > On Mon, Jul 25, 2016 at 10:04 AM, Sergi Vladykin <
> sergi.vlady...@gmail.com
> > > wrote:
> >
> >> I don't think it makes sense to extend JDBC this way because usually if
> >> one
> >> have access to Java API he most probably will use Ignite API. If for
> some
> >> reason they use JDBC it means that it is an application which was aimed
> to
> >> work with any RDBMS and should not know about quirks of some particular
> >> driver. Take any JDBC based SQL console for example, we have to support
> >> them out of the box.
> >>
> >> I think we should have a connection options which we can append to JDBC
> >> URL
> >> like it is done in H2:
> >>
> >> jdbc:h2:my_database;OPTION1=bla;OPTION2=blabla
> >>
> >> In our case it must be something like DISTRIBUTED_JOINS=true and it will
> >> affect the whole connection.
> >>
> >> Of course we have to support simultaneous connections to the same DB
> with
> >> different options.
> >>
> >> Sergi
> >>
> >>
> >> 2016-07-25 9:19 GMT+03:00 Semyon Boikov :
> >>
> >> > Hi,
> >> >
> >> > Last week distributed joins functionality was merged, but one thing
> was
> >> > overlooked. Distributed joins should be explicitly enabled using using
> >> > method 'setDistributedJoins' available in java API
> >> > (SqlQuery/SqlFieldsQuery). First, this parameter should be also added
> in
> >> > .Net/C++/REST API, this is straightforward. Also there should be
> >> > possibility to enable distributed joins for JDBC API. Does it make
> >> sense to
> >> > add Ignite-specific interface extending standard java.sql.Statement,
> so
> >> > 'setDistributedJoins' method can be added there.
> >> > JDBC API already have 'unwrap' method to deal with vendor-specific
> >> > interfaces, code will look like this:
> >> > * IgniteStatement stmt =
> >> > connection.createStatement().unwrap(IgniteStatement.class);*
> >> > * stmt.setDistributedJoins(true);*
> >> > *stmt.executeQuery("...");*
> >> >
> >> > What do you think?
> >> >
> >>
> >
> >
> >
> > --
> > Andrey Gura
> > GridGain Systems, Inc.
> > www.gridgain.com
> >
>
>
>
> --
> Andrey Gura
> GridGain Systems, Inc.
> www.gridgain.com
>

[GitHub] ignite pull request #893: IGNITE-3561 .NET: DistributedJoins property in Sql...

[ptupitsyn]

Github user ptupitsyn closed the pull request at:

https://github.com/apache/ignite/pull/893


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #893: IGNITE-3561 .NET: DistributedJoins property in Sql...

[ptupitsyn]

GitHub user ptupitsyn opened a pull request:

https://github.com/apache/ignite/pull/893

IGNITE-3561 .NET: DistributedJoins property in SqlQuery & SqlFieldsQuery



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ptupitsyn/ignite ignite-3561

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/893.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #893


commit fcad83052a5249ff531b29b8edf2e3c113a657d7
Author: Pavel Tupitsyn 
Date:   2016-07-25T13:42:58Z

Java side

commit 64f3c97bd2c4cc7b12c22d7dec1e4f82f70a2556
Author: Pavel Tupitsyn 
Date:   2016-07-25T13:55:53Z

Propagate

commit 182bf9355249222175d598791b38222fda1ff2b7
Author: Pavel Tupitsyn 
Date:   2016-07-25T13:56:54Z

revert example

commit 251104ef070efe1f1b8635907339ad94caf120c7
Author: Pavel Tupitsyn 
Date:   2016-07-25T14:07:40Z

wip LINQ

commit d28d93b516737f49269297eb0bfc55591a6bba62
Author: Pavel Tupitsyn 
Date:   2016-07-25T14:12:21Z

Propagate to LINQ

commit 97f3bd40aa48ce92d25c26cb315c2303525be2dc
Author: Pavel Tupitsyn 
Date:   2016-07-25T14:15:03Z

linq test

commit 9879eb6a7cc7a30345e74030a571061ed68571b9
Author: Pavel Tupitsyn 
Date:   2016-07-25T14:25:15Z

wip

commit 20755817f35744d7eb1e0255e0d560797cf1b983
Author: Pavel Tupitsyn 
Date:   2016-07-25T14:26:18Z

LINQ test done

commit b69a1363baff451401bdab52931bd90049838727
Author: Pavel Tupitsyn 
Date:   2016-07-25T14:32:22Z

more tests

commit a9607d375ce5704665d554701eac02dfef006398
Author: Pavel Tupitsyn 
Date:   2016-07-25T14:33:16Z

tests done




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #892: IGNITE-3504 .NET: Fix IBinaryObjectBuilder setter ...

[ptupitsyn]

Github user ptupitsyn closed the pull request at:

https://github.com/apache/ignite/pull/892


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-3577) ODBC: Replace "Server" and "Port" connection string arguments with single argument "Address"

[Igor Sapego (JIRA)]

Igor Sapego created IGNITE-3577:
---

 Summary: ODBC: Replace "Server" and "Port" connection string 
arguments with single argument "Address"
 Key: IGNITE-3577
 URL: https://issues.apache.org/jira/browse/IGNITE-3577
 Project: Ignite
  Issue Type: Task
  Components: odbc
Affects Versions: 1.6
Reporter: Igor Sapego
Assignee: Igor Sapego
 Fix For: 1.7


Replace {{Server}} and {{Port}} connection arguments with single argument 
{{Address}} with the following syntax: {{Address=host[:port]}}. Where {{:port}} 
part is optional. When port is not specified, default ODBC port is used.

These changes are needed as a first step for adding failover support in future, 
when we will be able to specify several nodes in the connection string.  E.g. 
{{Address=host1:port1|host2:port2|host3:port3..port4}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] ignite pull request #892: IGNITE-3504 .NET: Fix IBinaryObjectBuilder setter ...

[ptupitsyn]

GitHub user ptupitsyn opened a pull request:

https://github.com/apache/ignite/pull/892

IGNITE-3504 .NET: Fix IBinaryObjectBuilder setter methods



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ptupitsyn/ignite ignite-3504

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/892.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #892


commit 6ab2eff21b8662f5063beb56e9985eb09efc0b0a
Author: Pavel Tupitsyn 
Date:   2016-07-22T12:38:46Z

IGNITE-3504 .NET: Improve IBinaryObjectBuilder test coverage

commit 4a05f547f4d690b804faff5709fb567fa0a93c65
Author: Pavel Tupitsyn 
Date:   2016-07-22T16:06:28Z

wip

commit 2e3380fe1d4cf9b3471c3f47589a2d46ae6b4edd
Author: Pavel Tupitsyn 
Date:   2016-07-25T10:13:41Z

Merge remote-tracking branch 'remotes/upstream/master' into ignite-3504

commit a2e26b97e64a1e5e4e6dbf3dc189e10b4850923d
Author: Pavel Tupitsyn 
Date:   2016-07-25T10:43:14Z

wip

commit d16c68435560015ff960a3f664e7a6cae748763f
Author: Pavel Tupitsyn 
Date:   2016-07-25T10:47:50Z

wip

commit e16699889dd4a4aa82d907d2856b3b1e53cb1110
Author: Pavel Tupitsyn 
Date:   2016-07-25T10:55:19Z

wip array tests

commit 0f9f45adbf62140d405a08beb982ba210eebc7a3
Author: Pavel Tupitsyn 
Date:   2016-07-25T11:02:56Z

wip

commit 68cc59f9a0abd276d84d21f20956068d63aa5e84
Author: Pavel Tupitsyn 
Date:   2016-07-25T11:05:57Z

wip

commit 0144f1b2ab9ab452be106fcfcc3ff1c8f48fe1e7
Author: Pavel Tupitsyn 
Date:   2016-07-25T11:16:41Z

fix test

commit ad4ee63b86a7f2657b59990195c57b33abf45da8
Author: Pavel Tupitsyn 
Date:   2016-07-25T11:29:27Z

wip

commit 6d97d7202431b8f209585552fd5e49c5a42f895d
Author: Pavel Tupitsyn 
Date:   2016-07-25T11:43:28Z

wip tests

commit 0eec10babae23be9898687b861d55fce1811bbdb
Author: Pavel Tupitsyn 
Date:   2016-07-25T11:50:07Z

wip tests

commit e00b5c83d0db9b6027396040f1fa086147688471
Author: Pavel Tupitsyn 
Date:   2016-07-25T11:53:22Z

tests done, IBinaryObjectBuilder is fully covered




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #891: ignite-3573 Support of Distributed joins queries i...

[agura]

GitHub user agura opened a pull request:

https://github.com/apache/ignite/pull/891

ignite-3573 Support of Distributed joins queries in REST API



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/agura/incubator-ignite ignite-3573

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/891.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #891


commit 7fa7a41b758f445d5d7f2e8a5517e1f44c657990
Author: agura 
Date:   2016-07-25T11:31:10Z

ignite-3573 Support of Distributed joins queries in REST API




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-3576) .NET: Implement distributed data structures.

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-3576:
---

 Summary: .NET: Implement distributed data structures.
 Key: IGNITE-3576
 URL: https://issues.apache.org/jira/browse/IGNITE-3576
 Project: Ignite
  Issue Type: Task
  Components: platforms
Affects Versions: 1.6
Reporter: Vladimir Ozerov


Umbrella ticket to host data structures efforts.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3575) CPP: Support continuous queries remote filters.

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-3575:
---

 Summary: CPP: Support continuous queries remote filters.
 Key: IGNITE-3575
 URL: https://issues.apache.org/jira/browse/IGNITE-3575
 Project: Ignite
  Issue Type: Task
  Components: platforms
Affects Versions: 1.6
Reporter: Vladimir Ozerov






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3574) CPP: Implement compute API

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-3574:
---

 Summary: CPP: Implement compute API
 Key: IGNITE-3574
 URL: https://issues.apache.org/jira/browse/IGNITE-3574
 Project: Ignite
  Issue Type: Task
  Components: platforms
Affects Versions: 1.6
Reporter: Vladimir Ozerov
Priority: Critical


Umbrella ticket to host all compute API tickets.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3573) Support distributed joins queries in REST API

[Andrey Gura (JIRA)]

Andrey Gura created IGNITE-3573:
---

 Summary: Support distributed joins queries in REST API
 Key: IGNITE-3573
 URL: https://issues.apache.org/jira/browse/IGNITE-3573
 Project: Ignite
  Issue Type: Improvement
Reporter: Andrey Gura
Assignee: Andrey Gura
Priority: Minor
 Fix For: 1.7


REST API commands {{qryfldexe}} and {{qryexe}} should support 
{{distributedJoins}} flag.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3572) CPP: Improve binary marshaller capabilities.

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-3572:
---

 Summary: CPP: Improve binary marshaller capabilities.
 Key: IGNITE-3572
 URL: https://issues.apache.org/jira/browse/IGNITE-3572
 Project: Ignite
  Issue Type: Task
  Components: platforms
Affects Versions: 1.6
Reporter: Vladimir Ozerov


This is umbrella ticket to host all binary marshaller tickets.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3571) CPP: Improve Cache API.

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-3571:
---

 Summary: CPP: Improve Cache API.
 Key: IGNITE-3571
 URL: https://issues.apache.org/jira/browse/IGNITE-3571
 Project: Ignite
  Issue Type: Task
  Components: platforms
Affects Versions: 1.6
Reporter: Vladimir Ozerov
Priority: Critical


This is umbrella ticket to manage ongoing C++ cache API efforts.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3570) .NET: Rename Boolean to Bool

[Pavel Tupitsyn (JIRA)]

Pavel Tupitsyn created IGNITE-3570:
--

 Summary: .NET: Rename Boolean to Bool
 Key: IGNITE-3570
 URL: https://issues.apache.org/jira/browse/IGNITE-3570
 Project: Ignite
  Issue Type: Improvement
  Components: platforms
Reporter: Pavel Tupitsyn
 Fix For: 2.0


Currently there is an inconsistency, some members are Bool, some are Boolean. 

Let's rename everything to Bool, since it's shorter, and system type alias is 
`bool`.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Distributed joins for JDBC

[Andrey Gura]

Implemented. See corresponding ticket.

Not supported by Java client based driver because it is deprecated and does
not support any SqlFieldQuery properties. If we want support this property
(and may be other, like local and collocated) in Java client based driver
then we should implement it as separate task.

On Mon, Jul 25, 2016 at 10:32 AM, Andrey Gura  wrote:

> Our JDBC drivers already have connection properties that correspond to
> SqlFieldsQuery properties. So we can just add support of this parameter to
> connection string parser.
>
> Corresponding ticket created IGNITE-3563 (
> https://issues.apache.org/jira/browse/IGNITE-3563 ).
>
> On Mon, Jul 25, 2016 at 10:04 AM, Sergi Vladykin  > wrote:
>
>> I don't think it makes sense to extend JDBC this way because usually if
>> one
>> have access to Java API he most probably will use Ignite API. If for some
>> reason they use JDBC it means that it is an application which was aimed to
>> work with any RDBMS and should not know about quirks of some particular
>> driver. Take any JDBC based SQL console for example, we have to support
>> them out of the box.
>>
>> I think we should have a connection options which we can append to JDBC
>> URL
>> like it is done in H2:
>>
>> jdbc:h2:my_database;OPTION1=bla;OPTION2=blabla
>>
>> In our case it must be something like DISTRIBUTED_JOINS=true and it will
>> affect the whole connection.
>>
>> Of course we have to support simultaneous connections to the same DB with
>> different options.
>>
>> Sergi
>>
>>
>> 2016-07-25 9:19 GMT+03:00 Semyon Boikov :
>>
>> > Hi,
>> >
>> > Last week distributed joins functionality was merged, but one thing was
>> > overlooked. Distributed joins should be explicitly enabled using using
>> > method 'setDistributedJoins' available in java API
>> > (SqlQuery/SqlFieldsQuery). First, this parameter should be also added in
>> > .Net/C++/REST API, this is straightforward. Also there should be
>> > possibility to enable distributed joins for JDBC API. Does it make
>> sense to
>> > add Ignite-specific interface extending standard java.sql.Statement, so
>> > 'setDistributedJoins' method can be added there.
>> > JDBC API already have 'unwrap' method to deal with vendor-specific
>> > interfaces, code will look like this:
>> > * IgniteStatement stmt =
>> > connection.createStatement().unwrap(IgniteStatement.class);*
>> > * stmt.setDistributedJoins(true);*
>> > *stmt.executeQuery("...");*
>> >
>> > What do you think?
>> >
>>
>
>
>
> --
> Andrey Gura
> GridGain Systems, Inc.
> www.gridgain.com
>



-- 
Andrey Gura
GridGain Systems, Inc.
www.gridgain.com

[GitHub] ignite pull request #890: IGNITE-3569 - Set copyOnRead property to FALSE for...

[dkarachentsev]

GitHub user dkarachentsev opened a pull request:

https://github.com/apache/ignite/pull/890

IGNITE-3569 - Set copyOnRead property to FALSE for marshaller cache



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-3569

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/890.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #890


commit bd0a9c928798b951f7c4ef833f164178495daa9d
Author: dkarachentsev 
Date:   2016-07-25T10:13:14Z

IGNITE-3569 - Set copyOnRead property to FALSE for marshaller cache




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #889: ignite-3563 Support distributedJoins flag in JDBC ...

[agura]

GitHub user agura opened a pull request:

https://github.com/apache/ignite/pull/889

ignite-3563 Support distributedJoins flag in JDBC driver



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/agura/incubator-ignite ignite-3563

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/889.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #889


commit c8dad9dc9092d360bdef948c801ecbd83f479024
Author: agura 
Date:   2016-07-25T09:31:24Z

ignite-3563 Support distributedJoins flag in JDBC driver




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-3569) Set copyOnRead property to FALSE for marshaller cache

[Yakov Zhdanov (JIRA)]

Yakov Zhdanov created IGNITE-3569:
-

 Summary: Set copyOnRead property to FALSE for marshaller cache
 Key: IGNITE-3569
 URL: https://issues.apache.org/jira/browse/IGNITE-3569
 Project: Ignite
  Issue Type: Improvement
Reporter: Yakov Zhdanov
 Fix For: 1.7


Ignite unmarshalls type metadata when accessing it which is an overhead. Need 
to fix this for marshaller cache and, probably, for other internal caches (this 
needs to be investigated).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3568) .NET: Start JVM externally.

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-3568:
---

 Summary: .NET: Start JVM externally.
 Key: IGNITE-3568
 URL: https://issues.apache.org/jira/browse/IGNITE-3568
 Project: Ignite
  Issue Type: Task
  Components: platforms
Affects Versions: 1.6
Reporter: Vladimir Ozerov
Priority: Critical


Currently we start JVM inside .NET process. This is not good for several 
reasons:
1) Broken isolation - only one JVM can exist per process. This way it is 
impossible to start two Ignite instances with different JVM options.
2) JVM startup is expensive, cluster connection is expensive, and process must 
host both Java and .NET heaps. Should we have external JVM to connect to, we 
would allow for truly thin clients, when dozens thin processes will be able to 
work with the same client. We already see growing demand for this feature,



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Implement collecting what SQL statements executed on cluster and their metrics.

[Alexey Kuznetsov]

Hi All!

Just kindly remind for feedback.

Thanks!

On Thu, Jul 7, 2016 at 8:41 PM, Alexey Kuznetsov 
wrote:

> Hi, All!
>
> I'm working on issue "Implement collecting what SQL statements executed on
> cluster and their metrics" [1]
>
> For now I'm investigating code and have following questions:
>
> 1) We have different types of queries:  GridCacheQueryType: SPI, SCAN,
> SQL, SQL_FIELDS, TEXT, SET
> Should I implement collecting metrics for ALL of them?
> Or for example I could collect:
>SCAN: cache name
>SQL: type name + SQL clauseю
>SQL_FIELDS: SQL statement text
>TEXT: type + searched string
>
> What I could collect for SPI and SET query types? Or I may ignore them?
>
> 2) What metrics I could collect? For now I could collect per query:
>   Number of executions.
>   Number of failures.
>   Duration.
> Is there something else I could collect?
>
>
> 3) Configuration.
>   Should we have this feature configured per cache or globally on
> IgniteConfiguration?
>   Should this feature be ON or OFF by default?
>   How many items I could collect by default? 100? 1000?
>
> Any comments are welcome.
>
> --
> https://issues.apache.org/jira/browse/IGNITE-3443
>
> --
> Alexey Kuznetsov
>
>


-- 
Alexey Kuznetsov
GridGain Systems
www.gridgain.com

[jira] [Created] (IGNITE-3566) .NET: Memcached integration.

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-3566:
---

 Summary: .NET: Memcached integration.
 Key: IGNITE-3566
 URL: https://issues.apache.org/jira/browse/IGNITE-3566
 Project: Ignite
  Issue Type: Task
  Components: platforms
Affects Versions: 1.6
Reporter: Vladimir Ozerov






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3567) .NET: AppFabric integration.

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-3567:
---

 Summary: .NET: AppFabric integration.
 Key: IGNITE-3567
 URL: https://issues.apache.org/jira/browse/IGNITE-3567
 Project: Ignite
  Issue Type: Task
  Components: platforms
Affects Versions: 1.6
Reporter: Vladimir Ozerov






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Distributed joins for JDBC

[Sergi Vladykin]

Pavel,

There are two reasons for this:
1. This setting may affect performance for all the cases you mentioned,
because it does much more complex query optimization.
2. This feature is quite new and I want to release it in the safest
possible way because we may just brake existing applications if we enable
it always.

Andrey,

Good news!

Sergi




2016-07-25 10:32 GMT+03:00 Andrey Gura :

> Our JDBC drivers already have connection properties that correspond to
> SqlFieldsQuery properties. So we can just add support of this parameter to
> connection string parser.
>
> Corresponding ticket created IGNITE-3563 (
> https://issues.apache.org/jira/browse/IGNITE-3563 ).
>
> On Mon, Jul 25, 2016 at 10:04 AM, Sergi Vladykin  >
> wrote:
>
> > I don't think it makes sense to extend JDBC this way because usually if
> one
> > have access to Java API he most probably will use Ignite API. If for some
> > reason they use JDBC it means that it is an application which was aimed
> to
> > work with any RDBMS and should not know about quirks of some particular
> > driver. Take any JDBC based SQL console for example, we have to support
> > them out of the box.
> >
> > I think we should have a connection options which we can append to JDBC
> URL
> > like it is done in H2:
> >
> > jdbc:h2:my_database;OPTION1=bla;OPTION2=blabla
> >
> > In our case it must be something like DISTRIBUTED_JOINS=true and it will
> > affect the whole connection.
> >
> > Of course we have to support simultaneous connections to the same DB with
> > different options.
> >
> > Sergi
> >
> >
> > 2016-07-25 9:19 GMT+03:00 Semyon Boikov :
> >
> > > Hi,
> > >
> > > Last week distributed joins functionality was merged, but one thing was
> > > overlooked. Distributed joins should be explicitly enabled using using
> > > method 'setDistributedJoins' available in java API
> > > (SqlQuery/SqlFieldsQuery). First, this parameter should be also added
> in
> > > .Net/C++/REST API, this is straightforward. Also there should be
> > > possibility to enable distributed joins for JDBC API. Does it make
> sense
> > to
> > > add Ignite-specific interface extending standard java.sql.Statement, so
> > > 'setDistributedJoins' method can be added there.
> > > JDBC API already have 'unwrap' method to deal with vendor-specific
> > > interfaces, code will look like this:
> > > * IgniteStatement stmt =
> > > connection.createStatement().unwrap(IgniteStatement.class);*
> > > * stmt.setDistributedJoins(true);*
> > > *stmt.executeQuery("...");*
> > >
> > > What do you think?
> > >
> >
>
>
>
> --
> Andrey Gura
> GridGain Systems, Inc.
> www.gridgain.com
>

Re: Distributed joins for JDBC

[Alexey Kuznetsov]

Why we need to *TURN ON/OFF* this mode?
Why not have it always *ON*?

On Mon, Jul 25, 2016 at 2:32 PM, Andrey Gura  wrote:

> Our JDBC drivers already have connection properties that correspond to
> SqlFieldsQuery properties. So we can just add support of this parameter to
> connection string parser.
>
> Corresponding ticket created IGNITE-3563 (
> https://issues.apache.org/jira/browse/IGNITE-3563 ).
>
> On Mon, Jul 25, 2016 at 10:04 AM, Sergi Vladykin  >
> wrote:
>
> > I don't think it makes sense to extend JDBC this way because usually if
> one
> > have access to Java API he most probably will use Ignite API. If for some
> > reason they use JDBC it means that it is an application which was aimed
> to
> > work with any RDBMS and should not know about quirks of some particular
> > driver. Take any JDBC based SQL console for example, we have to support
> > them out of the box.
> >
> > I think we should have a connection options which we can append to JDBC
> URL
> > like it is done in H2:
> >
> > jdbc:h2:my_database;OPTION1=bla;OPTION2=blabla
> >
> > In our case it must be something like DISTRIBUTED_JOINS=true and it will
> > affect the whole connection.
> >
> > Of course we have to support simultaneous connections to the same DB with
> > different options.
> >
> > Sergi
> >
> >
> > 2016-07-25 9:19 GMT+03:00 Semyon Boikov :
> >
> > > Hi,
> > >
> > > Last week distributed joins functionality was merged, but one thing was
> > > overlooked. Distributed joins should be explicitly enabled using using
> > > method 'setDistributedJoins' available in java API
> > > (SqlQuery/SqlFieldsQuery). First, this parameter should be also added
> in
> > > .Net/C++/REST API, this is straightforward. Also there should be
> > > possibility to enable distributed joins for JDBC API. Does it make
> sense
> > to
> > > add Ignite-specific interface extending standard java.sql.Statement, so
> > > 'setDistributedJoins' method can be added there.
> > > JDBC API already have 'unwrap' method to deal with vendor-specific
> > > interfaces, code will look like this:
> > > * IgniteStatement stmt =
> > > connection.createStatement().unwrap(IgniteStatement.class);*
> > > * stmt.setDistributedJoins(true);*
> > > *stmt.executeQuery("...");*
> > >
> > > What do you think?
> > >
> >
>
>
>
> --
> Andrey Gura
> GridGain Systems, Inc.
> www.gridgain.com
>



-- 
Alexey Kuznetsov
GridGain Systems
www.gridgain.com

Re: Data compression in Ignite 2.0

[Alexey Kuznetsov]

Sergi,

Of course it will introduce some slowdown, but with compression more data
could be stored in memory
and not will be evicted to disk. In case of compress by dictionary
substitution it will be only one more lookup
and should be fast.

In general we could provide only API for compression out of the box, and
users that really need some sort of compression
will implement it by them self. This will not require much effort I think.



On Mon, Jul 25, 2016 at 2:18 PM, Sergi Vladykin 
wrote:

> This will make sense only for rare cases when you have very large objects
> stored, which can be effectively compressed. And even then it will
> introduce slowdown on all the operations, which often will not be
> acceptable. I guess only few users will find this feature useful, thus I
> think it does not worth the effort.
>
> Sergi
>
>
>
> 2016-07-25 9:28 GMT+03:00 Alexey Kuznetsov :
>
> > Hi, All!
> >
> > I would like to propose one more feature for Ignite 2.0.
> >
> > Data compression for data in binary format.
> >
> > Binary format is stored as field name + field data.
> > So we have a description.
> > How about to add one more byte to binary data descriptor:
> >
> > *Compressed*:
> >  0 - Data stored as is (no compression).
> >  1 - Data compressed by dictionary (something like DB2 row compression
> [1],
> >  but for all binary types). We could have system or user defined
> replicated
> > cache for such dictionary and *cache.compact()* method that will scan
> > cache, build dictionary and compact data.
> >  2 - Data compressed by Java built in ZIP.
> >  3 - Data compressed by some user custom algorithm.
> >
> > Of course it is possible to compress data in current Ignite 1.x but in
> this
> > case compressed data cannot be accessed from SQL engine, if we implement
> > support for compression on Ignite core level SQL engine will be able to
> > detect that data is compressed and properly handle such data.
> >
> > What do you think?
> > If community consider this feature useful I will create issue in JIRA.
> >
> > [1]
> >
> >
> http://www.ibm.com/developerworks/data/library/techarticle/dm-1205db210compression/
> >
> > --
> > Alexey Kuznetsov
> >
>



-- 
Alexey Kuznetsov

[jira] [Created] (IGNITE-3565) .NET: ASP.NET view stat cache

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-3565:
---

 Summary: .NET: ASP.NET view stat cache
 Key: IGNITE-3565
 URL: https://issues.apache.org/jira/browse/IGNITE-3565
 Project: Ignite
  Issue Type: Sub-task
  Components: platforms
Affects Versions: 1.6
Reporter: Vladimir Ozerov






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3564) .NET: Integrate with ASP.NET

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-3564:
---

 Summary: .NET: Integrate with ASP.NET
 Key: IGNITE-3564
 URL: https://issues.apache.org/jira/browse/IGNITE-3564
 Project: Ignite
  Issue Type: Task
  Components: platforms
Affects Versions: 1.6
Reporter: Vladimir Ozerov
Priority: Critical


This is an umbrella ticket for all ASP.NET integration sub-tasks.

We need to implement the following integrations:
1) Sessions caching;
2) Output cache provider;
3) View state caching.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Distributed joins for JDBC

[Andrey Gura]

Our JDBC drivers already have connection properties that correspond to
SqlFieldsQuery properties. So we can just add support of this parameter to
connection string parser.

Corresponding ticket created IGNITE-3563 (
https://issues.apache.org/jira/browse/IGNITE-3563 ).

On Mon, Jul 25, 2016 at 10:04 AM, Sergi Vladykin 
wrote:

> I don't think it makes sense to extend JDBC this way because usually if one
> have access to Java API he most probably will use Ignite API. If for some
> reason they use JDBC it means that it is an application which was aimed to
> work with any RDBMS and should not know about quirks of some particular
> driver. Take any JDBC based SQL console for example, we have to support
> them out of the box.
>
> I think we should have a connection options which we can append to JDBC URL
> like it is done in H2:
>
> jdbc:h2:my_database;OPTION1=bla;OPTION2=blabla
>
> In our case it must be something like DISTRIBUTED_JOINS=true and it will
> affect the whole connection.
>
> Of course we have to support simultaneous connections to the same DB with
> different options.
>
> Sergi
>
>
> 2016-07-25 9:19 GMT+03:00 Semyon Boikov :
>
> > Hi,
> >
> > Last week distributed joins functionality was merged, but one thing was
> > overlooked. Distributed joins should be explicitly enabled using using
> > method 'setDistributedJoins' available in java API
> > (SqlQuery/SqlFieldsQuery). First, this parameter should be also added in
> > .Net/C++/REST API, this is straightforward. Also there should be
> > possibility to enable distributed joins for JDBC API. Does it make sense
> to
> > add Ignite-specific interface extending standard java.sql.Statement, so
> > 'setDistributedJoins' method can be added there.
> > JDBC API already have 'unwrap' method to deal with vendor-specific
> > interfaces, code will look like this:
> > * IgniteStatement stmt =
> > connection.createStatement().unwrap(IgniteStatement.class);*
> > * stmt.setDistributedJoins(true);*
> > *stmt.executeQuery("...");*
> >
> > What do you think?
> >
>



-- 
Andrey Gura
GridGain Systems, Inc.
www.gridgain.com

[jira] [Created] (IGNITE-3563) JDBC drivers should support distributed joins flag

[Andrey Gura (JIRA)]

Andrey Gura created IGNITE-3563:
---

 Summary: JDBC drivers should support distributed joins flag
 Key: IGNITE-3563
 URL: https://issues.apache.org/jira/browse/IGNITE-3563
 Project: Ignite
  Issue Type: Improvement
Reporter: Andrey Gura
Assignee: Andrey Gura
 Fix For: 1.7


{{SqlFieldsQuery}} now supports distributed joins that should be switched onn 
using {{setDistributedJoins(true)}} method. JDBC drivers shoould also provide 
support this future using connection string parameters.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3562) Dependency to outdate Lucene 3.5.0

[Alexander Veit (JIRA)]

Alexander Veit created IGNITE-3562:
--

 Summary: Dependency to outdate Lucene 3.5.0
 Key: IGNITE-3562
 URL: https://issues.apache.org/jira/browse/IGNITE-3562
 Project: Ignite
  Issue Type: Bug
  Components: general
Affects Versions: 1.6
Reporter: Alexander Veit


Ignite 1.6.0 comes with Lucene 3.5.0 core as dependency, which dates back to 
the year 2011.

This makes it difficult to integrate with newer software.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Distributed joins for JDBC

[Pavel Tupitsyn]

Hi,

I've added IGNITE-3561 for .NET.

However, there is not enough documentation for the new property:

* What are the implications of enabling distributed joins?
* Does it affect performance of non-join queries?
* Does it affect performance of colocated joins?
* Why is it disabled by default? Can we just enable it always and remove
the property?

Pavel.


On Mon, Jul 25, 2016 at 10:07 AM, Sergi Vladykin 
wrote:

> BTW this approach will be applicable to ODBC as well as far as I
> understand, so it will be consistent.
>
> Sergi
>
> 2016-07-25 10:04 GMT+03:00 Sergi Vladykin :
>
> > I don't think it makes sense to extend JDBC this way because usually if
> > one have access to Java API he most probably will use Ignite API. If for
> > some reason they use JDBC it means that it is an application which was
> > aimed to work with any RDBMS and should not know about quirks of some
> > particular driver. Take any JDBC based SQL console for example, we have
> to
> > support them out of the box.
> >
> > I think we should have a connection options which we can append to JDBC
> > URL like it is done in H2:
> >
> > jdbc:h2:my_database;OPTION1=bla;OPTION2=blabla
> >
> > In our case it must be something like DISTRIBUTED_JOINS=true and it will
> > affect the whole connection.
> >
> > Of course we have to support simultaneous connections to the same DB with
> > different options.
> >
> > Sergi
> >
> >
> > 2016-07-25 9:19 GMT+03:00 Semyon Boikov :
> >
> >> Hi,
> >>
> >> Last week distributed joins functionality was merged, but one thing was
> >> overlooked. Distributed joins should be explicitly enabled using using
> >> method 'setDistributedJoins' available in java API
> >> (SqlQuery/SqlFieldsQuery). First, this parameter should be also added in
> >> .Net/C++/REST API, this is straightforward. Also there should be
> >> possibility to enable distributed joins for JDBC API. Does it make sense
> >> to
> >> add Ignite-specific interface extending standard java.sql.Statement, so
> >> 'setDistributedJoins' method can be added there.
> >> JDBC API already have 'unwrap' method to deal with vendor-specific
> >> interfaces, code will look like this:
> >> * IgniteStatement stmt =
> >> connection.createStatement().unwrap(IgniteStatement.class);*
> >> * stmt.setDistributedJoins(true);*
> >> *stmt.executeQuery("...");*
> >>
> >> What do you think?
> >>
> >
> >
>

[jira] [Created] (IGNITE-3561) .NET: DistributedJoins property in SqlQuery & SqlFieldsQuery

[Pavel Tupitsyn (JIRA)]

Pavel Tupitsyn created IGNITE-3561:
--

 Summary: .NET: DistributedJoins property in SqlQuery & 
SqlFieldsQuery
 Key: IGNITE-3561
 URL: https://issues.apache.org/jira/browse/IGNITE-3561
 Project: Ignite
  Issue Type: Improvement
  Components: platforms
Affects Versions: 1.7
Reporter: Pavel Tupitsyn
Assignee: Pavel Tupitsyn
 Fix For: 1.7


IGNITE-1232 added new properties to SqlQuery and SqlFieldsQuery classes. 
Propagate them to .NET.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Data compression in Ignite 2.0

[Sergi Vladykin]

This will make sense only for rare cases when you have very large objects
stored, which can be effectively compressed. And even then it will
introduce slowdown on all the operations, which often will not be
acceptable. I guess only few users will find this feature useful, thus I
think it does not worth the effort.

Sergi



2016-07-25 9:28 GMT+03:00 Alexey Kuznetsov :

> Hi, All!
>
> I would like to propose one more feature for Ignite 2.0.
>
> Data compression for data in binary format.
>
> Binary format is stored as field name + field data.
> So we have a description.
> How about to add one more byte to binary data descriptor:
>
> *Compressed*:
>  0 - Data stored as is (no compression).
>  1 - Data compressed by dictionary (something like DB2 row compression [1],
>  but for all binary types). We could have system or user defined replicated
> cache for such dictionary and *cache.compact()* method that will scan
> cache, build dictionary and compact data.
>  2 - Data compressed by Java built in ZIP.
>  3 - Data compressed by some user custom algorithm.
>
> Of course it is possible to compress data in current Ignite 1.x but in this
> case compressed data cannot be accessed from SQL engine, if we implement
> support for compression on Ignite core level SQL engine will be able to
> detect that data is compressed and properly handle such data.
>
> What do you think?
> If community consider this feature useful I will create issue in JIRA.
>
> [1]
>
> http://www.ibm.com/developerworks/data/library/techarticle/dm-1205db210compression/
>
> --
> Alexey Kuznetsov
>

Re: Distributed joins for JDBC

[Sergi Vladykin]

BTW this approach will be applicable to ODBC as well as far as I
understand, so it will be consistent.

Sergi

2016-07-25 10:04 GMT+03:00 Sergi Vladykin :

> I don't think it makes sense to extend JDBC this way because usually if
> one have access to Java API he most probably will use Ignite API. If for
> some reason they use JDBC it means that it is an application which was
> aimed to work with any RDBMS and should not know about quirks of some
> particular driver. Take any JDBC based SQL console for example, we have to
> support them out of the box.
>
> I think we should have a connection options which we can append to JDBC
> URL like it is done in H2:
>
> jdbc:h2:my_database;OPTION1=bla;OPTION2=blabla
>
> In our case it must be something like DISTRIBUTED_JOINS=true and it will
> affect the whole connection.
>
> Of course we have to support simultaneous connections to the same DB with
> different options.
>
> Sergi
>
>
> 2016-07-25 9:19 GMT+03:00 Semyon Boikov :
>
>> Hi,
>>
>> Last week distributed joins functionality was merged, but one thing was
>> overlooked. Distributed joins should be explicitly enabled using using
>> method 'setDistributedJoins' available in java API
>> (SqlQuery/SqlFieldsQuery). First, this parameter should be also added in
>> .Net/C++/REST API, this is straightforward. Also there should be
>> possibility to enable distributed joins for JDBC API. Does it make sense
>> to
>> add Ignite-specific interface extending standard java.sql.Statement, so
>> 'setDistributedJoins' method can be added there.
>> JDBC API already have 'unwrap' method to deal with vendor-specific
>> interfaces, code will look like this:
>> * IgniteStatement stmt =
>> connection.createStatement().unwrap(IgniteStatement.class);*
>> * stmt.setDistributedJoins(true);*
>> *stmt.executeQuery("...");*
>>
>> What do you think?
>>
>
>

Re: Distributed joins for JDBC

[Sergi Vladykin]

I don't think it makes sense to extend JDBC this way because usually if one
have access to Java API he most probably will use Ignite API. If for some
reason they use JDBC it means that it is an application which was aimed to
work with any RDBMS and should not know about quirks of some particular
driver. Take any JDBC based SQL console for example, we have to support
them out of the box.

I think we should have a connection options which we can append to JDBC URL
like it is done in H2:

jdbc:h2:my_database;OPTION1=bla;OPTION2=blabla

In our case it must be something like DISTRIBUTED_JOINS=true and it will
affect the whole connection.

Of course we have to support simultaneous connections to the same DB with
different options.

Sergi


2016-07-25 9:19 GMT+03:00 Semyon Boikov :

> Hi,
>
> Last week distributed joins functionality was merged, but one thing was
> overlooked. Distributed joins should be explicitly enabled using using
> method 'setDistributedJoins' available in java API
> (SqlQuery/SqlFieldsQuery). First, this parameter should be also added in
> .Net/C++/REST API, this is straightforward. Also there should be
> possibility to enable distributed joins for JDBC API. Does it make sense to
> add Ignite-specific interface extending standard java.sql.Statement, so
> 'setDistributedJoins' method can be added there.
> JDBC API already have 'unwrap' method to deal with vendor-specific
> interfaces, code will look like this:
> * IgniteStatement stmt =
> connection.createStatement().unwrap(IgniteStatement.class);*
> * stmt.setDistributedJoins(true);*
> *stmt.executeQuery("...");*
>
> What do you think?
>