[GitHub] ignite pull request: ignite-3073

[sboikov]

Github user sboikov closed the pull request at:

https://github.com/apache/ignite/pull/687


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-3086) Cache store validation should be done only when write entry is enlisted

[Alexey Goncharuk (JIRA)]

Alexey Goncharuk created IGNITE-3086:


 Summary: Cache store validation should be done only when write 
entry is enlisted
 Key: IGNITE-3086
 URL: https://issues.apache.org/jira/browse/IGNITE-3086
 Project: Ignite
  Issue Type: Bug
  Components: cache
Affects Versions: 1.5.0.final
Reporter: Alexey Goncharuk






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] ignite pull request: Correct cluster misses aggregation

[krnlMustard]

Github user krnlMustard closed the pull request at:

https://github.com/apache/ignite/pull/686


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: POJO store usability issues

[Dmitriy Setrakyan]

On Wed, May 4, 2016 at 9:12 AM, Alexey Kuznetsov 
wrote:

>
> May be, but sending closures is seems as workaround for me.
> This will require from user to write quite a lot of code.
> Just call of loadCache(...some descriptors...) will be much more user
> friendly.
>

I think documenting this properly will take us a long way and will help our
users. After this is documented, we can then come up with a better design
and include it into one of future releases.


> Also I'm not sure what will happen with key loaded from DB if it will be
> not affinity key for node where  localLoadCache will be executed.
> Will it be ignored or put into cache?
>

They will be ignored.


>
> Semen, Alexey Goncharuk - could you tell what will happen with such keys
> inside of localLoadCache ?
>
> On Tue, May 3, 2016 at 11:10 PM, Dmitriy Setrakyan 
> wrote:
>
> > Alexey,
> >
> > What you are saying should already be possible by sending compute
> closures
> > to nodes and calling localLoadCache(“myCustomSqlStatement”).
> >
> > Isn’t it just a case of providing proper example and documentation?
> >
> > D.
> >
> > On Tue, May 3, 2016 at 7:22 AM, Alexey Kuznetsov <
> akuznet...@gridgain.com>
> > wrote:
> >
> > > I totally agree with Vladimir.
> > >
> > > From JdbcPojo store side we could introduce support of some kind load
> > > descriptor
> > >  that will contains SQL to execute and node filter.
> > >
> > > On each node store will check node filter and execute SQL if node match
> > the
> > > filter.
> > >
> > > This will solve first problem - "do not load full database on each
> node"
> > .
> > >
> > > As for second problem - "not ignore non-primary non-backup entries" - I
> > > think this should be solved on a cache level,
> > >  because store does not know anything about primary / backup.
> > >
> > > Thoughts?
> > >
> > >
> > > On Wed, Apr 27, 2016 at 9:27 PM, Vladimir Ozerov  >
> > > wrote:
> > >
> > > > Igniters,
> > > >
> > > > We receive more and more questions about the same problem: "I have
> big
> > a
> > > > database. How should I load it to Ignite?"
> > > >
> > > > Obviously, users try to use POJO store as a most convenient approach,
> > but
> > > > it cannot handle this case properly.
> > > > 1) If user invoke *IgniteCache.loadCache()*, then the same request -
> > and
> > > > usually this is full table scan - will be invoked on every node
> leading
> > > to
> > > > very poor performance. For instance, we have a report of a load of
> 47M
> > > > entries to cache on 16 nodes which took ... 8 hours!!!
> > > > 2) If user invoke IgniteCache.localLoadCache(), then our internal
> cache
> > > > logic will filter out non-primary and non-backup entries. So this
> > > approach
> > > > doesn't work either.
> > > > 3) User could try using *IgniteDataStreamer*, but in this case he had
> > to
> > > > deal with all JDBC-related stuff on his own - not convenient.
> > > > 4) Another approach I heard several times - "user should have an
> > > attribute
> > > > for affinity in the table ...". And the idea that this way user will
> be
> > > > able to divide the whole data set into several disjoint sets with
> > > specific
> > > > affinity. Doesn't work. Consider the user with some legacy database -
> > the
> > > > most common use case. How is he going to work with affinity?
> > > >
> > > > Bottom line: Ignite has *no convenient way *to load millions of
> entries
> > > > from a database.
> > > >
> > > > We need to start thinking of possible solutions. Several ideas from
> my
> > > > side:
> > > >
> > > > 1) POJO store must be much more flexible. We should be able to pass
> > > > different queries to different nodes when calling "loadCache".
> > > >
> > > > 2) Cache store could have additional mode when it will not ignore
> > > > non-primary non-backup entries, but rather *distribute *it to other
> > > nodes.
> > > > E.g. with help of data streamer.
> > > >
> > > > Thoughts?
> > > >
> > > > Vladimir.
> > > >
> > >
> > >
> > >
> > > --
> > > Alexey Kuznetsov
> > > GridGain Systems
> > > www.gridgain.com
> > >
> >
>
>
>
> --
> Alexey Kuznetsov
> GridGain Systems
> www.gridgain.com
>

[GitHub] ignite pull request: Cluster metrics should merge remote size and ...

[krnlMustard]

GitHub user krnlMustard opened a pull request:

https://github.com/apache/ignite/pull/688

Cluster metrics should merge remote size and key size

CacheMetric for a cluster was not aggregating remote getSize() and 
getKeySize() values.  Thus the aggregate CacheMetric would report only the 
local node's values.  When local node is in client mode, this would always be 
zero.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/krnlMustard/ignite patch-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/688.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #688






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request: ignite-3073

[sboikov]

GitHub user sboikov opened a pull request:

https://github.com/apache/ignite/pull/687

ignite-3073



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-3073

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/687.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #687


commit 2e64d0d7cc51552fffc231cbc850cd615076fb85
Author: vozerov-gridgain 
Date:   2015-12-29T06:31:58Z

IGNITE-2258: IGFS: now default path modes could be optionally disabled 
using FileSystemConfiguration.isInitializeDefaultPathModes() property.

commit 4cd3b3dc2f1fa0f1a9cceb6bf544dd8fb505d7f5
Author: vozerov-gridgain 
Date:   2015-12-29T09:52:00Z

IGNITE-2258: Fixed type on getter/setter.

commit 5d58fcbf40fdb9114e4cbb32b72dd9bce7fa38ca
Author: iveselovskiy 
Date:   2016-01-04T06:47:28Z

IGNITE-2308: Fixed HadoopClassLoader dependency resolution. This closes 
#391.

commit 83a19179cee2bb15adc36c2265dd0a3c794b60bb
Author: vozerov-gridgain 
Date:   2016-01-04T08:14:58Z

IGNITE-2218: Fixed a problem with native Hadoop libraries load. This closes 
#378.

commit 1d7fb5702fa33cf395e797161f3a86a9600921a7
Author: vozerov-gridgain 
Date:   2016-01-05T06:59:31Z

IGNITE-2206: Hadoop file system creation is now abstracted out using 
factory interface.

commit a12ec7d08573d5396654a5ba05bb7d873e4c2677
Author: Ignite Teamcity 
Date:   2016-01-06T10:50:48Z

1.5.2

commit 090a5de6a930c10a3a57a6e28c486fe5c87e028d
Author: vozerov-gridgain 
Date:   2015-12-29T12:50:39Z

Minor fix.

commit c786820dda7f7cd1849c5593ac24ca9072945887
Author: vozerov-gridgain 
Date:   2016-01-07T13:48:14Z

IgniteHadoopIgfsSecondaryFileSystem.usrName field is renamed to "userName" 
to preserve backward compatibility.

commit 6ab4ce246316442fa4295f9941c372fea70c24ef
Author: vozerov-gridgain 
Date:   2016-01-08T06:23:55Z

IGNITE-2342: Set correct classlader before calling FileSystem.get().

commit 077ab1b3a77fdb1c2c2fd6360fc5b60fda6c50c3
Author: vozerov-gridgain 
Date:   2016-01-08T07:17:45Z

IGNITE-2341: Improved warning message when BinaryMarshaller cannot be used. 
Also it is not shown when "org.apache.ignite" classes is in described situation.

commit 86c4816edfd0e983014f136ffc92cde28ec56cca
Author: vozerov-gridgain 
Date:   2016-01-08T07:26:03Z

IGNITE-2340: Improved error thrown when PROXY mode exists, but secondary 
file system is not IgniteHadoopIgfsSecondaryFileSystem.

commit fc48a8429a84ef6c87ed3225a218d7d3ae617e14
Author: vozerov-gridgain 
Date:   2016-01-08T07:48:42Z

Merge branch 'ignite-1.5.2' into ignite-1.5.3

commit 86740cefe212ed0f506d81056dd8e76de9a31e4f
Author: Ignite Teamcity 
Date:   2016-01-08T09:32:11Z

1.5.3-SNAPSHOT

commit 92229d2a6c6ef86772a62cb52b3aa07a55c99d89
Author: sboikov 
Date:   2016-01-13T05:56:34Z

ignite-2359 Added locking for files used by MarshallerContextImpl. (cherry 
picked from commit 1d8c4e2)

commit 2e4ce585d5f54502c6511d3425b1cd5fbf0a7f4f
Author: Ignite Teamcity 
Date:   2016-01-13T10:37:33Z

1.5.4-SNAPSHOT

commit 6e5f9f0c7d4c86773b1f0cd5c5a673acb58c86c2
Author: Denis Magda 
Date:   2016-01-13T11:42:27Z

Changed year to 2016 in Copyrights

commit 02dbcfd8ed2701a4f415c8871d0b8fd08bfa0583
Author: Alexey Goncharuk 
Date:   2016-01-13T13:47:32Z

IGNITE-2365 - Notify policy if swap or offheap is enabled and rebalanced 
entry was not preloaded.
IGNITE-2099 - Fixing custom collections.
This closes #396

commit 86c2ba2a601e82b824cf17422683e5398a4d8c7d
Author: sboikov 
Date:   2016-01-13T15:40:08Z

ignite-2350 Pass update notifier flag in discovery data (all cluster nodes 
will have the same notifier status as first cluster node)
(cherry picked from commit 7175a42)

commit e1a494df400fc37ca04e8d88d1cf20bca02607b4
Author: sboikov 
Date:   2016-01-14T11:16:33Z

Renamed fields to change fields write order (to preserve backward 
compatibility).
(cherry picked from commit 2a4adf5)

commit 09f978234b6062afa1e1658d5a6439365a856aca
Author: sboikov 
Date:   2016-01-14T11:42:44Z

Merge remote-tracking branch 'origin/ignite-1.5.4' into ignite-1.5.4

commit 30043e7892d0b52dc851ce5ec79c7eb3b7cc44fb
Author: Denis Magda 
Date:   2016-01-14T13:02:50Z

Added release notes

commit cc3db35925698f1670a8bf1c6a1830c0c9b51290
Author: vershov 

[GitHub] ignite pull request: Correct cluster misses aggregation

[krnlMustard]

GitHub user krnlMustard opened a pull request:

https://github.com/apache/ignite/pull/686

Correct cluster misses aggregation

When merging metrics from a cluster, node hits instead of node misses were 
being aggregated into merged misses.  Thus for clusters the CacheMetrics were 
reporting the same value for both hits and misses.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/krnlMustard/ignite patch-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/686.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #686


commit ac8ae6ff0d05b99b9e352f56aac29b33af6e5c47
Author: krnlMustard 
Date:   2016-05-04T16:52:47Z

Correct cluster misses aggregation

When merging metrics from a cluster, node hits instead of node misses were 
being aggregated into merged misses.  Thus for clusters the CacheMetrics were 
reporting the same value for both hits and misses.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Spark/YARN

[vijayendra bhati]

Hi Naden,I am just wondering why you want to start ignite node in Yarn.You can 
stary the independent ignite node.What benefit you are getting by making part 
of yarn cluster.I am asking this because i too had the same queries but latet 
on i switched to starting independent ignite node on the same nodes where my 
node managers are running.I guess this too will give data locality etc.Also you 
can create RDD and save data directly to cache using RDD,you dont need to 
explicitly invoke ignite() or start().I can share the code if you want.
Regards,Vij

Sent from Yahoo Mail on Android 
 
  On Wed, 4 May, 2016 at 5:19 am, Franciscus, 
Naden wrote:   
Hey Alexey,

Sorry I meant HDFS :)

It would be good for IgniteContext to have the option to read
configuration from HDFS.
Again many users of Hadoop aren't going to have the ability to modify data
nodes to install
Ignite. But I will raise that as a separate issue.

The big issue is that there is a lot of code like this in documentation
and elsewhere:
ignite.cache("myCache").put(personId, person);

Which we can't use in any RDD/DF map, filter etc because neither
Ignite/IgniteCache are serializable.


Cheers,
Naden





On 4/05/2016, 5:42 AM, "Alexey Goncharuk" 
wrote:

>Hi Naden,
>
> But the IgniteContext within Spark doesn't allow you to read
>configuration
>> files from YARN.
>>
>
>I am a little bit confused. Ignite can be configured via basic Spring XML,
>and you can definitely read those XML files off HDFS or any other source.
>Is there any reason why XML does not work for you?
>
>Now it is impossible to make IgniteConfiguration serializable because it
>contains such components as SPIs, which are non-serializable by it's
>nature.
  

Re: Ignite 1.6 release timelines

[Dmitriy Setrakyan]

Resending (fixed links)

On Wed, May 4, 2016 at 8:46 AM, Dmitriy Setrakyan 
wrote:

> Igniters,
>
> Looks like we are almost done with the outstanding issues, mainly ODBC
> support and cache concurrent map changes. As far as the concurrent map, we
> had a performance drop, and I think it is pretty important that we address
> it prior to the release.
>
> Can we please get an update for these tickets?
>
> IGNITE-2948  -
> Optimize usage of GridCacheConcurrentMap
> IGNITE-1786  - Need to
> implement ODBC driver for Ignite
>
> Once these issues are resolved, we can move to the testing phase (probably
> a week), and then send it for the vote.
>
> D.
>
> On Fri, Apr 29, 2016 at 3:12 AM, Yakov Zhdanov 
> wrote:
>
>> I think we are almost done with odbc support and I hope that we can
>> release
>> about mid or second half of May.
>>
>> --Yakov
>>
>> 2016-04-27 11:13 GMT+03:00 sssow :
>>
>> > Hi,
>> > When will you release 1.6?
>> > Thank you,
>> >
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> >
>> http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-1-6-release-timelines-tp7388p8618.html
>> > Sent from the Apache Ignite Developers mailing list archive at
>> Nabble.com.
>> >
>>
>
>

[GitHub] ignite pull request: Ignite gg 11121

[dkarachentsev]

GitHub user dkarachentsev opened a pull request:

https://github.com/apache/ignite/pull/685

Ignite gg 11121



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-gg-11121

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/685.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #685


commit 2e64d0d7cc51552fffc231cbc850cd615076fb85
Author: vozerov-gridgain 
Date:   2015-12-29T06:31:58Z

IGNITE-2258: IGFS: now default path modes could be optionally disabled 
using FileSystemConfiguration.isInitializeDefaultPathModes() property.

commit 4cd3b3dc2f1fa0f1a9cceb6bf544dd8fb505d7f5
Author: vozerov-gridgain 
Date:   2015-12-29T09:52:00Z

IGNITE-2258: Fixed type on getter/setter.

commit 5d58fcbf40fdb9114e4cbb32b72dd9bce7fa38ca
Author: iveselovskiy 
Date:   2016-01-04T06:47:28Z

IGNITE-2308: Fixed HadoopClassLoader dependency resolution. This closes 
#391.

commit 83a19179cee2bb15adc36c2265dd0a3c794b60bb
Author: vozerov-gridgain 
Date:   2016-01-04T08:14:58Z

IGNITE-2218: Fixed a problem with native Hadoop libraries load. This closes 
#378.

commit 1d7fb5702fa33cf395e797161f3a86a9600921a7
Author: vozerov-gridgain 
Date:   2016-01-05T06:59:31Z

IGNITE-2206: Hadoop file system creation is now abstracted out using 
factory interface.

commit a12ec7d08573d5396654a5ba05bb7d873e4c2677
Author: Ignite Teamcity 
Date:   2016-01-06T10:50:48Z

1.5.2

commit 090a5de6a930c10a3a57a6e28c486fe5c87e028d
Author: vozerov-gridgain 
Date:   2015-12-29T12:50:39Z

Minor fix.

commit c786820dda7f7cd1849c5593ac24ca9072945887
Author: vozerov-gridgain 
Date:   2016-01-07T13:48:14Z

IgniteHadoopIgfsSecondaryFileSystem.usrName field is renamed to "userName" 
to preserve backward compatibility.

commit 6ab4ce246316442fa4295f9941c372fea70c24ef
Author: vozerov-gridgain 
Date:   2016-01-08T06:23:55Z

IGNITE-2342: Set correct classlader before calling FileSystem.get().

commit 077ab1b3a77fdb1c2c2fd6360fc5b60fda6c50c3
Author: vozerov-gridgain 
Date:   2016-01-08T07:17:45Z

IGNITE-2341: Improved warning message when BinaryMarshaller cannot be used. 
Also it is not shown when "org.apache.ignite" classes is in described situation.

commit 86c4816edfd0e983014f136ffc92cde28ec56cca
Author: vozerov-gridgain 
Date:   2016-01-08T07:26:03Z

IGNITE-2340: Improved error thrown when PROXY mode exists, but secondary 
file system is not IgniteHadoopIgfsSecondaryFileSystem.

commit fc48a8429a84ef6c87ed3225a218d7d3ae617e14
Author: vozerov-gridgain 
Date:   2016-01-08T07:48:42Z

Merge branch 'ignite-1.5.2' into ignite-1.5.3

commit 86740cefe212ed0f506d81056dd8e76de9a31e4f
Author: Ignite Teamcity 
Date:   2016-01-08T09:32:11Z

1.5.3-SNAPSHOT

commit 92229d2a6c6ef86772a62cb52b3aa07a55c99d89
Author: sboikov 
Date:   2016-01-13T05:56:34Z

ignite-2359 Added locking for files used by MarshallerContextImpl. (cherry 
picked from commit 1d8c4e2)

commit 2e4ce585d5f54502c6511d3425b1cd5fbf0a7f4f
Author: Ignite Teamcity 
Date:   2016-01-13T10:37:33Z

1.5.4-SNAPSHOT

commit 6e5f9f0c7d4c86773b1f0cd5c5a673acb58c86c2
Author: Denis Magda 
Date:   2016-01-13T11:42:27Z

Changed year to 2016 in Copyrights

commit 02dbcfd8ed2701a4f415c8871d0b8fd08bfa0583
Author: Alexey Goncharuk 
Date:   2016-01-13T13:47:32Z

IGNITE-2365 - Notify policy if swap or offheap is enabled and rebalanced 
entry was not preloaded.
IGNITE-2099 - Fixing custom collections.
This closes #396

commit 86c2ba2a601e82b824cf17422683e5398a4d8c7d
Author: sboikov 
Date:   2016-01-13T15:40:08Z

ignite-2350 Pass update notifier flag in discovery data (all cluster nodes 
will have the same notifier status as first cluster node)
(cherry picked from commit 7175a42)

commit e1a494df400fc37ca04e8d88d1cf20bca02607b4
Author: sboikov 
Date:   2016-01-14T11:16:33Z

Renamed fields to change fields write order (to preserve backward 
compatibility).
(cherry picked from commit 2a4adf5)

commit 09f978234b6062afa1e1658d5a6439365a856aca
Author: sboikov 
Date:   2016-01-14T11:42:44Z

Merge remote-tracking branch 'origin/ignite-1.5.4' into ignite-1.5.4

commit 30043e7892d0b52dc851ce5ec79c7eb3b7cc44fb
Author: Denis Magda 
Date:   2016-01-14T13:02:50Z

Added release notes

commit cc3db35925698f1670a8bf1c6a1830c0c9b51290
Author: vershov 

Re: Ignite 1.6 release timelines

[Dmitriy Setrakyan]

Igniters,

Looks like we are almost done with the outstanding issues, mainly ODBC
support and cache concurrent map changes. As far as the concurrent map, we
had a performance drop, and I think it is pretty important that we address
it prior to the release.

Can we please get an update for these tickets?

IGNITE-2948  - Optimize usage of
GridCacheConcurrentMap
IGNITE-1786  - Need to
implement ODBC driver for Ignite

Once these issues are resolved, we can move to the testing phase (probably
a week), and then send it for the vote.

D.

On Fri, Apr 29, 2016 at 3:12 AM, Yakov Zhdanov  wrote:

> I think we are almost done with odbc support and I hope that we can release
> about mid or second half of May.
>
> --Yakov
>
> 2016-04-27 11:13 GMT+03:00 sssow :
>
> > Hi,
> > When will you release 1.6?
> > Thank you,
> >
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-1-6-release-timelines-tp7388p8618.html
> > Sent from the Apache Ignite Developers mailing list archive at
> Nabble.com.
> >
>

[GitHub] ignite pull request: Ignite gg 11121

[dkarachentsev]

Github user dkarachentsev closed the pull request at:

https://github.com/apache/ignite/pull/682


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request: Ignite 3018

[tledkov-gridgain]

GitHub user tledkov-gridgain opened a pull request:

https://github.com/apache/ignite/pull/684

Ignite 3018



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-3018

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/684.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #684


commit 04c1c159293972af1197c86d69e662ed19933fcb
Author: tledkov-gridgain 
Date:   2016-04-28T07:29:40Z

IGNITE-3018 raw sources

commit b772b57304267ead713df0aaf9bfd8f3ff16df4f
Author: tledkov-gridgain 
Date:   2016-04-29T09:44:07Z

IGNITE-3018 implement  bucket-based hashing

commit e63a69764beac879f3c848607df1c321f7cfb29d
Author: tledkov-gridgain 
Date:   2016-05-04T14:42:32Z

IGNITE-3018 implement  bucket based algorithm

commit ed958091a0d8b0a97ab59986943be69906c58bcd
Author: tledkov-gridgain 
Date:   2016-05-04T14:44:48Z

Merge remote-tracking branch 'remotes/origin/master' into ignite-3018




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request: ignite-3041

[ascherbakoff]

Github user ascherbakoff closed the pull request at:

https://github.com/apache/ignite/pull/663


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request: Ignite-2854 Deadlock detection for pessimisti...

[agura]

Github user agura closed the pull request at:

https://github.com/apache/ignite/pull/656


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request: ignite-2982

[ascherbakoff]

Github user ascherbakoff closed the pull request at:

https://github.com/apache/ignite/pull/676


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request: Ignite gg 11121

[dkarachentsev]

GitHub user dkarachentsev opened a pull request:

https://github.com/apache/ignite/pull/682

Ignite gg 11121



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-gg-11121

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/682.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #682


commit 22d5d4d53a4355f11572baad591ae548551d708b
Author: dkarachentsev 
Date:   2016-05-04T10:13:38Z

GG-11121 - Do not deserialize binary objects if set keepBinary flag on 
loadCache() operation. Do not deserialize binary objects in DR receive 
operation.

commit 9d7cb654693c36c1e8073608d4e774d9f7663243
Author: dkarachentsev 
Date:   2016-05-04T11:51:19Z

GG-11121 - Remove DR keep binary in DrReceiver.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Cache errors with enabled indexing

[Alexei Scherbakov]

Guys,

I've implemented CacheOffheapBatchIndexingTest

containing various tests for batch cache operations
with enabled indexing.

*testBatchRemovetest *fails when H2 row value is the
instance of BinaryObjectOffheapImpl.
I was able to fix it by reading value contents into heap
and need someone to review the fix.

Other tests contain multiple value objects
in cache and fail with various indexing errors.
Particularly *testPutAllMultupleEntitiesAndStreamerDfltOffHeapRowCacheSize
*sometimes
hangs in timed wait.
Should we do anything about it or stick with rule *one value type per cache*
? I was not able to reproduce problems in case the cache contains only one
value type.

Alex G, Serg V, Sam B please take a look.


-- 

Best regards,
Alexei Scherbakov

[GitHub] ignite pull request: ignite-3073

[sboikov]

Github user sboikov closed the pull request at:

https://github.com/apache/ignite/pull/680


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-3084) Investigate how Ignite can support Spark DataFrame

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-3084:
---

 Summary: Investigate how Ignite can support Spark DataFrame
 Key: IGNITE-3084
 URL: https://issues.apache.org/jira/browse/IGNITE-3084
 Project: Ignite
  Issue Type: Task
  Components: Ignite RDD
Affects Versions: 1.5.0.final
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
Priority: Critical
 Fix For: 1.7


We see increasing demand on nice DataFrame support for our Spark integration. 
Need to investigate how could we do that.

Looks like we can investigate how MemSQL do that and take it as a starting 
point.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3083) Shmem loader failed with OverlappingFileLockException

[Nikolay Tikhonov (JIRA)]

Nikolay Tikhonov created IGNITE-3083:


 Summary: Shmem loader failed with OverlappingFileLockException
 Key: IGNITE-3083
 URL: https://issues.apache.org/jira/browse/IGNITE-3083
 Project: Ignite
  Issue Type: Bug
Affects Versions: 1.5.0.final
Reporter: Nikolay Tikhonov
Priority: Minor


IpcSharedMemoryNativeLoader can try to get lock on a file twice in the same JVM 
in case when nodes are starting in parallel in one JVM and they loaded by 
different classloaders. 
{{java.nio.channels.OverlappingFileLockException
at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1012)
at java.nio.channels.FileChannel.lock(FileChannel.java:1052)
at 
org.apache.ignite.internal.util.ipc.shmem.IpcSharedMemoryNativeLoader.doLoad(IpcSharedMemoryNativeLoader.java:141)
at 
org.apache.ignite.internal.util.ipc.shmem.IpcSharedMemoryNativeLoader.load(IpcSharedMemoryNativeLoader.java:111)
at 
org.apache.ignite.internal.util.ipc.shmem.IpcSharedMemoryServerEndpoint.start(IpcSharedMemoryServerEndpoint.java:167)
at 
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.resetShmemServer(TcpCommunicationSpi.java:1737)
at 
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.getNodeAttributes(TcpCommunicationSpi.java:1422)
at 
org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:219)
at 
org.apache.ignite.internal.managers.communication.GridIoManager.start(GridIoManager.java:236)
at 
org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1505)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:861)
at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1688)
at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1547)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1003)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:534)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:515)
at org.apache.ignite.Ignition.start(Ignition.java:322)}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-3081) NearCacheConfiguration incorrectly extends MutableConfiguration

[Pavel Tupitsyn (JIRA)]

Pavel Tupitsyn created IGNITE-3081:
--

 Summary: NearCacheConfiguration incorrectly extends 
MutableConfiguration
 Key: IGNITE-3081
 URL: https://issues.apache.org/jira/browse/IGNITE-3081
 Project: Ignite
  Issue Type: Bug
  Components: cache, community
Affects Versions: 1.1.4
Reporter: Pavel Tupitsyn
Priority: Critical
 Fix For: 1.6


NearCacheConfiguration extends MutableConfiguration, but does not use anything 
inherited.

This confuses both developers and users (there were questions in Gitter).

Two solutions:
* Remove this base class
* Or, if compatibility is needed, override all members and throw exceptions 
from them; update documentation to make this clear.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] ignite pull request: gridgain-7.5.18

[sboikov]

GitHub user sboikov opened a pull request:

https://github.com/apache/ignite/pull/681

gridgain-7.5.18



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite gridgain-7.5.18

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/681.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #681


commit 2e64d0d7cc51552fffc231cbc850cd615076fb85
Author: vozerov-gridgain 
Date:   2015-12-29T06:31:58Z

IGNITE-2258: IGFS: now default path modes could be optionally disabled 
using FileSystemConfiguration.isInitializeDefaultPathModes() property.

commit 4cd3b3dc2f1fa0f1a9cceb6bf544dd8fb505d7f5
Author: vozerov-gridgain 
Date:   2015-12-29T09:52:00Z

IGNITE-2258: Fixed type on getter/setter.

commit 5d58fcbf40fdb9114e4cbb32b72dd9bce7fa38ca
Author: iveselovskiy 
Date:   2016-01-04T06:47:28Z

IGNITE-2308: Fixed HadoopClassLoader dependency resolution. This closes 
#391.

commit 83a19179cee2bb15adc36c2265dd0a3c794b60bb
Author: vozerov-gridgain 
Date:   2016-01-04T08:14:58Z

IGNITE-2218: Fixed a problem with native Hadoop libraries load. This closes 
#378.

commit 1d7fb5702fa33cf395e797161f3a86a9600921a7
Author: vozerov-gridgain 
Date:   2016-01-05T06:59:31Z

IGNITE-2206: Hadoop file system creation is now abstracted out using 
factory interface.

commit a12ec7d08573d5396654a5ba05bb7d873e4c2677
Author: Ignite Teamcity 
Date:   2016-01-06T10:50:48Z

1.5.2

commit 090a5de6a930c10a3a57a6e28c486fe5c87e028d
Author: vozerov-gridgain 
Date:   2015-12-29T12:50:39Z

Minor fix.

commit c786820dda7f7cd1849c5593ac24ca9072945887
Author: vozerov-gridgain 
Date:   2016-01-07T13:48:14Z

IgniteHadoopIgfsSecondaryFileSystem.usrName field is renamed to "userName" 
to preserve backward compatibility.

commit 6ab4ce246316442fa4295f9941c372fea70c24ef
Author: vozerov-gridgain 
Date:   2016-01-08T06:23:55Z

IGNITE-2342: Set correct classlader before calling FileSystem.get().

commit 077ab1b3a77fdb1c2c2fd6360fc5b60fda6c50c3
Author: vozerov-gridgain 
Date:   2016-01-08T07:17:45Z

IGNITE-2341: Improved warning message when BinaryMarshaller cannot be used. 
Also it is not shown when "org.apache.ignite" classes is in described situation.

commit 86c4816edfd0e983014f136ffc92cde28ec56cca
Author: vozerov-gridgain 
Date:   2016-01-08T07:26:03Z

IGNITE-2340: Improved error thrown when PROXY mode exists, but secondary 
file system is not IgniteHadoopIgfsSecondaryFileSystem.

commit fc48a8429a84ef6c87ed3225a218d7d3ae617e14
Author: vozerov-gridgain 
Date:   2016-01-08T07:48:42Z

Merge branch 'ignite-1.5.2' into ignite-1.5.3

commit 86740cefe212ed0f506d81056dd8e76de9a31e4f
Author: Ignite Teamcity 
Date:   2016-01-08T09:32:11Z

1.5.3-SNAPSHOT

commit 92229d2a6c6ef86772a62cb52b3aa07a55c99d89
Author: sboikov 
Date:   2016-01-13T05:56:34Z

ignite-2359 Added locking for files used by MarshallerContextImpl. (cherry 
picked from commit 1d8c4e2)

commit 2e4ce585d5f54502c6511d3425b1cd5fbf0a7f4f
Author: Ignite Teamcity 
Date:   2016-01-13T10:37:33Z

1.5.4-SNAPSHOT

commit 6e5f9f0c7d4c86773b1f0cd5c5a673acb58c86c2
Author: Denis Magda 
Date:   2016-01-13T11:42:27Z

Changed year to 2016 in Copyrights

commit 02dbcfd8ed2701a4f415c8871d0b8fd08bfa0583
Author: Alexey Goncharuk 
Date:   2016-01-13T13:47:32Z

IGNITE-2365 - Notify policy if swap or offheap is enabled and rebalanced 
entry was not preloaded.
IGNITE-2099 - Fixing custom collections.
This closes #396

commit 86c2ba2a601e82b824cf17422683e5398a4d8c7d
Author: sboikov 
Date:   2016-01-13T15:40:08Z

ignite-2350 Pass update notifier flag in discovery data (all cluster nodes 
will have the same notifier status as first cluster node)
(cherry picked from commit 7175a42)

commit e1a494df400fc37ca04e8d88d1cf20bca02607b4
Author: sboikov 
Date:   2016-01-14T11:16:33Z

Renamed fields to change fields write order (to preserve backward 
compatibility).
(cherry picked from commit 2a4adf5)

commit 09f978234b6062afa1e1658d5a6439365a856aca
Author: sboikov 
Date:   2016-01-14T11:42:44Z

Merge remote-tracking branch 'origin/ignite-1.5.4' into ignite-1.5.4

commit 30043e7892d0b52dc851ce5ec79c7eb3b7cc44fb
Author: Denis Magda 
Date:   2016-01-14T13:02:50Z

Added release notes

commit cc3db35925698f1670a8bf1c6a1830c0c9b51290
Author: vershov 

[jira] [Created] (IGNITE-3080) Improve Spark documentation for CDH deployment

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-3080:
---

 Summary: Improve Spark documentation for CDH deployment
 Key: IGNITE-3080
 URL: https://issues.apache.org/jira/browse/IGNITE-3080
 Project: Ignite
  Issue Type: Improvement
  Components: Ignite RDD
Affects Versions: 1.5.0.final
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
 Fix For: 1.6


Users might experience classpath issues when running Spark on CDH with YARN. 

The solution is to add Ignite JARs to YARN classpath through CDH console. 

Originally explained by Michael Dolgonos here: 
http://apache-ignite-users.70518.x6.nabble.com/Ignite-Installation-with-Spark-under-CDH-tp4457p4624.html
 

{quote}
Cloudera Manager -> YARN (MR2 Included) -> Configuration -> Service Wide ->
Advanced -> Spark Client Advanced Configuration Snippet (Safety Valve) for
spark-conf/spark-defaults.conf
Added etc/ignite-fabric-1.5.0/libs/* to already present
spark.executor.extraClassPath=/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/jars/htrace-core-3.2.0-incubating.jar.
The combined line looks like this:
spark.executor.extraClassPath=/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/jars/htrace-core-3.2.0-incubating.jar:/etc/ignite-fabric-1.5.0/libs/*{quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: 1.5.0.final is breaking packaging: osgi dependency is non-existent

[Anton Vinogradov]

Roman, Raul,

As far as I remember you're familiar with OSGI. Could you please have a
look at this?

On Wed, May 4, 2016 at 6:09 AM, Konstantin Boudnik  wrote:

> Hey guys
>
> For your information: adding osgi into the product is breaking the
> dependency matrix of the bigtop packages. For more info see
> https://issues.apache.org/jira/browse/BIGTOP-2421
>
> --
> Take care,
> Cos
> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> Cos' pubkey: http://people.apache.org/~cos/cos.asc
>
>   Wisdom of the hour 
>
>

[GitHub] ignite pull request: ignite-3073

[sboikov]

GitHub user sboikov opened a pull request:

https://github.com/apache/ignite/pull/680

ignite-3073

ignite-3073

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-3073

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/680.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #680






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---