[GitHub] ignite pull request: IGNITE-1467: Standalone node application

[isapego]

GitHub user isapego opened a pull request:

https://github.com/apache/ignite/pull/142

IGNITE-1467: Standalone node application



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/isapego/ignite ignite-1467

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/142.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #142


commit 52728ecc9c39d32880ba4ae178e21eb60f0416cf
Author: isapego 
Date:   2015-10-07T14:02:08Z

IGNITE-1467: Application stub. Can only be compiled for Windows now.

commit 4aa76b8a2b0450c8dc67c7fc72bb37bb282f7fb5
Author: Igor Sapego 
Date:   2015-10-07T14:30:33Z

Support for the autotools build system added.

commit 073e17704d78ae736e081aebe7ad796c6d0d55d0
Author: isapego 
Date:   2015-10-08T08:06:06Z

Internal DestroyJvm method added to be used in stand alone application




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Name of .Net product

[Gianfranco Murador]

+1 Ignite.NET

2015-10-07 18:31 GMT+02:00 Konstantin Boudnik :

> Technically, "Apache Ignite" is the project trademark (which could be
> registered, if needed). Hence the longer name is probably more correct.
>
> Cos
>
> On Wed, Oct 07, 2015 at 07:31AM, Dmitriy Setrakyan wrote:
> > +1 for Ignite.NET or Apache Ignite.NET. I think both should be valid.
> >
> > On Wed, Oct 7, 2015 at 7:12 AM, Vladimir Ozerov 
> > wrote:
> >
> > > Raul, good point!
> > >
> > > I have no information about .Net trademark, but other products use
> ".NET"
> > > without a blank space.
> > >
> > > On Wed, Oct 7, 2015 at 4:37 PM, Raul Kripalani 
> wrote:
> > >
> > > > I like Ignite.NET.
> > > >
> > > > I wonder if the .NET portion may be trademarked by Microsoft. If
> that's
> > > the
> > > > case, I guess we cannot put a blank space between Ignite and .NET
> (Ignite
> > > > .NET), it'll all need to be a single word, like proposed.
> > > >
> > > > Regards,
> > > >
> > > > *Raúl Kripalani*
> > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big
> Data and
> > > > Messaging Engineer
> > > > http://about.me/raulkripalani |
> http://www.linkedin.com/in/raulkripalani
> > > > http://blog.raulkr.net | twitter: @raulvk
> > > >
> > > > On Wed, Oct 7, 2015 at 1:04 PM, Vladimir Ozerov <
> voze...@gridgain.com>
> > > > wrote:
> > > >
> > > > > Igniters,
> > > > >
> > > > > Let's agree on the final name of our .Net product. For now it is
> > > > referenced
> > > > > in various places as "Apache Ignite .Net", "Apache Ignite for
> .Net",
> > > > > "Apache Ignite .Net platform", etc..
> > > > >
> > > > > Ports of other Java projects to .Net are usually named as
> > > > "[Product].NET".
> > > > > E.g. Akka.NET, Spring.NET.
> > > > >
> > > > > I suggest use the same scheme and pick the name "Apache
> Ignite.NET".
> > > > >
> > > > > Any ideas or objections?
> > > > >
> > > > > Vladimir.
> > > > >
> > > >
> > >
>



-- 
Gianfranco Murador
Igniter and Software Engineer.

Re: IGFS concurrency issue

[Vladimir Ozerov]

Cos,

Initially IGFS was designed to support concurrent structural updates. E.g.,
creation of directories /A/C/D and /E/F/G can be performed concurrently.
Now we revealed that it might cause concurrency issues in case of
conflicting updates.

To fix this problem we now perform every update holding a kind of global
file system lock. This doesn't affect file read/write operations - they
still can be performed concurrently of course. The question was whether
this could cause severe performance degradation or not. If we assume that
in average Hadoop jobs file operations dominate over structural operations
on directories, then it should not cause any significant performance issues.

Vladimir.

On Wed, Oct 7, 2015 at 10:38 PM, Konstantin Boudnik  wrote:

> On Wed, Oct 07, 2015 at 09:11AM, Vladimir Ozerov wrote:
> > Cos,
> > Yes, no long-time locking is expected here.
>
> Sorry, I musta be still dense from the jet-lag: could you put in a bit more
> details? Thanks in advance!
>
> Cos
>
> > On Wed, Oct 7, 2015 at 6:57 AM, Konstantin Boudnik 
> wrote:
> >
> > > IIRC NN should be locking on these ops anyway, shouldn't it? The
> situation
> > > is
> > > no different if multiple clients are doing these operations
> > > near-simultaneously. Unless I missed something here...
> > >
> > > On Thu, Sep 24, 2015 at 11:28AM, Sergi Vladykin wrote:
> > > > May be just check that they are not parent-child within the tx?
> > > >
> > > > Sergi
> > > > Igniters,
> > > >
> > > > We revealed concurrency problem in IGFS and I would like to discuss
> > > > possible solutions to it.
> > > >
> > > > Consider the following file system structure:
> > > > root
> > > > |-- A
> > > > |   |-- B
> > > > |   |   |-- C
> > > > |   |-- D
> > > >
> > > > ... two concurrent operations in different threads:
> > > > T1: move(/A/B, /A/D);
> > > > T2: move(/A/D, /A/B/C);
> > > >
> > > > ... and how IGFS handles it now:
> > > > T1: verify that "/A/B" and "/A/D" exist, they are not child-parent to
> > > each
> > > > other, etc. -> OK.
> > > > T2: do the same for "A/D" and "A/B/C" -> OK.
> > > > T1: get IDs of "/A", "/A/B" and "/A/D" to lock them later inside tx.
> > > > T2: get IDs of "/A", "/A/D", "/A/B" and "/A/B/C" to lock them later
> > > inside
> > > > tx.
> > > >
> > > > T1: Start pessimistic tx, lock IDs of "/A", "/A/B", "/A/D", perform
> move
> > > ->
> > > > OK.
> > > > root
> > > > |-- A
> > > > |   |-- D
> > > > |   |   |-- B
> > > > |   |   |   |-- C
> > > >
> > > > T2: Start pessimistic tx, lock IDs of "/A", "/A/D", "/A/B" and
> > > > "/A/B/C" (*directory
> > > > structure already changed at this time!*), perform move -> OK.
> > > > root
> > > > |-- A
> > > > B
> > > > |-- D
> > > > |   |-- C
> > > > |   |   |-- B (loop!)
> > > >
> > > > File system is corrupted. Folders B, C and D are not reacheable from
> > > root.
> > > >
> > > > To fix this now we additionaly check if directory structure is still
> > > > valid *inside
> > > > transaction*. It works, no more corruptions. But it requres taking
> locks
> > > on
> > > > the whole paths *including root*. So move, delete and mkdirs
> opeartions
> > > *can
> > > > no longer be concurrent*.
> > > >
> > > > Probably there is a way to relax this while still ensuring
> consistency,
> > > but
> > > > I do not see how. One idea is to store real path inside each entry.
> This
> > > > way we will be able to ensure that it is still at a valid location
> > > without
> > > > blocking parents, so concurrnecy will be restored. But we will have
> to
> > > > propagate strucutral changes to children. E.g. move of a folder with
> 100
> > > > items will lead to update of >100 cache entries. Not so good.
> > > >
> > > > Any other ideas?
> > > >
> > > > Vladimir.
> > >
>

[jira] [Created] (IGNITE-1634) We should generate POJO classes in Client configuration too

[Pavel Konstantinov (JIRA)]

Pavel Konstantinov created IGNITE-1634:
--

 Summary: We should generate POJO classes in Client configuration 
too
 Key: IGNITE-1634
 URL: https://issues.apache.org/jira/browse/IGNITE-1634
 Project: Ignite
  Issue Type: Sub-task
Reporter: Pavel Konstantinov
Assignee: Vasiliy Sisko


Currently we do not generate POJO for Client.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] ignite pull request: IGNITE-1397 Load/consistency test framework

[ashutakGG]

GitHub user ashutakGG opened a pull request:

https://github.com/apache/ignite/pull/145

IGNITE-1397 Load/consistency test framework



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ashutakGG/incubator-ignite ignite-1397-yards

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/145.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #145


commit 5747042f189aacb524c560473bb2f0f9ebb333ab
Author: ashutak 
Date:   2015-09-09T14:04:05Z

tmp

commit 220ee836c1004d49460dc927a0282cb6470996f7
Author: ashutak 
Date:   2015-09-11T14:08:16Z

[migration from gg-6 to ignite] gg-6 client

commit fbc3d196ee3254fc34a0dca98536efa0f0caf337
Author: ashutak 
Date:   2015-09-11T16:37:33Z

[migration from gg-6 to ignite] ignite + gg

commit 2a5454fe9ab4bc1bd6cd6e6e378a8224009d51a7
Author: ashutak 
Date:   2015-09-11T16:46:09Z

[migration from gg-6 to ignite] stabilization

commit e1c47b24e426bf4168a878ce2037004674650ced
Author: ashutak 
Date:   2015-09-11T18:05:32Z

[migration from gg-6 to ignite] the end

commit 6a3c60db814ac519abe742347613314f3984d394
Author: ashutak 
Date:   2015-09-14T12:11:14Z

kill9

commit 218a7b3a40eeb9bb49a9f47814441db6226444bd
Author: ashutak 
Date:   2015-09-14T14:58:09Z

Kill9 implementation

commit 8fdcafbe15d2a19b8b0f5b3c388645c4474010d9
Author: ashutak 
Date:   2015-09-15T09:13:41Z

sh-s

commit 6eef978e063526995ffb7a6cb26d2ec2926f1aa7
Author: ashutak 
Date:   2015-09-15T09:14:03Z

sh

commit 9e3c538703ef98eefa8e2a625ac28bf6071fc291
Author: ashutak 
Date:   2015-09-15T15:06:42Z

tmp

commit 464774d71e4053df2ef25672a8e2df509bbe5c42
Author: ashutak 
Date:   2015-09-16T14:54:25Z

sh-s

commit 6a90380142c964a76e65240cb6cf25df259d664d
Author: ashutak 
Date:   2015-09-16T17:09:07Z

first working version

commit 62695ad164f59616ade4fa6265f82fde37e392f5
Author: ashutak 
Date:   2015-09-16T18:31:01Z

cyclic implementation

commit fae9d417eb8fb5db2f338dcb7eea62a671b764b4
Author: ashutak 
Date:   2015-09-16T19:27:02Z

impelment todos

commit 477b1183866a90b23701a664ff4f25a1f419e204
Author: ashutak 
Date:   2015-09-17T12:44:10Z

clean-up

commit 704abf39af8ddcbe0dbcc2615ecf3a17fc3b6ccb
Author: ashutak 
Date:   2015-09-17T13:29:56Z

fix log file name

commit 20b9feee706415c8b7950abc3bfea78d26b29a88
Author: ashutak 
Date:   2015-09-17T13:40:30Z

clean up at bin-s

commit bb7aa8d1dc693da2e2c623e388ef7719bf90c02c
Author: ashutak 
Date:   2015-09-17T14:20:52Z

atomic retries failover benchmark

commit 23d5b2de5919ccf1fe9cd1af2dce56a6dac83810
Author: ashutak 
Date:   2015-09-17T14:50:50Z

fix benchmark name at benchmark-failover.properties

commit 865afcde101be577d475c87148a7d144e09a48b6
Author: ashutak 
Date:   2015-09-17T18:36:21Z

fix IgniteAtomicRetriesFailoverBenchmark, implement 
IgniteTransactionalWriteReadBenchmark

commit a7d83b6c0e077c40b6f9c7901a26d33ba4b1f2cd
Author: ashutak 
Date:   2015-09-17T19:04:43Z

fix cache configuration for IgniteTransactionalWriteReadBenchmark

commit 5b60c381f71ae4c1f7318c213296718ec0439a8a
Author: ashutak 
Date:   2015-09-17T19:29:21Z

impelement  IgniteTransactionalWriteInvokeBenchmark

commit 0392a9907c4f98bada6103773b0cb94ee3bda62a
Author: ashutak 
Date:   2015-09-18T09:15:19Z

fix NPE at IgniteTransactionalWriteInvokeBenchmark

commit 7451b3032a50d506b7994ad4747882a1147fe3ce
Author: ashutak 
Date:   2015-09-18T09:49:55Z

NPE + change benchmark description

commit 47fb94c235ba3eb07dfb7eda51d8dd6e6ef87675
Author: ashutak 
Date:   2015-09-18T13:45:25Z

Merge remote-tracking branch 'apache/ignite-1.4' into ignite-1.4

commit 2137d029c3074cb9f26a96e3e8d0d8651ffbf6e5
Author: ashutak 
Date:   2015-09-18T17:06:30Z

IgniteTransactionalInvokeRetryBenchmark impl

commit 7063d8ed01bc28382ff17792b602e90ed3be7816
Author: ashutak 
Date:   2015-09-21T12:57:34Z

ignite-failover-configs

commit 955a213e1eef5c0a3fd68a757de8dfb7c559ca7f
Author: ashutak 
Date:   2015-09-21T17:46:44Z

refactoring + failureDetectionTimeout + updates + implement new args

commit 5de87cdf3810d806f280dd3619003d0c783b3fd2
Author: ashutak 
Date:   

[GitHub] ignite pull request: IGNITE-1627 .Net: Consistent product naming.

[ptupitsyn]

GitHub user ptupitsyn opened a pull request:

https://github.com/apache/ignite/pull/143

IGNITE-1627 .Net: Consistent product naming.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ptupitsyn/ignite ignite-1627

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/143.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #143


commit 47f1f39cc2bf458bae52622f0960490e74633b10
Author: ptupitsyn 
Date:   2015-10-08T10:34:44Z

IGNITE-1627 .Net: Consistent product naming.

commit bd87e25d66f4f19ff0443ef282ff97e552976e27
Author: ptupitsyn 
Date:   2015-10-08T10:42:56Z

wip

commit 618d8e9cee1ee5fb0f1257350716dd563c6e752a
Author: ptupitsyn 
Date:   2015-10-08T10:46:10Z

IGNITE-1627 .Net: Consistent product naming: Apache Ignite.NET




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-1635) Cache.invoke() can work wrong in a failover scenario

[Artem Shutak (JIRA)]

Artem Shutak created IGNITE-1635:


 Summary: Cache.invoke() can work wrong in a failover scenario
 Key: IGNITE-1635
 URL: https://issues.apache.org/jira/browse/IGNITE-1635
 Project: Ignite
  Issue Type: Bug
Reporter: Artem Shutak
Assignee: Artem Shutak
 Fix For: ignite-1.5


In process of working on IGNITE-1397 (Load/consistency test framework) I've 
faced with a wrong cache state after Cache.invoke().

Configuration:
- 1 client node
- 3 server nodes (1 backup)
- duration - 3 hours
- one random server node is killed every 10 min and after 2 minutes started 
again

h5. Transactional invoke retry consistency
Each client generates a random integer K in a limited range (say, [0, 10]) 
and creates a batch of 5 (may be configured) keys in the form 'key-' + K + '-' 
+ n + '-1', 'key-' + K + '-' + n + '-2', ... where n is a unique sequence 
number assigned to the client. Each client maintains a local map that it 
updates together with cache. Client invokes an increment closure for all 
generated keys (no explicit transaction is used) and atomically increments 
value for corresponding keys in the local map. No exceptions must be thrown to 
the user code. To validate cache contents, all writes from the client should be 
stopped, values in the local map must be equal to the values in the cache.

After 25 min in logs of client: {{Got different values [key='key-89611-0-3', 
cacheVal=213, localMapVal=214]}}.

All other "key-89611-0-*" keys in cache have value 214. For the key 
'key-89611-0-3' incremental cache invoke didn't happen.







--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] ignite pull request: IGNITE-1627 .Net: Consistent product naming.

[ptupitsyn]

GitHub user ptupitsyn opened a pull request:

https://github.com/apache/ignite/pull/144

IGNITE-1627 .Net: Consistent product naming.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ptupitsyn/ignite ignite-1627

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/144.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #144


commit e9bdd74e459c86f940bbebefc41d0372c7f40f34
Author: ptupitsyn 
Date:   2015-10-08T11:17:25Z

IGNITE-1627 .Net: Consistent product naming.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request: IGNITE-1627 .Net: Consistent product naming.

[ptupitsyn]

Github user ptupitsyn closed the pull request at:

https://github.com/apache/ignite/pull/143


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-1638) IGFS: Review IgfsImpl.checkConflictWithPrimary() method.

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-1638:
---

 Summary: IGFS: Review IgfsImpl.checkConflictWithPrimary() method.
 Key: IGNITE-1638
 URL: https://issues.apache.org/jira/browse/IGNITE-1638
 Project: Ignite
  Issue Type: Task
  Components: hadoop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
 Fix For: ignite-1.5


This method looks very suspicious to me. It checks whether secondary file 
system contains any file under the same path as IGFS primary direcotry. If yes 
- an exception is thrown.
But why? What is the use case we are trying to resolve with this?

We need to re-think this approach and decide whether to leave it or remove it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1637) IGFS: Consistent properties propagation.

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-1637:
---

 Summary: IGFS: Consistent properties propagation.
 Key: IGNITE-1637
 URL: https://issues.apache.org/jira/browse/IGNITE-1637
 Project: Ignite
  Issue Type: Task
  Components: hadoop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.5


IGFS has methods accepting properties map. E.g., create, append, mkdirs. 
How we should apply these properties? Two strategies are possible:
1) Apply there properties only to leaf node, and set defaults to parent nodes.
2) Apply there properties to all created nodes.

Lets take HDFS as a reference for this. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: ignite-1.4 version is still "unreleased" in Jira

[Yakov Zhdanov]

I released "ignite-1.4".

And also removed "1.5". Who created it? Why not rename "ignite-1.5"?

--Yakov

2015-10-07 19:28 GMT+03:00 Konstantin Boudnik :

> Techically it should be the RM. I am looking through the website and wiki
> and
> I can not find the description of the release procedure. Unless it is
> _fully_
> automatic - it'd make sense to write down, so the hard-gained knowledge
> isn't
> at loss.
>
> Thanks,
>   Cos
>
> On Wed, Oct 07, 2015 at 03:53PM, Pavel Tupitsyn wrote:
> > Hi Igniters,
> >
> > 1.4 has been released, but Jira shows "ignite-1.4" version as
> "unreleased".
> > Can anyone with corresponding rights please fix this?
> >
> > Thanks,
> >
> > --
> > --
> > Pavel Tupitsyn
> > GridGain Systems, Inc.
> > www.gridgain.com
>

[jira] [Created] (IGNITE-1639) IGFS: IgfsMetaManager() contains duplicated "delete" method.

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-1639:
---

 Summary: IGFS: IgfsMetaManager() contains duplicated "delete" 
method.
 Key: IGNITE-1639
 URL: https://issues.apache.org/jira/browse/IGNITE-1639
 Project: Ignite
  Issue Type: Task
  Components: hadoop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
 Fix For: ignite-1.6


Our metamanager has two "delete" methods which do roughly the same thing: 
delete leafs and update parent listing. But one method remove collection of 
leafs, while another remove only single leaf.

Do we really need to have them as separate methods? Looks like them can be 
merged into one.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1636) CPP: Add read/write methods with iterators for maps.

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-1636:
---

 Summary: CPP: Add read/write methods with iterators for maps.
 Key: IGNITE-1636
 URL: https://issues.apache.org/jira/browse/IGNITE-1636
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
 Fix For: ignite-1.5


This needs to be done in the same way as we did it for collections.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1641) .Net: Get rid of ContinuousQueryFilterHolder

[Pavel Tupitsyn (JIRA)]

Pavel  Tupitsyn created IGNITE-1641:
---

 Summary: .Net: Get rid of ContinuousQueryFilterHolder
 Key: IGNITE-1641
 URL: https://issues.apache.org/jira/browse/IGNITE-1641
 Project: Ignite
  Issue Type: Improvement
  Components: interop
Affects Versions: ignite-1.5
Reporter: Pavel  Tupitsyn
Assignee: Pavel  Tupitsyn
 Fix For: ignite-1.5


ContinuousQueryFilterHolder is just a filter+flag.
We should serialize them separately so that any platform can see the 
keepPortable flag (as opposed to a wrapper that only .Net can read).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: ignite-1.4 version is still "unreleased" in Jira

[Yakov Zhdanov]

I renamed "ignite-1.5" and "ignite-1.6" to 1.5 and 1.6

--Yakov

2015-10-08 17:38 GMT+03:00 Yakov Zhdanov :

> This is very easy.
>
> --Yakov
>
> 2015-10-08 17:35 GMT+03:00 Dmitriy Setrakyan :
>
>> On Thu, Oct 8, 2015 at 7:31 AM, Yakov Zhdanov 
>> wrote:
>>
>> > I released "ignite-1.4".
>> >
>> > And also removed "1.5". Who created it? Why not rename "ignite-1.5"?
>> >
>>
>> Yakov, there was discussion to remove "ignite" word from all the release
>> names in Jira and I think you agreed there. I also think it makes sense.
>>
>> Is it easy to just rename the release names in Jira?
>>
>>
>> > --Yakov
>> >
>> > 2015-10-07 19:28 GMT+03:00 Konstantin Boudnik :
>> >
>> > > Techically it should be the RM. I am looking through the website and
>> wiki
>> > > and
>> > > I can not find the description of the release procedure. Unless it is
>> > > _fully_
>> > > automatic - it'd make sense to write down, so the hard-gained
>> knowledge
>> > > isn't
>> > > at loss.
>> > >
>> > > Thanks,
>> > >   Cos
>> > >
>> > > On Wed, Oct 07, 2015 at 03:53PM, Pavel Tupitsyn wrote:
>> > > > Hi Igniters,
>> > > >
>> > > > 1.4 has been released, but Jira shows "ignite-1.4" version as
>> > > "unreleased".
>> > > > Can anyone with corresponding rights please fix this?
>> > > >
>> > > > Thanks,
>> > > >
>> > > > --
>> > > > --
>> > > > Pavel Tupitsyn
>> > > > GridGain Systems, Inc.
>> > > > www.gridgain.com
>> > >
>> >
>>
>
>

[jira] [Created] (IGNITE-1642) .Net: Rename EventType members

[Pavel Tupitsyn (JIRA)]

Pavel  Tupitsyn created IGNITE-1642:
---

 Summary: .Net: Rename EventType members
 Key: IGNITE-1642
 URL: https://issues.apache.org/jira/browse/IGNITE-1642
 Project: Ignite
  Issue Type: Improvement
  Components: interop
Affects Versions: 1.5
Reporter: Pavel  Tupitsyn
Assignee: Pavel  Tupitsyn
 Fix For: 1.5


* Remove Event/Events prefix
* For collections of events use All postfix (JobAll, CacheAll, etc)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1643) Cleanup platform communication protocol.

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-1643:
---

 Summary: Cleanup platform communication protocol.
 Key: IGNITE-1643
 URL: https://issues.apache.org/jira/browse/IGNITE-1643
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Blocker
 Fix For: 1.5


At the moment we have lots of .Net-specific objects travelling over a wire. 
E.g., user's .Net job is wrapped in some holder, then serialized and passed to 
Java, then Java pass this object over wire, etc..

This approach have two drawbacks:
1) Adds unnecessary overhead on wrapping.
2) Java have no chance to understand what it is passing. 

If we unwrap all .Net objects and pass them to Java as is, we can achieve 
slightly better performance. But what is *much more* important - we will be 
able to unleash possibilities for very cool transparent interoperability 
between platforms. E.g., .Net will be able to execute Java services, Java node 
will be able to execute .Net/CPP jobs if it has matching portable class 
implemented, etc..

Alternatively we can leave wrappers as is, but implement them in Java/CPP as 
well. This is not very good idea because of performance overhead.

The feature *must be implemented in 1.5* because now we are free to change 
protocol as we want.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: ignite-1.4 version is still "unreleased" in Jira

[Konstantin Boudnik]

Thank you very much !

On October 8, 2015 7:45:39 AM PDT, Yakov Zhdanov  wrote:
>I renamed "ignite-1.5" and "ignite-1.6" to 1.5 and 1.6
>
>--Yakov
>
>2015-10-08 17:38 GMT+03:00 Yakov Zhdanov :
>
>> This is very easy.
>>
>> --Yakov
>>
>> 2015-10-08 17:35 GMT+03:00 Dmitriy Setrakyan :
>>
>>> On Thu, Oct 8, 2015 at 7:31 AM, Yakov Zhdanov 
>>> wrote:
>>>
>>> > I released "ignite-1.4".
>>> >
>>> > And also removed "1.5". Who created it? Why not rename
>"ignite-1.5"?
>>> >
>>>
>>> Yakov, there was discussion to remove "ignite" word from all the
>release
>>> names in Jira and I think you agreed there. I also think it makes
>sense.
>>>
>>> Is it easy to just rename the release names in Jira?
>>>
>>>
>>> > --Yakov
>>> >
>>> > 2015-10-07 19:28 GMT+03:00 Konstantin Boudnik :
>>> >
>>> > > Techically it should be the RM. I am looking through the website
>and
>>> wiki
>>> > > and
>>> > > I can not find the description of the release procedure. Unless
>it is
>>> > > _fully_
>>> > > automatic - it'd make sense to write down, so the hard-gained
>>> knowledge
>>> > > isn't
>>> > > at loss.
>>> > >
>>> > > Thanks,
>>> > >   Cos
>>> > >
>>> > > On Wed, Oct 07, 2015 at 03:53PM, Pavel Tupitsyn wrote:
>>> > > > Hi Igniters,
>>> > > >
>>> > > > 1.4 has been released, but Jira shows "ignite-1.4" version as
>>> > > "unreleased".
>>> > > > Can anyone with corresponding rights please fix this?
>>> > > >
>>> > > > Thanks,
>>> > > >
>>> > > > --
>>> > > > --
>>> > > > Pavel Tupitsyn
>>> > > > GridGain Systems, Inc.
>>> > > > www.gridgain.com
>>> > >
>>> >
>>>
>>
>>

Re: ignite-1.4 version is still "unreleased" in Jira

[Dmitriy Setrakyan]

On Thu, Oct 8, 2015 at 7:31 AM, Yakov Zhdanov  wrote:

> I released "ignite-1.4".
>
> And also removed "1.5". Who created it? Why not rename "ignite-1.5"?
>

Yakov, there was discussion to remove "ignite" word from all the release
names in Jira and I think you agreed there. I also think it makes sense.

Is it easy to just rename the release names in Jira?


> --Yakov
>
> 2015-10-07 19:28 GMT+03:00 Konstantin Boudnik :
>
> > Techically it should be the RM. I am looking through the website and wiki
> > and
> > I can not find the description of the release procedure. Unless it is
> > _fully_
> > automatic - it'd make sense to write down, so the hard-gained knowledge
> > isn't
> > at loss.
> >
> > Thanks,
> >   Cos
> >
> > On Wed, Oct 07, 2015 at 03:53PM, Pavel Tupitsyn wrote:
> > > Hi Igniters,
> > >
> > > 1.4 has been released, but Jira shows "ignite-1.4" version as
> > "unreleased".
> > > Can anyone with corresponding rights please fix this?
> > >
> > > Thanks,
> > >
> > > --
> > > --
> > > Pavel Tupitsyn
> > > GridGain Systems, Inc.
> > > www.gridgain.com
> >
>

Ignite nodes under different JDKs

[Andrey Gura]

Igniters,

There are cases when Ignite cluster nodes work on different environments
and JDKs (versions and/or vendors). GridDiscoveryManager class contains
check that all nodes in topology ran under JDKs with the same major Java
version and throws exception if check failed. I want to replace this on
warning message. So I have two questions:

1. Is there any objections about it?
2. What message will be more informative than current one ("Local node's
java major version is different from remote node's one
[locJvmMajVer=, rmtJvmMajVer=]")?


-- 
Andrey Gura
GridGain Systems, Inc.
www.gridgain.com

Re: Ignite nodes under different JDKs

[Dmitriy Setrakyan]

On Thu, Oct 8, 2015 at 9:52 AM, Andrey Gura  wrote:

> Igniters,
>
> There are cases when Ignite cluster nodes work on different environments
> and JDKs (versions and/or vendors). GridDiscoveryManager class contains
> check that all nodes in topology ran under JDKs with the same major Java
> version and throws exception if check failed. I want to replace this on
> warning message. So I have two questions:
>
> 1. Is there any objections about it?
>

I agree. Ignite should allow nodes with different JDKs join the cluster.


> 2. What message will be more informative than current one ("Local node's
> java major version is different from remote node's one
> [locJvmMajVer=, rmtJvmMajVer=]")?
>

How about:
---
Local java version is different from remote [loc=,
rmt=]"
---


>
> --
> Andrey Gura
> GridGain Systems, Inc.
> www.gridgain.com
>

Re: Ignite nodes under different JDKs

[Dmitriy Setrakyan]

On Thu, Oct 8, 2015 at 11:31 AM, Vladimir Ozerov 
wrote:

> Why are we sticked to version? If both JVM has the same major version, but
> different vendors, it might be even more important concern, than different
> major versions of the same vendor.
>

I don't think even this should matter. Java has a well defined Class
format, so different vendors should not have any affect on this.

However, I do agree, that we should test not only different versions, but
different JDK vendors as well, e.g. Oracle JDK, Open JDK, and IBM JDK.


>
> On Thu, Oct 8, 2015 at 8:48 PM, Dmitriy Setrakyan 
> wrote:
>
> > On Thu, Oct 8, 2015 at 9:52 AM, Andrey Gura  wrote:
> >
> > > Igniters,
> > >
> > > There are cases when Ignite cluster nodes work on different
> environments
> > > and JDKs (versions and/or vendors). GridDiscoveryManager class contains
> > > check that all nodes in topology ran under JDKs with the same major
> Java
> > > version and throws exception if check failed. I want to replace this on
> > > warning message. So I have two questions:
> > >
> > > 1. Is there any objections about it?
> > >
> >
> > I agree. Ignite should allow nodes with different JDKs join the cluster.
> >
> >
> > > 2. What message will be more informative than current one ("Local
> node's
> > > java major version is different from remote node's one
> > > [locJvmMajVer=, rmtJvmMajVer=]")?
> > >
> >
> > How about:
> > ---
> > Local java version is different from remote [loc=,
> > rmt=]"
> > ---
> >
> >
> > >
> > > --
> > > Andrey Gura
> > > GridGain Systems, Inc.
> > > www.gridgain.com
> > >
> >
>

Re: Ignite nodes under different JDKs

[Konstantin Boudnik]

This conversation reminds me of the situation with Spark and akka that I just
ran into. Or rather with Akka and the way they designed the remote execution.
The situation is actually _completely_ ridiculous. I stood up a small Spark
cluster and then tried to submit a job into it, which had some
Spark dependencies. The way the job is written it pulls the dependencies
automatically from the maven repo. To my horror, the job was crashing because
local and remote serialIDs of the classes differed, although the dependency
versions were the same. The root cause is this: the versions are compiled with
the same version of JDK (like JDK7) or something, but one is Open and the
other one is Oracle's.

I think this is a very shaky way of designing the software for distributed
environments and it badly complicates the operation and integration of the
clusters. It clearly shows the lack practical experience beyond the academic
ivory towers on the account of Akka guys. RPC, while not without its own
issues, allows to get around such problems with ease.

I guess what I am saying: aren't we trying to find an even more complex
solution for already pretty tough problem?

Cos

On Thu, Oct 08, 2015 at 09:31PM, Vladimir Ozerov wrote:
> Why are we sticked to version? If both JVM has the same major version, but
> different vendors, it might be even more important concern, than different
> major versions of the same vendor.
> 
> On Thu, Oct 8, 2015 at 8:48 PM, Dmitriy Setrakyan 
> wrote:
> 
> > On Thu, Oct 8, 2015 at 9:52 AM, Andrey Gura  wrote:
> >
> > > Igniters,
> > >
> > > There are cases when Ignite cluster nodes work on different environments
> > > and JDKs (versions and/or vendors). GridDiscoveryManager class contains
> > > check that all nodes in topology ran under JDKs with the same major Java
> > > version and throws exception if check failed. I want to replace this on
> > > warning message. So I have two questions:
> > >
> > > 1. Is there any objections about it?
> > >
> >
> > I agree. Ignite should allow nodes with different JDKs join the cluster.
> >
> >
> > > 2. What message will be more informative than current one ("Local node's
> > > java major version is different from remote node's one
> > > [locJvmMajVer=, rmtJvmMajVer=]")?
> > >
> >
> > How about:
> > ---
> > Local java version is different from remote [loc=,
> > rmt=]"
> > ---
> >
> >
> > >
> > > --
> > > Andrey Gura
> > > GridGain Systems, Inc.
> > > www.gridgain.com
> > >
> >

Using IgniteContext in... well, Java context

[Konstantin Boudnik]

Guys,

I've tried to play a little bit with IgniteContext and the whole IgniteRDD
stuff from the perspective of not touching Scala ever again. And here's what I
have found: IgniteContext isn't usable from Java (or Groovy for that matter).
And it isn't an attempt to critique Ignite's RDD implementation, because we
have to follow the design patterns setup by the 3rd party platform, Spark in
this case.

If I want to submit a Java-based job into spark cluster I need to do something
like this:

import org.apache.ignite.spark.*
import org.apache.spark.SparkConf
import org.apache.spark.api.java.JavaSparkContext

SparkConf sparkConf = new SparkConf().setAppName('SharedCache')
sparkConf.setMaster('spark://master.spark.mydomain.com:7077')
JavaSparkContext jsc = new JavaSparkContext(sparkConf)

so far so good. Now, if I want to use Ignite in it, I should be doing 

def ic = new IgniteContext(jsc, "spark-ignite-config.xml")

and that's where I hit the wall, because JavaSparkContext isn't a subtype of
SparkContext in Spark world, and I can not cast one to another.

Well then, perhaps I can use SparkContext and hope for the best? Turns out I
can not, because the use of the SparkContext gets me into the swamp of Scala
type-system, with long-mnemonic names like Function1 and Tuple2 (I reckon the
day will come soon, when they will have Function27 and so on). This leads me
to believe the current implementation of IgniteRDD is only good to be used
from Scala, unless I am completely wrong and don't know what I am talking
about. Which might be quite possible, of course.

My question is very simple: is there a way to use IgniteRDD from Java-language
family, e.g. Java and Groovy, or there has to be JavaIgniteRDD implementation
of it much like the state of things is in the Spark itself?

Thanks for any feedback
-- 
  Cos

Using IgniteContext in... well, Java context

[Konstantin Boudnik]

Guys,

I've tried to play a little bit with IgniteContext and the whole IgniteRDD
stuff from the perspective of not touching Scala ever again. And here's what I
have found: IgniteContext isn't usable from Java (or Groovy for that matter).
And it isn't an attempt to critique Ignite's RDD implementation, because we
have to follow the design patterns setup by the 3rd party platform, Spark in
this case.

If I want to submit a Java-based job into spark cluster I need to do something
like this:

import org.apache.ignite.spark.*
import org.apache.spark.SparkConf
import org.apache.spark.api.java.JavaSparkContext

SparkConf sparkConf = new SparkConf().setAppName('SharedCache')
sparkConf.setMaster('spark://master.spark.mydomain.com:7077')
JavaSparkContext jsc = new JavaSparkContext(sparkConf)

so far so good. Now, if I want to use Ignite in it, I should be doing 

def ic = new IgniteContext(jsc, "spark-ignite-config.xml")

and that's where I hit the wall, because JavaSparkContext isn't a subtype of
SparkContext in Spark world, and I can not cast one to another.

Well then, perhaps I can use SparkContext and hope for the best? Turns out I
can not, because the use of the SparkContext gets me into the swamp of Scala
type-system, with long-mnemonic names like Function1 and Tuple2 (I reckon the
day will come soon, when they will have Function27 and so on). This leads me
to believe the current implementation of IgniteRDD is only good to be used
from Scala, unless I am completely wrong and don't know what I am talking
about. Which might be quite possible, of course.

My question is very simple: is there a way to use IgniteRDD from Java-language
family, e.g. Java and Groovy, or there has to be JavaIgniteRDD implementation
of it much like the state of things is in the Spark itself?

Thanks for any feedback
-- 
  Cos

Re: Ignite nodes under different JDKs

[Dmitriy Setrakyan]

On Thu, Oct 8, 2015 at 1:56 PM, Konstantin Boudnik  wrote:

> On Thu, Oct 08, 2015 at 12:46PM, Dmitriy Setrakyan wrote:
> > On Thu, Oct 8, 2015 at 12:28 PM, Konstantin Boudnik 
> wrote:
> >
> > > This conversation reminds me of the situation with Spark and akka that
> I
> > > just
> > > ran into. Or rather with Akka and the way they designed the remote
> > > execution.
> > > The situation is actually _completely_ ridiculous. I stood up a small
> Spark
> > > cluster and then tried to submit a job into it, which had some
> > > Spark dependencies. The way the job is written it pulls the
> dependencies
> > > automatically from the maven repo. To my horror, the job was crashing
> > > because
> > > local and remote serialIDs of the classes differed, although the
> dependency
> > > versions were the same. The root cause is this: the versions are
> compiled
> > > with
> > > the same version of JDK (like JDK7) or something, but one is Open and
> the
> > > other one is Oracle's.
> > >
> > > I think this is a very shaky way of designing the software for
> distributed
> > > environments and it badly complicates the operation and integration of
> the
> > > clusters. It clearly shows the lack practical experience beyond the
> > > academic
> > > ivory towers on the account of Akka guys. RPC, while not without its
> own
> > > issues, allows to get around such problems with ease.
> > >
> > > I guess what I am saying: aren't we trying to find an even more complex
> > > solution for already pretty tough problem?
> > >
> >
> > I think that the problem you are describing is not the same. What we are
> > solving here is, for example, ability to run Ignite with IBM WebSphere on
> > the client side and OpenJDK on the server side.
> >
> > This issue has little to do with dependencies, and mostly with removing a
> > legacy restriction from the project about matching JDK versions.
>
> The problem is the same: the use of dynamic dependencies just illustrates
> it
> clearly. Different JDKs are producing different serial.vers. of the classes
> and it will come and haunt you one way or another. The manifestation of the
> problem could be different, but you can count that the problem will be
> there
> for you on any heterogeneous cluster.
>

In the upcoming 1.5 release, this will only apply to compute grid and not
to data grid. I think we should print out a warning, but not disallow the
cluster startup, like we do now.


> Cos
>

Re: Ignite nodes under different JDKs

[Andrey Gura]

Vladimir,

different JDK versions in topology can lead to problems with P2P deploy for
example. Nodes under different vendors' JDK should work correctly in case
of properly defined SUID.

On Thu, Oct 8, 2015 at 9:31 PM, Vladimir Ozerov 
wrote:

> Why are we sticked to version? If both JVM has the same major version, but
> different vendors, it might be even more important concern, than different
> major versions of the same vendor.
>
> On Thu, Oct 8, 2015 at 8:48 PM, Dmitriy Setrakyan 
> wrote:
>
> > On Thu, Oct 8, 2015 at 9:52 AM, Andrey Gura  wrote:
> >
> > > Igniters,
> > >
> > > There are cases when Ignite cluster nodes work on different
> environments
> > > and JDKs (versions and/or vendors). GridDiscoveryManager class contains
> > > check that all nodes in topology ran under JDKs with the same major
> Java
> > > version and throws exception if check failed. I want to replace this on
> > > warning message. So I have two questions:
> > >
> > > 1. Is there any objections about it?
> > >
> >
> > I agree. Ignite should allow nodes with different JDKs join the cluster.
> >
> >
> > > 2. What message will be more informative than current one ("Local
> node's
> > > java major version is different from remote node's one
> > > [locJvmMajVer=, rmtJvmMajVer=]")?
> > >
> >
> > How about:
> > ---
> > Local java version is different from remote [loc=,
> > rmt=]"
> > ---
> >
> >
> > >
> > > --
> > > Andrey Gura
> > > GridGain Systems, Inc.
> > > www.gridgain.com
> > >
> >
>



-- 
Andrey Gura
GridGain Systems, Inc.
www.gridgain.com

Re: Ignite nodes under different JDKs

[Vladimir Ozerov]

Why are we sticked to version? If both JVM has the same major version, but
different vendors, it might be even more important concern, than different
major versions of the same vendor.

On Thu, Oct 8, 2015 at 8:48 PM, Dmitriy Setrakyan 
wrote:

> On Thu, Oct 8, 2015 at 9:52 AM, Andrey Gura  wrote:
>
> > Igniters,
> >
> > There are cases when Ignite cluster nodes work on different environments
> > and JDKs (versions and/or vendors). GridDiscoveryManager class contains
> > check that all nodes in topology ran under JDKs with the same major Java
> > version and throws exception if check failed. I want to replace this on
> > warning message. So I have two questions:
> >
> > 1. Is there any objections about it?
> >
>
> I agree. Ignite should allow nodes with different JDKs join the cluster.
>
>
> > 2. What message will be more informative than current one ("Local node's
> > java major version is different from remote node's one
> > [locJvmMajVer=, rmtJvmMajVer=]")?
> >
>
> How about:
> ---
> Local java version is different from remote [loc=,
> rmt=]"
> ---
>
>
> >
> > --
> > Andrey Gura
> > GridGain Systems, Inc.
> > www.gridgain.com
> >
>