Re: [jira] [Commented] (IGNITE-2693) withKeepBinary and non-binary marshallers

[Dood]

Anyone willing to help me a bit to finish this ticket for 1.6? :-) Thanks!

On 4/12/2016 6:03 AM, Vladimir Ozerov (JIRA) wrote:

 [ 
https://issues.apache.org/jira/browse/IGNITE-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15237114#comment-15237114
 ]

Vladimir Ozerov commented on IGNITE-2693:
-

Hi Oddo,

Any news on the matter?

Vladimir.


withKeepBinary and non-binary marshallers
-

 Key: IGNITE-2693
 URL: https://issues.apache.org/jira/browse/IGNITE-2693
 Project: Ignite
  Issue Type: Bug
  Components: general
Reporter: Sergey Kozlov
Assignee: Oddo
  Labels: newbie
 Fix For: 1.6


Currently the user is able to set {{.withKeepBinary()}} for any used 
marshaller. But it obviously causes ClassCastException for non-binary 
marshallers and should be available only for binary marshaller.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: IGNITE-2693: question about setting BinaryMarshaller on a cache

[Dood]

On 3/9/2016 7:46 PM, Alexey Goncharuk wrote:

Note that withKeepBinary() is just a way to tell a cache not to deserialize
values when doing a get or running an entry processor. The concept of
binary object does not belong solely to caches - you can get an instance of
IgniteBinary interface from Ignite and use binary objects in computations,
for example.

For me there would be more confusion if each cache had a separate
marshaller. What would then happen if you put an instance of BinaryObject
to a cache with JDK marshaller? When marshaller is global, the answer is
simple - BinaryObject is either available or not :)


Alexey, thanks for the taking the time to explain the reasoning!

Re: IGNITE-2693: question about setting BinaryMarshaller on a cache

[Dood]

On 3/9/2016 6:43 PM, Alexey Goncharuk wrote:

Hi,

The current version of test is not very clean and it works only because
withKeepBinary() is a noop. The correct version would be to use plain cache
for non-binary-object entry processor and use withKeepBinary for
binary-object entry processor. You can see that EntryProcessor creation is
encapsulated in a separate method testClosure() which is overridden in test
inheritors. The same thing should be done for the cache.


Alexey, thank you for the comment.

On a side note: you do not find it confusing that you can set a 
marshaller on a grid but you get a binary cache from another cache via 
withKeepBinary()?


Thanks!

IGNITE-2693: question about setting BinaryMarshaller on a cache

[Dood]

Hello all,

I am working on IGNITE-2693 with Vlad Ozerov's help. I am somewhat of a 
Java newbie so please be gentle ;-)


I am curious about something - after reading the Javadocs and Binary 
Marshaller docs on Ignite's documentation websites, I think that the 
documentation is not very friendly or even somewhat misleading. Or maybe 
it is the design that is puzzling to me :-)


For example, we use withKeepBinary() on a cache instance to get a binary 
cache that utilizes the binary marshaller. But this is not a setting 
that is "settable" on a per cache basis - we do not allow for a 
per-cache method to set a desired marshaller, this seems to be reserved 
for the IgniteConfiguration() interface/implementation(s) 
setMarshaller() method and is a grid-wide setting.


The background to this is that I have "fixed" the withKeepBinary() 
interface to throw an exception if the marshaller used is not binary 
(the ticket explains the reason why we want this). Apparently we 
(silently?) assume a binary marshaller everywhere but in one of the 
unrelated tests in the test suite for some reason an optimized 
marshaller is used and as a result of this (with my new change) these 
tests are failing [1]. I am trying to fix this but in the process I 
realized that you cannot set the marshaller through a 
CacheConfiguration() method (no such thing exists), this has to be done 
at a higher level (the IgniteConfiguration). However, the whole test is 
written to inherit a grid configuration with an optimized marshaller (is 
what it looks like to me)


Am I just horribly confused and missing something very obvious? Thanks!

[1] 
org.apache.ignite.internal.processors.cache.GridCacheOffHeapTieredEvictionAbstractSelfTest 

IGNITE-2693 - could someone take a look and help me with a quick review

[Dood]

Hello,

This is my second Ignite ticket and if I understand correctly, it is a 
simple fix - I submitted a patch recently, it is only a few lines. Can 
someone take a look and see if I am on the right track or did I 
completely misunderstand it... :-)


Thanks!

Re: Switching back to review-then-commit process

[Dood]

+1 - sounds very reasonable and practical.

On 3/3/2016 5:54 AM, Denis Magda wrote:

Igniters,

I would propose to switch back to review-then-commit process. This 
process has to be followed by both contributors and committers.


There is a reason for this I have in mind. Ignite is a complex 
platform with several big modules. Some of the people may be experts 
in module A while others in module B etc.
If a committer, who is good in module A, makes changes in module B 
merging the changes without a review this can break module's B 
internal functionality that the committer didn't take into account.


My proposal is to introduce a list of maintainers for every Ignite 
module like it's done in Spark [1] and a rule that will require a 
committer to get an approval from a module maintainer before merging 
changes.


Thoughts?

--
Denis

[1] 
https://cwiki.apache.org/confluence/display/SPARK/Committers#Committers-ReviewProcessandMaintainers 

Re: Apache Arrow and Apache Ignite

[Dood]

On 2/25/2016 11:06 AM, Konstantin Boudnik wrote:

On Wed, Feb 24, 2016 at 02:02PM, Dood@ODDO wrote:

On 2/24/2016 1:31 PM, Konstantin Boudnik wrote:

On Sat, Feb 20, 2016 at 02:13PM, Dood@ODDO wrote:

That's the million dollar question - I think we should approach the
Arrow people and get a conversation going. We want to be ahead of
the curve, not behind it - Arrow seems to be making quite a stir,
not to mention that it was fast-tracked to mature project status
apparently solely based on the people/companies involved ;-)

I am afraid you misunderstand the point of the 'fast tracking'. To start with,
ASF doesn't case what companies are involved nor there's no such thing as fast
tracking. However, there's 'direct to TLP' track, that allows for a community
that has enough ASF members and other people, with lengthy involvement into
Apache projects, to go directly into TLP. The reason is simple: these people
are well versed in Apache Way and the incubation isn't required for me.

So, please stop using this 'fast-tracked' confusion.
   Cos

P.S. Another point: TLP != 'mature project status'. Please educate yourself
about the Apache incubation, before making statement like this in the future.


I will educate myself. What you need to do is learn to be polite. I
am not sure you can teach that though ;(

If you don't mind elaborating on what particularly was impolite in my email, 
you'll
put me forever into your debts. And teach me a lesson of good manners too.

Cos
I don't know, you just came off as angered by what I said. You took the 
time to supposedly educate me how "fast tracking" is not the same as " 
taking a project directly to TLP" (which I think is obviously the same 
as fast tracking since it skipped the incubating phase). Then you took 
more time to educate me how people/companies are not what Apache looked 
at when fast tracking this project (pardon, taking the route called 
"direct to TLP") but you said that people experienced in the ways of 
Apache are what allowed for this project to be fast tracked (pardon 
again, taken "direct to TLP").


Incidentally, who can join the dev list on the Arrow project at this 
stage? I tried a week ago and was unsuccessful. I was under the 
impression that anyone can join it.

Re: Apache Arrow and Apache Ignite

[Dood]

On 2/24/2016 1:31 PM, Konstantin Boudnik wrote:

On Sat, Feb 20, 2016 at 02:13PM, Dood@ODDO wrote:

That's the million dollar question - I think we should approach the
Arrow people and get a conversation going. We want to be ahead of
the curve, not behind it - Arrow seems to be making quite a stir,
not to mention that it was fast-tracked to mature project status
apparently solely based on the people/companies involved ;-)

I am afraid you misunderstand the point of the 'fast tracking'. To start with,
ASF doesn't case what companies are involved nor there's no such thing as fast
tracking. However, there's 'direct to TLP' track, that allows for a community
that has enough ASF members and other people, with lengthy involvement into
Apache projects, to go directly into TLP. The reason is simple: these people
are well versed in Apache Way and the incubation isn't required for me.

So, please stop using this 'fast-tracked' confusion.
   Cos

P.S. Another point: TLP != 'mature project status'. Please educate yourself
about the Apache incubation, before making statement like this in the future.



I will educate myself. What you need to do is learn to be polite. I am 
not sure you can teach that though ;(

Re: Binary mode for Data Structures

[Dood]

IGNITE-1144 has been fixed so at least that obstacle is out of the way :-)

On 2/24/2016 9:10 AM, Denis Magda wrote:
The community has already faced with the issue that the binary mode is 
not supported for data structures.

https://issues.apache.org/jira/browse/IGNITE-2339

I've linked this ticket into IGNITE-2701.

--
Denis

On 2/23/2016 4:29 AM, Valentin Kulichenko wrote:
+1 to this change. It sounds like a very important feature for 
collocated

queues, especially after [1] is fixed.

[1] https://issues.apache.org/jira/browse/IGNITE-1144

-Val

On Sat, Feb 20, 2016 at 3:42 AM, Pavel Tupitsyn 
wrote:


No replies, so I've created an issue:
https://issues.apache.org/jira/browse/IGNITE-2701

On Thu, Feb 18, 2016 at 3:06 PM, Pavel Tupitsyn 


wrote:


Igniters,

Currently we don't have binary mode (withKeepBinary) in Data 
Structures

(Queue, AtomicReference).
Are there any plans for implementing this, or may be a workaround?

Lack of binary mode prevents us from implementing IgniteQueue in .NET
(.NET types in most cases can't be deserialized in Java).

AtomicReference is already implemented, and it works because values 
are

wrapped in GridCacheAtomicReferenceValue. So there is inconsistent

behavior

between data structures.

Thoughts?

--
--
Pavel Tupitsyn
GridGain Systems, Inc.
www.gridgain.com




--
--
Pavel Tupitsyn
GridGain Systems, Inc.
www.gridgain.com

Re: IGNITE-1144

[Dood]

On 2/23/2016 7:05 PM, Valentin Kulichenko wrote:

I added couple more tests and merged your changes into master. Thanks for
the contribution!


Thanks! What other ticket do you think I could/should tackle?

Re: IGNITE-1144

[Dood]

Val,

All the test passed on TC - you can look at the PR at any time. Thanks!

On 2/22/2016 8:13 PM, Valentin Kulichenko wrote:

Hi,

Thanks for the pull request! I will take a look as soon as possible.

-Val

On Mon, Feb 22, 2016 at 9:14 AM, Dood@ODDO <oddodao...@gmail.com> wrote:


Hello all,

I was wondering if anyone can take a look at the PR I submitted for
IGNITE-1144 [1]. It compiles and I think the code is good but the tests I
submitted are failing - specifically, one test: I am using assertThrows()
to make sure that an exception is thrown if affinityRun() is invoked on a
queue that is not collocated. In the tests I basically set up a queue that
has a configuration that invokes setCollocated(false) and then I invoke an
affinityRun() with an "empty" job. For some reason the tests are failing
with an error that the exception wasn't thrown. However, in this case I
actually think that my test is bad and the PR code is good ;-). Can someone
with more Java experience take a quick look?

Thanks!

[1] https://issues.apache.org/jira/browse/IGNITE-1144

IGNITE-1144

[Dood]

Hello all,

I was wondering if anyone can take a look at the PR I submitted for 
IGNITE-1144 [1]. It compiles and I think the code is good but the tests 
I submitted are failing - specifically, one test: I am using 
assertThrows() to make sure that an exception is thrown if affinityRun() 
is invoked on a queue that is not collocated. In the tests I basically 
set up a queue that has a configuration that invokes 
setCollocated(false) and then I invoke an affinityRun() with an "empty" 
job. For some reason the tests are failing with an error that the 
exception wasn't thrown. However, in this case I actually think that my 
test is bad and the PR code is good ;-). Can someone with more Java 
experience take a quick look?


Thanks!

[1] https://issues.apache.org/jira/browse/IGNITE-1144

Re: Apache Arrow and Apache Ignite

[Dood]

That's the million dollar question - I think we should approach the 
Arrow people and get a conversation going. We want to be ahead of the 
curve, not behind it - Arrow seems to be making quite a stir, not to 
mention that it was fast-tracked to mature project status apparently 
solely based on the people/companies involved ;-)


On 2/20/2016 11:14 AM, Dmitriy Setrakyan wrote:

It sounds to me that it does not store it’s own data, but allows to process
data from somewhere else, right? If yes, will I be able to take Ignite data
with Arrow and convert it to a columnar format for some other processing?

On Sat, Feb 20, 2016 at 2:48 AM, Roman Shtykh 
wrote:


It's a standard (including format and algorithms) to share in-memory data
between several big data platforms.
In my understanding, it is like placing data in in-memory columns in
Parquet and being able to use it in Kudu without serializing/deserializing.

-Roman

Async and sync ops in IgniteQueue and IgniteSet implementations

[Dood]

Hello all,

This may be a very dumb question (and feel free to "reprimand" me ;-) 
but I will ask it anyways.


I am working on https://issues.apache.org/jira/browse/IGNITE-1144 and 
one of the comments on the submitted code was that I marked both methods 
I am implemented as @IgniteAsyncSupported but I never actually provided 
an async implementation. All my methods do is call on methods already 
implemented in IgniteCompute.java (after verifying that the methods are 
called on data structures that are collocated) and my assumption was 
that the methods being called are already implemented in async/sync 
versions. Specifically, these methods are affinityRun() and affinityCall().


Furthermore, I noticed the following comment in IgniteQueue.java, for 
example:

""* All queue operations have synchronous and asynchronous counterparts."".

However, after looking at all the implementations of IgniteQueue 
interface in the code base, I could not find any actual implementations 
of asynchronous calls on any methods of queue or set. Am I just missing 
something really basic?


Thanks!

Re: Async and sync ops in IgniteQueue and IgniteSet implementations

[Dood]

Val,

My question was also - if all I am doing is calling affinityRun() and 
affinityCall() that is already implemented elsewhere (just making sure 
it is done on a collocated queue/set) - do I need to do anything special 
looking at IgniteComputeImpl.java it looks like affinityRun()/Call() are 
already implemented in async fashion?)?


Thanks!

On 2/12/2016 1:49 PM, Valentin Kulichenko wrote:

Hi,

First of all, the JavaDoc is incorrect, there are no async counterparts for
queue and set operations in the current API.

The question is - do we need them? I think we should have have them for new
affinityRun and affinityCall methods that you're adding, but I'm not sure
about others.

Does anyone has thoughts on this?

-Val

On Fri, Feb 12, 2016 at 11:07 AM, Dood@ODDO <oddodao...@gmail.com> wrote:


Hello all,

This may be a very dumb question (and feel free to "reprimand" me ;-) but
I will ask it anyways.

I am working on https://issues.apache.org/jira/browse/IGNITE-1144 and one
of the comments on the submitted code was that I marked both methods I am
implemented as @IgniteAsyncSupported but I never actually provided an async
implementation. All my methods do is call on methods already implemented in
IgniteCompute.java (after verifying that the methods are called on data
structures that are collocated) and my assumption was that the methods
being called are already implemented in async/sync versions. Specifically,
these methods are affinityRun() and affinityCall().

Furthermore, I noticed the following comment in IgniteQueue.java, for
example:
""* All queue operations have synchronous and asynchronous counterparts."".

However, after looking at all the implementations of IgniteQueue interface
in the code base, I could not find any actual implementations of
asynchronous calls on any methods of queue or set. Am I just missing
something really basic?

Thanks!

Compiling Ignite against different JDKs

[Dood]

Hello all,

I am working on a ticket that is not very difficult, if you are good 
with Idea, Java, Maven etc. I have been a developer in various languages 
but new to Java ;).


I had JDK8 installed and I introduced some changes to the code base that 
cause it to not compile (apparently) against the JDK7. I downloaded JDK7 
and installed it, I went into Project Structure->JDKs in Idea and added 
the version 7 and set it as default for the Ignite project. However, 
every time I try to build the ignite-core module, it complains about 
"invalid source release: 1.8". Clearly I am missing something - but 
what? Are there any instructions anyone can point to for switching JDKs 
in IntelliJ?


Thanks!

Re: affinityRun() and affinityCall() (JIRA ticket)

[Dood]

Val,

Before I go on and submit pull requests etc. - would you comment on the 
path I am taking with this? As I said I am not a JAVA developer but I am 
trying to teach myself the language and contribute at the same time ;)


Here are my thoughts on implementing this for the queue 
(GridCacheQueueAdapter.java). I have also declared the following in 
IgniteQueue.java:


@IgniteAsyncSupported
public void affinityRun(IgniteRunnable job) throws IgniteException;

@IgniteAsyncSupported
public  R affinityCall(IgniteCallable job) throws IgniteException;

Here is what is in GridCacheQueueAdapter.java

/** {@inheritDoc} */
public void affinityRun(IgniteRunnable job) {
if (!collocated)
throw new IgniteException("Illegal operation requested on non-collocated 
queue:affinityRun().");


try {
compute.affinityRun(cache.name(),queueKey,job);
}
catch (IgniteException e) {
throw e;
}
}

/** {@inheritDoc} */
public  R affinityCall(IgniteCallable job) {
if (!collocated)
throw new IgniteException("Illegal operation requested on non-collocated 
queue:affinityCall().");


try {
return compute.affinityCall(cache.name(),queueKey,job);
}
catch (IgniteException e) {
throw e;
}
}

I have included the following at the top of the class 
GridCacheQueueAdapter:

private final IgniteCompute compute;

this.compute = cctx.kernalContext().grid().compute();

Let me know what you think!

On 1/27/2016 3:55 PM, Valentin Kulichenko wrote:

Hi,

Both GridCacheQueueAdapter and GridCacheSetImpl have a reference to
GridCacheContext which represents the underlying cache for the data
structure. GridCacheContext.name() will give you the correct cache name
that you can use when calling affinityRun method.

-Val

On Wed, Jan 27, 2016 at 9:13 AM, Dood@ODDO <oddodao...@gmail.com> wrote:


Hello,

I am playing with https://issues.apache.org/jira/browse/IGNITE-1144 as
introduction to hacking on Ignite. I am not a Java developer by day but
have experience writing code in various languages. This is my first
in-depth exposure to Ignite internals (have lightly used it as a user in a
POC project).

Looking at this ticket, I am guessing that what it needs to do is get the
cache name from the kernel context. After that it can just pass on the call
(such as affinityRun()) to the regular affinityRun() call with the cache
name filled in as the first parameter. This is because an internal
(un-exposed) cache is used to track the queue/set data structures. Is this
all correct?

My question is: how do I get the cache name from within the queue
implementation.

Thanks!

affinityRun() and affinityCall() (JIRA ticket)

[Dood]

Hello,

I am playing with https://issues.apache.org/jira/browse/IGNITE-1144 as 
introduction to hacking on Ignite. I am not a Java developer by day but 
have experience writing code in various languages. This is my first 
in-depth exposure to Ignite internals (have lightly used it as a user in 
a POC project).


Looking at this ticket, I am guessing that what it needs to do is get 
the cache name from the kernel context. After that it can just pass on 
the call (such as affinityRun()) to the regular affinityRun() call with 
the cache name filled in as the first parameter. This is because an 
internal (un-exposed) cache is used to track the queue/set data 
structures. Is this all correct?


My question is: how do I get the cache name from within the queue 
implementation.


Thanks!