date:20151012

[GitHub] ignite pull request: IGNITE-1617: Strings usage adjusted

2015-10-12 Thread isapego

Github user isapego closed the pull request at:

https://github.com/apache/ignite/pull/133


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-1653) Need to remove lgpl-dependent examples from Ignite binaries

2015-10-12 Thread Vasilisa Sidorova (JIRA)

Vasilisa  Sidorova created IGNITE-1653:
--

 Summary: Need to remove lgpl-dependent examples from Ignite 
binaries
 Key: IGNITE-1653
 URL: https://issues.apache.org/jira/browse/IGNITE-1653
 Project: Ignite
  Issue Type: Bug
  Components: general
Affects Versions: 1.5
 Environment: Apache-Ignite-1.5.0 build #29
Reporter: Vasilisa  Sidorova
Priority: Critical
 Fix For: 1.5


Examples project from Ignite binaries couldn't be compiled due to there is 
lgpl-dependencies.

Solution: lgpl-dependent examples should be removed from Ignite src and bin




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1650) Add ability to specify thread pool for IgniteFuture listen/chain methods.

2015-10-12 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1650:
---

 Summary: Add ability to specify thread pool for IgniteFuture 
listen/chain methods.
 Key: IGNITE-1650
 URL: https://issues.apache.org/jira/browse/IGNITE-1650
 Project: Ignite
  Issue Type: Task
  Components: general
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: 1.6


Closures passed to IgniteFuture listen() and chain() methods are executed 
either in the same thread if future is completed, or in a completion thread 
(usually this is a thread from one of Ignite pools).

This enforces restrictions on what user can do in closures. He cannot use call 
operations, he cannot call any Ignite operations. Otherwise deadlocks or 
starvation could occur.

To fix that we should allow user to pass optional thread pool where passed 
closure should be executed. This already done in Java 8 CompletableFuture. We 
should do almost the same.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1651) Add version to portable object header.

2015-10-12 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1651:
---

 Summary: Add version to portable object header.
 Key: IGNITE-1651
 URL: https://issues.apache.org/jira/browse/IGNITE-1651
 Project: Ignite
  Issue Type: Task
  Components: general, interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
Priority: Blocker
 Fix For: 1.5






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: .Net: separate methods for async operations.

2015-10-12 Thread Vladimir Ozerov

No. "await" is actually return from the method immediately. Let me show it
again:

async Task GetAndMultiply() {
Task res =  cache.GetAsync(1);

await res;

return res.Result * res.Result;
}

maps to the following pseudo-code in Java:

Future getAndMultiply() {
Future res =  cache.getAsync(1);

return res.chain((f) => {
return f.get() * f.get();
});
}



On Mon, Oct 12, 2015 at 1:36 PM, Yakov Zhdanov  wrote:

> Is current thread blocked until "await" instruction is completed in
> parallel thread?
>
> --Yakov
>
> 2015-10-12 10:41 GMT+03:00 Vladimir Ozerov :
>
> > Example with Get() operation:
> >
> > async Task GetAndMultiply() {
> > // This line is executed in current thread.
> > Task res = cache.GetAsync(1);
> >
> > await res;
> >
> > // This code is executed in another thread when res is ready.
> > int mul = res.Result * res.Result;
> >
> > return mul;
> > }
> >
> > On Mon, Oct 12, 2015 at 10:12 AM, Dmitriy Setrakyan <
> dsetrak...@apache.org
> > >
> > wrote:
> >
> > > On Sun, Oct 11, 2015 at 3:42 AM, Vladimir Ozerov  >
> > > wrote:
> > >
> > > > Guys, let's try keeping this topic focused on .Net please, because
> the
> > > > product is not released yet and we can create any API we like.
> > > >
> > > > Dima, answering your question about async/await - this is actually
> > native
> > > > continuation support in .Net. Consider the following .Net method:
> > > >
> > > > async void PutAndPrint() {
> > > > await cache.PutAsync(1, 1);
> > > >
> > > > Console.WriteLine("Put value to cache.");
> > > > }
> > > >
> > >
> > > And what if the method putAsync would return a value. How would this
> code
> > > change?
> > >
> >
>

Re: .Net: separate methods for async operations.

2015-10-12 Thread Dmitriy Setrakyan

Do I understand correctly that you are suggesting adding "Async(..)"
counterparts for all the synchronous methods?

Are there any objections about this? If we do it in .NET, then we might as
well do it in Java, because in my view the same reasoning can be made for
Java. This will cause significant proliferation of Async methods. For
example just on IgniteCache API, we will have to add about 40 Async()
methods.

D.



On Mon, Oct 12, 2015 at 3:45 AM, Vladimir Ozerov 
wrote:

> No. "await" is actually return from the method immediately. Let me show it
> again:
>
> async Task GetAndMultiply() {
> Task res =  cache.GetAsync(1);
>
> await res;
>
> return res.Result * res.Result;
> }
>
> maps to the following pseudo-code in Java:
>
> Future getAndMultiply() {
> Future res =  cache.getAsync(1);
>
> return res.chain((f) => {
> return f.get() * f.get();
> });
> }
>
>
>
> On Mon, Oct 12, 2015 at 1:36 PM, Yakov Zhdanov 
> wrote:
>
> > Is current thread blocked until "await" instruction is completed in
> > parallel thread?
> >
> > --Yakov
> >
> > 2015-10-12 10:41 GMT+03:00 Vladimir Ozerov :
> >
> > > Example with Get() operation:
> > >
> > > async Task GetAndMultiply() {
> > > // This line is executed in current thread.
> > > Task res = cache.GetAsync(1);
> > >
> > > await res;
> > >
> > > // This code is executed in another thread when res is ready.
> > > int mul = res.Result * res.Result;
> > >
> > > return mul;
> > > }
> > >
> > > On Mon, Oct 12, 2015 at 10:12 AM, Dmitriy Setrakyan <
> > dsetrak...@apache.org
> > > >
> > > wrote:
> > >
> > > > On Sun, Oct 11, 2015 at 3:42 AM, Vladimir Ozerov <
> voze...@gridgain.com
> > >
> > > > wrote:
> > > >
> > > > > Guys, let's try keeping this topic focused on .Net please, because
> > the
> > > > > product is not released yet and we can create any API we like.
> > > > >
> > > > > Dima, answering your question about async/await - this is actually
> > > native
> > > > > continuation support in .Net. Consider the following .Net method:
> > > > >
> > > > > async void PutAndPrint() {
> > > > > await cache.PutAsync(1, 1);
> > > > >
> > > > > Console.WriteLine("Put value to cache.");
> > > > > }
> > > > >
> > > >
> > > > And what if the method putAsync would return a value. How would this
> > code
> > > > change?
> > > >
> > >
> >
>

Re: Which thread is used to run IgniteFuture continuations?

2015-10-12 Thread Vladimir Ozerov

It would be cool, but CompletableFuture is available since Java 8.

On Mon, Oct 12, 2015 at 7:25 PM, Dmitriy Setrakyan 
wrote:

> Would it be possible for us to support CompletableFuture from Java instead
> of adding one-off methods?
>
> On Mon, Oct 12, 2015 at 6:53 AM, Vladimir Ozerov 
> wrote:
>
> > Yakov, two points there:
> >
> > 1) This is a matter of convenience. Current mainstream solution for
> similar
> > tasks is Java's CompletableFuture. Here is what it offers:
> >
> >
> https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html#runAsync-java.lang.Runnable-
> >
> >
> https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html#runAsync-java.lang.Runnable-java.util.concurrent.Executor-
> >
> > 2) Probably we should go event further and do not execute continuations
> in
> > user thread or our system threads by default because this is error prone
> > and is subject to deadlocks and unpredictable slowdowns in normal
> > compute/cache operations. By default we can either delegate to FJP,
> provide
> > configurable pool for continuations, or use Ignite public pool.
> >
> > On Mon, Oct 12, 2015 at 4:25 PM, Yakov Zhdanov 
> > wrote:
> >
> > > Not sure if I get the point. You can do exactly the same inside your
> > > listener.
> > >
> > > final Executor executor = ...;
> > >
> > > // Bla-bla
> > >
> > > cache.future().listen(() => {
> > > executor.execute(new Runnable() {
> > > latch.await();
> > > /** Do something useful. */
> > > }
> > > });
> > >
> > > --Yakov
> > >
> > > 2015-10-09 17:22 GMT+03:00 Vladimir Ozerov :
> > >
> > > > Igniters,
> > > >
> > > > We are missing an ability to specify which thread should run
> > continuation
> > > > routine when future is completed.
> > > >
> > > > Currently we either run routine in the callee thread or in completion
> > > > thread (usually system pool thread). It means user cannot have any
> > > blocking
> > > > or cache operations inside the continuation. The code below could
> > > surprise
> > > > user with either a deadlock or system pool starvation:
> > > >
> > > > final CountDownLatch latch = new CountDownLatch();
> > > >
> > > > Cache cache = ignite.cache().withAsync();
> > > > cache.invoke(...);
> > > >
> > > > cache.future().listen(() => {
> > > > latch.await();
> > > > /** Do something useful. */
> > > > });
> > > >
> > > > /** Do something else. */
> > > > latch.countDown();
> > > >
> > > > Java 8 and Hazelcast already support custom thread pools for
> > > continuations.
> > > > E.g.:
> > > > Hazelcast.CompletableFutureTask.andThen(ExecutionCallback
> callback,
> > > > Executor executor);
> > > >
> > > > Looks like we should allow users to specify optional thread pool in
> > > futures
> > > > likewise.
> > > >
> > > > Thoughts?
> > > >
> > > > Vladimir.
> > > >
> > >
> >
>

Re: Getting rid of md5 and sha1

2015-10-12 Thread Dmitriy Setrakyan

Ivan,

I think the description of the ticket should say "get rid of sha1 in favor
of sha512". I have updated the ticket, let me know if you have objections.

D.

On Mon, Oct 12, 2015 at 8:18 AM, Ivan Veselovskiy  wrote:

> I created https://issues.apache.org/jira/browse/IGNITE-1656 on that.
>
> On Sat, Oct 10, 2015 at 12:13 AM, Konstantin Boudnik 
> wrote:
>
> > Guys,
> >
> > We had to get rid of md5 sum long time ago, but it seems that sha1 is
> > hitting
> > the wall as well. Here's the good description of the problem:
> > https://sites.google.com/site/itstheshappening/
> >
> > I'd suggest to scrape both of them in the next release. Any objections?
> >
> > Cos
> >
> >
>

Re: Which thread is used to run IgniteFuture continuations?

2015-10-12 Thread Dmitriy Setrakyan

Would it be possible for us to support CompletableFuture from Java instead
of adding one-off methods?

On Mon, Oct 12, 2015 at 6:53 AM, Vladimir Ozerov 
wrote:

> Yakov, two points there:
>
> 1) This is a matter of convenience. Current mainstream solution for similar
> tasks is Java's CompletableFuture. Here is what it offers:
>
> https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html#runAsync-java.lang.Runnable-
>
> https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html#runAsync-java.lang.Runnable-java.util.concurrent.Executor-
>
> 2) Probably we should go event further and do not execute continuations in
> user thread or our system threads by default because this is error prone
> and is subject to deadlocks and unpredictable slowdowns in normal
> compute/cache operations. By default we can either delegate to FJP, provide
> configurable pool for continuations, or use Ignite public pool.
>
> On Mon, Oct 12, 2015 at 4:25 PM, Yakov Zhdanov 
> wrote:
>
> > Not sure if I get the point. You can do exactly the same inside your
> > listener.
> >
> > final Executor executor = ...;
> >
> > // Bla-bla
> >
> > cache.future().listen(() => {
> > executor.execute(new Runnable() {
> > latch.await();
> > /** Do something useful. */
> > }
> > });
> >
> > --Yakov
> >
> > 2015-10-09 17:22 GMT+03:00 Vladimir Ozerov :
> >
> > > Igniters,
> > >
> > > We are missing an ability to specify which thread should run
> continuation
> > > routine when future is completed.
> > >
> > > Currently we either run routine in the callee thread or in completion
> > > thread (usually system pool thread). It means user cannot have any
> > blocking
> > > or cache operations inside the continuation. The code below could
> > surprise
> > > user with either a deadlock or system pool starvation:
> > >
> > > final CountDownLatch latch = new CountDownLatch();
> > >
> > > Cache cache = ignite.cache().withAsync();
> > > cache.invoke(...);
> > >
> > > cache.future().listen(() => {
> > > latch.await();
> > > /** Do something useful. */
> > > });
> > >
> > > /** Do something else. */
> > > latch.countDown();
> > >
> > > Java 8 and Hazelcast already support custom thread pools for
> > continuations.
> > > E.g.:
> > > Hazelcast.CompletableFutureTask.andThen(ExecutionCallback callback,
> > > Executor executor);
> > >
> > > Looks like we should allow users to specify optional thread pool in
> > futures
> > > likewise.
> > >
> > > Thoughts?
> > >
> > > Vladimir.
> > >
> >
>

Re: .Net: separate methods for async operations.

2015-10-12 Thread Vladimir Ozerov

Well, as I mentioned before, paired "Async()" methods are very natural to
.Net. When Microsoft creates new API it usually contains such sync/async
pairs.

I am not sure we should do the same thing in Java only because we are going
to do this in .Net. There must be other strong reasons.

On Mon, Oct 12, 2015 at 7:10 PM, Dmitriy Setrakyan 
wrote:

> Do I understand correctly that you are suggesting adding "Async(..)"
> counterparts for all the synchronous methods?
>
> Are there any objections about this? If we do it in .NET, then we might as
> well do it in Java, because in my view the same reasoning can be made for
> Java. This will cause significant proliferation of Async methods. For
> example just on IgniteCache API, we will have to add about 40 Async()
> methods.
>
> D.
>
>
>
> On Mon, Oct 12, 2015 at 3:45 AM, Vladimir Ozerov 
> wrote:
>
> > No. "await" is actually return from the method immediately. Let me show
> it
> > again:
> >
> > async Task GetAndMultiply() {
> > Task res =  cache.GetAsync(1);
> >
> > await res;
> >
> > return res.Result * res.Result;
> > }
> >
> > maps to the following pseudo-code in Java:
> >
> > Future getAndMultiply() {
> > Future res =  cache.getAsync(1);
> >
> > return res.chain((f) => {
> > return f.get() * f.get();
> > });
> > }
> >
> >
> >
> > On Mon, Oct 12, 2015 at 1:36 PM, Yakov Zhdanov 
> > wrote:
> >
> > > Is current thread blocked until "await" instruction is completed in
> > > parallel thread?
> > >
> > > --Yakov
> > >
> > > 2015-10-12 10:41 GMT+03:00 Vladimir Ozerov :
> > >
> > > > Example with Get() operation:
> > > >
> > > > async Task GetAndMultiply() {
> > > > // This line is executed in current thread.
> > > > Task res = cache.GetAsync(1);
> > > >
> > > > await res;
> > > >
> > > > // This code is executed in another thread when res is ready.
> > > > int mul = res.Result * res.Result;
> > > >
> > > > return mul;
> > > > }
> > > >
> > > > On Mon, Oct 12, 2015 at 10:12 AM, Dmitriy Setrakyan <
> > > dsetrak...@apache.org
> > > > >
> > > > wrote:
> > > >
> > > > > On Sun, Oct 11, 2015 at 3:42 AM, Vladimir Ozerov <
> > voze...@gridgain.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Guys, let's try keeping this topic focused on .Net please,
> because
> > > the
> > > > > > product is not released yet and we can create any API we like.
> > > > > >
> > > > > > Dima, answering your question about async/await - this is
> actually
> > > > native
> > > > > > continuation support in .Net. Consider the following .Net method:
> > > > > >
> > > > > > async void PutAndPrint() {
> > > > > > await cache.PutAsync(1, 1);
> > > > > >
> > > > > > Console.WriteLine("Put value to cache.");
> > > > > > }
> > > > > >
> > > > >
> > > > > And what if the method putAsync would return a value. How would
> this
> > > code
> > > > > change?
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (IGNITE-1655) Decouple Date and Timestamp types.

2015-10-12 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1655:
---

 Summary: Decouple Date and Timestamp types.
 Key: IGNITE-1655
 URL: https://issues.apache.org/jira/browse/IGNITE-1655
 Project: Ignite
  Issue Type: Task
  Components: general, interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
Priority: Critical
 Fix For: 1.5


Date and Timestamp types must be decoupled from each other:
1) New type ID must be introduced for Timestamp
2) New read/write methods should be added to portable readers, writers and 
builder.
3) Configuration parameter "isUseTimestamp" should be removed.
4) .Net DateTime type should be re-mapped to Timestamp.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1654) .Net: Ensure consistent "keepPortable" behavior.

2015-10-12 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1654:
---

 Summary: .Net: Ensure consistent "keepPortable" behavior.
 Key: IGNITE-1654
 URL: https://issues.apache.org/jira/browse/IGNITE-1654
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: 1.5


We need to ensure that "keep portable" concept is
a) Consistent;
b) Exists for all logical units where it is necessary.

Lets split "keep portable" behavior in two scenarios:
1) Client mode. This is when client (caller) operates on portable objects. 
Example: ICache.Get(...);
2) Server mode. This is when some server side object (caller) operates on 
portable objects. Example: a remote service which should not deserialize passed 
arguments and operate on portables instead.

Looks like both cases can be handled using the same "keep portable" flag on 
related logical unit.

Logical units affected:
1) Compute - at the very least closures can be executed in this mode;
2) Cache;
3) Data streamer;
4) Data structures: atomic stamped, atmoc reference;
5) Cache events;
6) Messages;
7) Services.

Probably we should introduce a kind of "IKeepPortableEnabled" (or so) 
interface, which will have a method "T WithKeepPortable()". This way users will 
be able to easily understand that "keep portable" semantics is applicable.

Lets split it into several children tickets if necessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1656) Get rid of md5 and sha1

2015-10-12 Thread Ivan Veselovsky (JIRA)

Ivan Veselovsky created IGNITE-1656:
---

 Summary: Get rid of md5 and sha1
 Key: IGNITE-1656
 URL: https://issues.apache.org/jira/browse/IGNITE-1656
 Project: Ignite
  Issue Type: Bug
  Components: cache
Affects Versions: ignite-1.4
Reporter: Ivan Veselovsky
 Fix For: 1.5


Description of the problem wrt sha1 is there:   
https://sites.google.com/site/itstheshappening/ .



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: .Net: separate methods for async operations.

2015-10-12 Thread Dmitriy Setrakyan

I think I like Sergey's idea. Any way to make it backward compatible?

On Mon, Oct 12, 2015 at 9:53 AM, Sergi Vladykin 
wrote:

> In my view we should not pollute sync APIs with all async methods,
> definitely we have to separate them
> for better UX.
>
> Currently on Java we have IgniteAsyncSupport with method withAsync() which
> returns the same sync API
> but that API works in broken manner. Instead it should look like the
> following IMO
>
> interface AsyncSupport {
> X async();
> }
>
> Where X will be an interface with respective async API.  For example for
> IngneCache we will have AsyncCache
> with all the respective async variants of all methods. Like this
>
> interface IgniteCache extends AsyncSupport> {
> V get(K key);
> }
>
>
> interface AsyncCache {
> IgniteFuture get(K key);
> }
>
> From implementation standpoint both interfaces can be implemented by the
> same class but for user API
> they will be conveniently separated. Implementation of every sync method is
> trivial if we have
> async counterpart: just call get() on received future.
>
> From documentation point of view we just have to write on each method that
> it is a async variant of some
> method on main API like following:
>
>/**
>  * Asynchronous variant of method {@link IgniteCache#get(Object)}.
>  */
>
> This way we will not need to maintain the same docs for all sync and async
> methods.
>
> Sorry, I'm not an expert in .Net but I believe this approach will fit .Net
> as well, so it can be consistent across platforms.
>
> Sergi
>
>
>
> 2015-10-12 19:10 GMT+03:00 Dmitriy Setrakyan :
>
> > Do I understand correctly that you are suggesting adding "Async(..)"
> > counterparts for all the synchronous methods?
> >
> > Are there any objections about this? If we do it in .NET, then we might
> as
> > well do it in Java, because in my view the same reasoning can be made for
> > Java. This will cause significant proliferation of Async methods. For
> > example just on IgniteCache API, we will have to add about 40 Async()
> > methods.
> >
> > D.
> >
> >
> >
> > On Mon, Oct 12, 2015 at 3:45 AM, Vladimir Ozerov 
> > wrote:
> >
> > > No. "await" is actually return from the method immediately. Let me show
> > it
> > > again:
> > >
> > > async Task GetAndMultiply() {
> > > Task res =  cache.GetAsync(1);
> > >
> > > await res;
> > >
> > > return res.Result * res.Result;
> > > }
> > >
> > > maps to the following pseudo-code in Java:
> > >
> > > Future getAndMultiply() {
> > > Future res =  cache.getAsync(1);
> > >
> > > return res.chain((f) => {
> > > return f.get() * f.get();
> > > });
> > > }
> > >
> > >
> > >
> > > On Mon, Oct 12, 2015 at 1:36 PM, Yakov Zhdanov 
> > > wrote:
> > >
> > > > Is current thread blocked until "await" instruction is completed in
> > > > parallel thread?
> > > >
> > > > --Yakov
> > > >
> > > > 2015-10-12 10:41 GMT+03:00 Vladimir Ozerov :
> > > >
> > > > > Example with Get() operation:
> > > > >
> > > > > async Task GetAndMultiply() {
> > > > > // This line is executed in current thread.
> > > > > Task res = cache.GetAsync(1);
> > > > >
> > > > > await res;
> > > > >
> > > > > // This code is executed in another thread when res is ready.
> > > > > int mul = res.Result * res.Result;
> > > > >
> > > > > return mul;
> > > > > }
> > > > >
> > > > > On Mon, Oct 12, 2015 at 10:12 AM, Dmitriy Setrakyan <
> > > > dsetrak...@apache.org
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > On Sun, Oct 11, 2015 at 3:42 AM, Vladimir Ozerov <
> > > voze...@gridgain.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Guys, let's try keeping this topic focused on .Net please,
> > because
> > > > the
> > > > > > > product is not released yet and we can create any API we like.
> > > > > > >
> > > > > > > Dima, answering your question about async/await - this is
> > actually
> > > > > native
> > > > > > > continuation support in .Net. Consider the following .Net
> method:
> > > > > > >
> > > > > > > async void PutAndPrint() {
> > > > > > > await cache.PutAsync(1, 1);
> > > > > > >
> > > > > > > Console.WriteLine("Put value to cache.");
> > > > > > > }
> > > > > > >
> > > > > >
> > > > > > And what if the method putAsync would return a value. How would
> > this
> > > > code
> > > > > > change?
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: .Net: separate methods for async operations.

2015-10-12 Thread Vladimir Ozerov

Well, if we asume that the most common usage will be
IgniteCache.async().doSomething(), then yes - backwards transformation is
not necessary.

All in all this approach seems to be the most clean among other suggested.

On Mon, Oct 12, 2015 at 8:38 PM, Sergi Vladykin 
wrote:

> Dmitriy,
>
> I mostly agree with your points except naming and hierarchy:
>
> 1. I don't like CacheAsync, it is ugly.
>
> 2. If IgniteCache extends AsyncCache then we can't use the same names for
> methods, we will be forced to use *blaAsync(...)* format
> which is ugly for me as well. Also this will pollute sync API with async
> one, while we are trying to avoid that.
>
> Sergi
>
> 2015-10-12 20:28 GMT+03:00 Dmitriy Setrakyan :
>
> > On Mon, Oct 12, 2015 at 10:15 AM, Vladimir Ozerov 
> > wrote:
> >
> >
> > > The problem with this approach is that not all methods are async. E.g.
> > > name(), lock(K), iterator(), etc.. So you should either mix sync and
> > async
> > > methods in AsyncCache still, or expose only async methods.
> >
> >
> > I think AsyncCache, or rather CacheAsync, should expose only async
> methods.
> > Moreover, should IgniteCache simply extend CacheAsync API?
> >
> >
> > > In the latter case we will require backwards
> > > transformation: IgniteCache AsyncCache.sync().
> > >
> >
> > Not sure this is needed.
> >
> >
> > > Consistency between platforms should have minimal priority. .Net and
> Java
> > > are very different. For example we cannot even have "V Get(K)" method
> in
> > > .Net cache. Instead we have "V TryGet(K, out bool)" because .Net
> supports
> > > structs and have full generics support and naive Java approach simply
> > > doesn't work here. Base concepts must be the same across platforms, but
> > > methods signatures and grouping will be different.
> > >
> >
> > I disagree here. I think consistency matters. Moreover, based on the
> > previous .NET examples you have provided, I do not see much difference
> > between .NET and Java, other than different syntax. I think the same
> > CacheAsync design can be applied to both.
> >
> >
> > >
> > >
> > >
> > > On Mon, Oct 12, 2015 at 7:53 PM, Sergi Vladykin <
> > sergi.vlady...@gmail.com>
> > > wrote:
> > >
> > > > In my view we should not pollute sync APIs with all async methods,
> > > > definitely we have to separate them
> > > > for better UX.
> > > >
> > > > Currently on Java we have IgniteAsyncSupport with method withAsync()
> > > which
> > > > returns the same sync API
> > > > but that API works in broken manner. Instead it should look like the
> > > > following IMO
> > > >
> > > > interface AsyncSupport {
> > > > X async();
> > > > }
> > > >
> > > > Where X will be an interface with respective async API.  For example
> > for
> > > > IngneCache we will have AsyncCache
> > > > with all the respective async variants of all methods. Like this
> > > >
> > > > interface IgniteCache extends AsyncSupport> {
> > > > V get(K key);
> > > > }
> > > >
> > > >
> > > > interface AsyncCache {
> > > > IgniteFuture get(K key);
> > > > }
> > > >
> > > > From implementation standpoint both interfaces can be implemented by
> > the
> > > > same class but for user API
> > > > they will be conveniently separated. Implementation of every sync
> > method
> > > is
> > > > trivial if we have
> > > > async counterpart: just call get() on received future.
> > > >
> > > > From documentation point of view we just have to write on each method
> > > that
> > > > it is a async variant of some
> > > > method on main API like following:
> > > >
> > > >/**
> > > >  * Asynchronous variant of method {@link
> IgniteCache#get(Object)}.
> > > >  */
> > > >
> > > > This way we will not need to maintain the same docs for all sync and
> > > async
> > > > methods.
> > > >
> > > > Sorry, I'm not an expert in .Net but I believe this approach will fit
> > > .Net
> > > > as well, so it can be consistent across platforms.
> > > >
> > > > Sergi
> > > >
> > > >
> > > >
> > > > 2015-10-12 19:10 GMT+03:00 Dmitriy Setrakyan  >:
> > > >
> > > > > Do I understand correctly that you are suggesting adding
> "Async(..)"
> > > > > counterparts for all the synchronous methods?
> > > > >
> > > > > Are there any objections about this? If we do it in .NET, then we
> > might
> > > > as
> > > > > well do it in Java, because in my view the same reasoning can be
> made
> > > for
> > > > > Java. This will cause significant proliferation of Async methods.
> For
> > > > > example just on IgniteCache API, we will have to add about 40
> Async()
> > > > > methods.
> > > > >
> > > > > D.
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Oct 12, 2015 at 3:45 AM, Vladimir Ozerov <
> > voze...@gridgain.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > No. "await" is actually return from the method immediately. Let
> me
> > > show
> > > > > it
> > > > > > again:
> > > > > >
> > > > > > async

Re: Which thread is used to run IgniteFuture continuations?

2015-10-12 Thread Dmitriy Setrakyan

On Mon, Oct 12, 2015 at 9:39 AM, Vladimir Ozerov 
wrote:

> It would be cool, but CompletableFuture is available since Java 8.
>

Well, we could add some methods from CompletableFuture to IgniteFuture
then. I will take a look and suggest a few.


>
> On Mon, Oct 12, 2015 at 7:25 PM, Dmitriy Setrakyan 
> wrote:
>
> > Would it be possible for us to support CompletableFuture from Java
> instead
> > of adding one-off methods?
> >
> > On Mon, Oct 12, 2015 at 6:53 AM, Vladimir Ozerov 
> > wrote:
> >
> > > Yakov, two points there:
> > >
> > > 1) This is a matter of convenience. Current mainstream solution for
> > similar
> > > tasks is Java's CompletableFuture. Here is what it offers:
> > >
> > >
> >
> https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html#runAsync-java.lang.Runnable-
> > >
> > >
> >
> https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html#runAsync-java.lang.Runnable-java.util.concurrent.Executor-
> > >
> > > 2) Probably we should go event further and do not execute continuations
> > in
> > > user thread or our system threads by default because this is error
> prone
> > > and is subject to deadlocks and unpredictable slowdowns in normal
> > > compute/cache operations. By default we can either delegate to FJP,
> > provide
> > > configurable pool for continuations, or use Ignite public pool.
> > >
> > > On Mon, Oct 12, 2015 at 4:25 PM, Yakov Zhdanov 
> > > wrote:
> > >
> > > > Not sure if I get the point. You can do exactly the same inside your
> > > > listener.
> > > >
> > > > final Executor executor = ...;
> > > >
> > > > // Bla-bla
> > > >
> > > > cache.future().listen(() => {
> > > > executor.execute(new Runnable() {
> > > > latch.await();
> > > > /** Do something useful. */
> > > > }
> > > > });
> > > >
> > > > --Yakov
> > > >
> > > > 2015-10-09 17:22 GMT+03:00 Vladimir Ozerov :
> > > >
> > > > > Igniters,
> > > > >
> > > > > We are missing an ability to specify which thread should run
> > > continuation
> > > > > routine when future is completed.
> > > > >
> > > > > Currently we either run routine in the callee thread or in
> completion
> > > > > thread (usually system pool thread). It means user cannot have any
> > > > blocking
> > > > > or cache operations inside the continuation. The code below could
> > > > surprise
> > > > > user with either a deadlock or system pool starvation:
> > > > >
> > > > > final CountDownLatch latch = new CountDownLatch();
> > > > >
> > > > > Cache cache = ignite.cache().withAsync();
> > > > > cache.invoke(...);
> > > > >
> > > > > cache.future().listen(() => {
> > > > > latch.await();
> > > > > /** Do something useful. */
> > > > > });
> > > > >
> > > > > /** Do something else. */
> > > > > latch.countDown();
> > > > >
> > > > > Java 8 and Hazelcast already support custom thread pools for
> > > > continuations.
> > > > > E.g.:
> > > > > Hazelcast.CompletableFutureTask.andThen(ExecutionCallback
> > callback,
> > > > > Executor executor);
> > > > >
> > > > > Looks like we should allow users to specify optional thread pool in
> > > > futures
> > > > > likewise.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > Vladimir.
> > > > >
> > > >
> > >
> >
>

Re: .Net: separate methods for async operations.

2015-10-12 Thread Dmitriy Setrakyan

On Mon, Oct 12, 2015 at 10:15 AM, Vladimir Ozerov 
wrote:


> The problem with this approach is that not all methods are async. E.g.
> name(), lock(K), iterator(), etc.. So you should either mix sync and async
> methods in AsyncCache still, or expose only async methods.


I think AsyncCache, or rather CacheAsync, should expose only async methods.
Moreover, should IgniteCache simply extend CacheAsync API?


> In the latter case we will require backwards
> transformation: IgniteCache AsyncCache.sync().
>

Not sure this is needed.


> Consistency between platforms should have minimal priority. .Net and Java
> are very different. For example we cannot even have "V Get(K)" method in
> .Net cache. Instead we have "V TryGet(K, out bool)" because .Net supports
> structs and have full generics support and naive Java approach simply
> doesn't work here. Base concepts must be the same across platforms, but
> methods signatures and grouping will be different.
>

I disagree here. I think consistency matters. Moreover, based on the
previous .NET examples you have provided, I do not see much difference
between .NET and Java, other than different syntax. I think the same
CacheAsync design can be applied to both.


>
>
>
> On Mon, Oct 12, 2015 at 7:53 PM, Sergi Vladykin 
> wrote:
>
> > In my view we should not pollute sync APIs with all async methods,
> > definitely we have to separate them
> > for better UX.
> >
> > Currently on Java we have IgniteAsyncSupport with method withAsync()
> which
> > returns the same sync API
> > but that API works in broken manner. Instead it should look like the
> > following IMO
> >
> > interface AsyncSupport {
> > X async();
> > }
> >
> > Where X will be an interface with respective async API.  For example for
> > IngneCache we will have AsyncCache
> > with all the respective async variants of all methods. Like this
> >
> > interface IgniteCache extends AsyncSupport> {
> > V get(K key);
> > }
> >
> >
> > interface AsyncCache {
> > IgniteFuture get(K key);
> > }
> >
> > From implementation standpoint both interfaces can be implemented by the
> > same class but for user API
> > they will be conveniently separated. Implementation of every sync method
> is
> > trivial if we have
> > async counterpart: just call get() on received future.
> >
> > From documentation point of view we just have to write on each method
> that
> > it is a async variant of some
> > method on main API like following:
> >
> >/**
> >  * Asynchronous variant of method {@link IgniteCache#get(Object)}.
> >  */
> >
> > This way we will not need to maintain the same docs for all sync and
> async
> > methods.
> >
> > Sorry, I'm not an expert in .Net but I believe this approach will fit
> .Net
> > as well, so it can be consistent across platforms.
> >
> > Sergi
> >
> >
> >
> > 2015-10-12 19:10 GMT+03:00 Dmitriy Setrakyan :
> >
> > > Do I understand correctly that you are suggesting adding "Async(..)"
> > > counterparts for all the synchronous methods?
> > >
> > > Are there any objections about this? If we do it in .NET, then we might
> > as
> > > well do it in Java, because in my view the same reasoning can be made
> for
> > > Java. This will cause significant proliferation of Async methods. For
> > > example just on IgniteCache API, we will have to add about 40 Async()
> > > methods.
> > >
> > > D.
> > >
> > >
> > >
> > > On Mon, Oct 12, 2015 at 3:45 AM, Vladimir Ozerov  >
> > > wrote:
> > >
> > > > No. "await" is actually return from the method immediately. Let me
> show
> > > it
> > > > again:
> > > >
> > > > async Task GetAndMultiply() {
> > > > Task res =  cache.GetAsync(1);
> > > >
> > > > await res;
> > > >
> > > > return res.Result * res.Result;
> > > > }
> > > >
> > > > maps to the following pseudo-code in Java:
> > > >
> > > > Future getAndMultiply() {
> > > > Future res =  cache.getAsync(1);
> > > >
> > > > return res.chain((f) => {
> > > > return f.get() * f.get();
> > > > });
> > > > }
> > > >
> > > >
> > > >
> > > > On Mon, Oct 12, 2015 at 1:36 PM, Yakov Zhdanov 
> > > > wrote:
> > > >
> > > > > Is current thread blocked until "await" instruction is completed in
> > > > > parallel thread?
> > > > >
> > > > > --Yakov
> > > > >
> > > > > 2015-10-12 10:41 GMT+03:00 Vladimir Ozerov :
> > > > >
> > > > > > Example with Get() operation:
> > > > > >
> > > > > > async Task GetAndMultiply() {
> > > > > > // This line is executed in current thread.
> > > > > > Task res = cache.GetAsync(1);
> > > > > >
> > > > > > await res;
> > > > > >
> > > > > > // This code is executed in another thread when res is ready.
> > > > > > int mul = res.Result * res.Result;
> > > > > >
> > > > > > return mul;
> > > > > > }
> > > > > >
> > > > > > On Mon, Oct 12, 2015 at 10:12 AM, Dmitriy Setrakyan <
> >

Re: .Net: separate methods for async operations.

2015-10-12 Thread Sergi Vladykin

Dmitriy,

I mostly agree with your points except naming and hierarchy:

1. I don't like CacheAsync, it is ugly.

2. If IgniteCache extends AsyncCache then we can't use the same names for
methods, we will be forced to use *blaAsync(...)* format
which is ugly for me as well. Also this will pollute sync API with async
one, while we are trying to avoid that.

Sergi

2015-10-12 20:28 GMT+03:00 Dmitriy Setrakyan :

> On Mon, Oct 12, 2015 at 10:15 AM, Vladimir Ozerov 
> wrote:
>
>
> > The problem with this approach is that not all methods are async. E.g.
> > name(), lock(K), iterator(), etc.. So you should either mix sync and
> async
> > methods in AsyncCache still, or expose only async methods.
>
>
> I think AsyncCache, or rather CacheAsync, should expose only async methods.
> Moreover, should IgniteCache simply extend CacheAsync API?
>
>
> > In the latter case we will require backwards
> > transformation: IgniteCache AsyncCache.sync().
> >
>
> Not sure this is needed.
>
>
> > Consistency between platforms should have minimal priority. .Net and Java
> > are very different. For example we cannot even have "V Get(K)" method in
> > .Net cache. Instead we have "V TryGet(K, out bool)" because .Net supports
> > structs and have full generics support and naive Java approach simply
> > doesn't work here. Base concepts must be the same across platforms, but
> > methods signatures and grouping will be different.
> >
>
> I disagree here. I think consistency matters. Moreover, based on the
> previous .NET examples you have provided, I do not see much difference
> between .NET and Java, other than different syntax. I think the same
> CacheAsync design can be applied to both.
>
>
> >
> >
> >
> > On Mon, Oct 12, 2015 at 7:53 PM, Sergi Vladykin <
> sergi.vlady...@gmail.com>
> > wrote:
> >
> > > In my view we should not pollute sync APIs with all async methods,
> > > definitely we have to separate them
> > > for better UX.
> > >
> > > Currently on Java we have IgniteAsyncSupport with method withAsync()
> > which
> > > returns the same sync API
> > > but that API works in broken manner. Instead it should look like the
> > > following IMO
> > >
> > > interface AsyncSupport {
> > > X async();
> > > }
> > >
> > > Where X will be an interface with respective async API.  For example
> for
> > > IngneCache we will have AsyncCache
> > > with all the respective async variants of all methods. Like this
> > >
> > > interface IgniteCache extends AsyncSupport> {
> > > V get(K key);
> > > }
> > >
> > >
> > > interface AsyncCache {
> > > IgniteFuture get(K key);
> > > }
> > >
> > > From implementation standpoint both interfaces can be implemented by
> the
> > > same class but for user API
> > > they will be conveniently separated. Implementation of every sync
> method
> > is
> > > trivial if we have
> > > async counterpart: just call get() on received future.
> > >
> > > From documentation point of view we just have to write on each method
> > that
> > > it is a async variant of some
> > > method on main API like following:
> > >
> > >/**
> > >  * Asynchronous variant of method {@link IgniteCache#get(Object)}.
> > >  */
> > >
> > > This way we will not need to maintain the same docs for all sync and
> > async
> > > methods.
> > >
> > > Sorry, I'm not an expert in .Net but I believe this approach will fit
> > .Net
> > > as well, so it can be consistent across platforms.
> > >
> > > Sergi
> > >
> > >
> > >
> > > 2015-10-12 19:10 GMT+03:00 Dmitriy Setrakyan :
> > >
> > > > Do I understand correctly that you are suggesting adding "Async(..)"
> > > > counterparts for all the synchronous methods?
> > > >
> > > > Are there any objections about this? If we do it in .NET, then we
> might
> > > as
> > > > well do it in Java, because in my view the same reasoning can be made
> > for
> > > > Java. This will cause significant proliferation of Async methods. For
> > > > example just on IgniteCache API, we will have to add about 40 Async()
> > > > methods.
> > > >
> > > > D.
> > > >
> > > >
> > > >
> > > > On Mon, Oct 12, 2015 at 3:45 AM, Vladimir Ozerov <
> voze...@gridgain.com
> > >
> > > > wrote:
> > > >
> > > > > No. "await" is actually return from the method immediately. Let me
> > show
> > > > it
> > > > > again:
> > > > >
> > > > > async Task GetAndMultiply() {
> > > > > Task res =  cache.GetAsync(1);
> > > > >
> > > > > await res;
> > > > >
> > > > > return res.Result * res.Result;
> > > > > }
> > > > >
> > > > > maps to the following pseudo-code in Java:
> > > > >
> > > > > Future getAndMultiply() {
> > > > > Future res =  cache.getAsync(1);
> > > > >
> > > > > return res.chain((f) => {
> > > > > return f.get() * f.get();
> > > > > });
> > > > > }
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Oct 12, 2015 at 1:36 PM, Yakov Zhdanov <
> yzhda...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Is

Re: .Net: separate methods for async operations.

2015-10-12 Thread Sergi Vladykin

In my view we should not pollute sync APIs with all async methods,
definitely we have to separate them
for better UX.

Currently on Java we have IgniteAsyncSupport with method withAsync() which
returns the same sync API
but that API works in broken manner. Instead it should look like the
following IMO

interface AsyncSupport {
X async();
}

Where X will be an interface with respective async API.  For example for
IngneCache we will have AsyncCache
with all the respective async variants of all methods. Like this

interface IgniteCache extends AsyncSupport> {
V get(K key);
}


interface AsyncCache {
IgniteFuture get(K key);
}

>From implementation standpoint both interfaces can be implemented by the
same class but for user API
they will be conveniently separated. Implementation of every sync method is
trivial if we have
async counterpart: just call get() on received future.

>From documentation point of view we just have to write on each method that
it is a async variant of some
method on main API like following:

   /**
 * Asynchronous variant of method {@link IgniteCache#get(Object)}.
 */

This way we will not need to maintain the same docs for all sync and async
methods.

Sorry, I'm not an expert in .Net but I believe this approach will fit .Net
as well, so it can be consistent across platforms.

Sergi



2015-10-12 19:10 GMT+03:00 Dmitriy Setrakyan :

> Do I understand correctly that you are suggesting adding "Async(..)"
> counterparts for all the synchronous methods?
>
> Are there any objections about this? If we do it in .NET, then we might as
> well do it in Java, because in my view the same reasoning can be made for
> Java. This will cause significant proliferation of Async methods. For
> example just on IgniteCache API, we will have to add about 40 Async()
> methods.
>
> D.
>
>
>
> On Mon, Oct 12, 2015 at 3:45 AM, Vladimir Ozerov 
> wrote:
>
> > No. "await" is actually return from the method immediately. Let me show
> it
> > again:
> >
> > async Task GetAndMultiply() {
> > Task res =  cache.GetAsync(1);
> >
> > await res;
> >
> > return res.Result * res.Result;
> > }
> >
> > maps to the following pseudo-code in Java:
> >
> > Future getAndMultiply() {
> > Future res =  cache.getAsync(1);
> >
> > return res.chain((f) => {
> > return f.get() * f.get();
> > });
> > }
> >
> >
> >
> > On Mon, Oct 12, 2015 at 1:36 PM, Yakov Zhdanov 
> > wrote:
> >
> > > Is current thread blocked until "await" instruction is completed in
> > > parallel thread?
> > >
> > > --Yakov
> > >
> > > 2015-10-12 10:41 GMT+03:00 Vladimir Ozerov :
> > >
> > > > Example with Get() operation:
> > > >
> > > > async Task GetAndMultiply() {
> > > > // This line is executed in current thread.
> > > > Task res = cache.GetAsync(1);
> > > >
> > > > await res;
> > > >
> > > > // This code is executed in another thread when res is ready.
> > > > int mul = res.Result * res.Result;
> > > >
> > > > return mul;
> > > > }
> > > >
> > > > On Mon, Oct 12, 2015 at 10:12 AM, Dmitriy Setrakyan <
> > > dsetrak...@apache.org
> > > > >
> > > > wrote:
> > > >
> > > > > On Sun, Oct 11, 2015 at 3:42 AM, Vladimir Ozerov <
> > voze...@gridgain.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Guys, let's try keeping this topic focused on .Net please,
> because
> > > the
> > > > > > product is not released yet and we can create any API we like.
> > > > > >
> > > > > > Dima, answering your question about async/await - this is
> actually
> > > > native
> > > > > > continuation support in .Net. Consider the following .Net method:
> > > > > >
> > > > > > async void PutAndPrint() {
> > > > > > await cache.PutAsync(1, 1);
> > > > > >
> > > > > > Console.WriteLine("Put value to cache.");
> > > > > > }
> > > > > >
> > > > >
> > > > > And what if the method putAsync would return a value. How would
> this
> > > code
> > > > > change?
> > > > >
> > > >
> > >
> >
>

Distributed SQL joins affinity key configuration

2015-10-12 Thread Sergi Vladykin

Guys,

I'm about to finish distributed SQL joins feature for Ignite and we need to
decide
what to do with configuration because currently I don't see how to change
it and
not to brake backward compatibility.

Currently we declare SQL schema for cache using indexedTypes with
annotations
or CacheTypeMetadata. Now we will need to specify affinity key field name
for every type.
By default if it is not specified we will assume that it is a *_KEY* field
which is actual
cache key at SQL layer.

I know about two ways to specify affinity keys in IgniteCache:

1. Using @AffinityKeyMapped to annotate fields or methods inside of cache
key class.

We could use it in SQL as is but there is a possible problem: if user did
not specify
@QuerySqlField annotation for that field. It may be was done intentionally
or
by mistake. The code which should start a distributed join for this case
must
throw exception as I understand.

2. Using custom AffinityKeyMapper which allows to extract affinity key from
key.

With it we can extract affinity key from cache key, but for SQL we need a
affinity field name. Probably we need additional method affinityKeyName() or
something like that but it will brake backward compatibility.

Thoughts?

Sergi

Re: .Net: separate methods for async operations.

2015-10-12 Thread Nikita Ivanov

+1 on Sergey's idea too.

--
Nikita Ivanov


On Mon, Oct 12, 2015 at 10:10 AM, Dmitriy Setrakyan 
wrote:

> I think I like Sergey's idea. Any way to make it backward compatible?
>
> On Mon, Oct 12, 2015 at 9:53 AM, Sergi Vladykin 
> wrote:
>
> > In my view we should not pollute sync APIs with all async methods,
> > definitely we have to separate them
> > for better UX.
> >
> > Currently on Java we have IgniteAsyncSupport with method withAsync()
> which
> > returns the same sync API
> > but that API works in broken manner. Instead it should look like the
> > following IMO
> >
> > interface AsyncSupport {
> > X async();
> > }
> >
> > Where X will be an interface with respective async API.  For example for
> > IngneCache we will have AsyncCache
> > with all the respective async variants of all methods. Like this
> >
> > interface IgniteCache extends AsyncSupport> {
> > V get(K key);
> > }
> >
> >
> > interface AsyncCache {
> > IgniteFuture get(K key);
> > }
> >
> > From implementation standpoint both interfaces can be implemented by the
> > same class but for user API
> > they will be conveniently separated. Implementation of every sync method
> is
> > trivial if we have
> > async counterpart: just call get() on received future.
> >
> > From documentation point of view we just have to write on each method
> that
> > it is a async variant of some
> > method on main API like following:
> >
> >/**
> >  * Asynchronous variant of method {@link IgniteCache#get(Object)}.
> >  */
> >
> > This way we will not need to maintain the same docs for all sync and
> async
> > methods.
> >
> > Sorry, I'm not an expert in .Net but I believe this approach will fit
> .Net
> > as well, so it can be consistent across platforms.
> >
> > Sergi
> >
> >
> >
> > 2015-10-12 19:10 GMT+03:00 Dmitriy Setrakyan :
> >
> > > Do I understand correctly that you are suggesting adding "Async(..)"
> > > counterparts for all the synchronous methods?
> > >
> > > Are there any objections about this? If we do it in .NET, then we might
> > as
> > > well do it in Java, because in my view the same reasoning can be made
> for
> > > Java. This will cause significant proliferation of Async methods. For
> > > example just on IgniteCache API, we will have to add about 40 Async()
> > > methods.
> > >
> > > D.
> > >
> > >
> > >
> > > On Mon, Oct 12, 2015 at 3:45 AM, Vladimir Ozerov  >
> > > wrote:
> > >
> > > > No. "await" is actually return from the method immediately. Let me
> show
> > > it
> > > > again:
> > > >
> > > > async Task GetAndMultiply() {
> > > > Task res =  cache.GetAsync(1);
> > > >
> > > > await res;
> > > >
> > > > return res.Result * res.Result;
> > > > }
> > > >
> > > > maps to the following pseudo-code in Java:
> > > >
> > > > Future getAndMultiply() {
> > > > Future res =  cache.getAsync(1);
> > > >
> > > > return res.chain((f) => {
> > > > return f.get() * f.get();
> > > > });
> > > > }
> > > >
> > > >
> > > >
> > > > On Mon, Oct 12, 2015 at 1:36 PM, Yakov Zhdanov 
> > > > wrote:
> > > >
> > > > > Is current thread blocked until "await" instruction is completed in
> > > > > parallel thread?
> > > > >
> > > > > --Yakov
> > > > >
> > > > > 2015-10-12 10:41 GMT+03:00 Vladimir Ozerov :
> > > > >
> > > > > > Example with Get() operation:
> > > > > >
> > > > > > async Task GetAndMultiply() {
> > > > > > // This line is executed in current thread.
> > > > > > Task res = cache.GetAsync(1);
> > > > > >
> > > > > > await res;
> > > > > >
> > > > > > // This code is executed in another thread when res is ready.
> > > > > > int mul = res.Result * res.Result;
> > > > > >
> > > > > > return mul;
> > > > > > }
> > > > > >
> > > > > > On Mon, Oct 12, 2015 at 10:12 AM, Dmitriy Setrakyan <
> > > > > dsetrak...@apache.org
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > On Sun, Oct 11, 2015 at 3:42 AM, Vladimir Ozerov <
> > > > voze...@gridgain.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Guys, let's try keeping this topic focused on .Net please,
> > > because
> > > > > the
> > > > > > > > product is not released yet and we can create any API we
> like.
> > > > > > > >
> > > > > > > > Dima, answering your question about async/await - this is
> > > actually
> > > > > > native
> > > > > > > > continuation support in .Net. Consider the following .Net
> > method:
> > > > > > > >
> > > > > > > > async void PutAndPrint() {
> > > > > > > > await cache.PutAsync(1, 1);
> > > > > > > >
> > > > > > > > Console.WriteLine("Put value to cache.");
> > > > > > > > }
> > > > > > > >
> > > > > > >
> > > > > > > And what if the method putAsync would return a value. How would
> > > this
> > > > > code
> > > > > > > change?
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Distributed SQL joins affinity key configuration

2015-10-12 Thread Dmitriy Setrakyan

I thought that there is a way to specify affinity keys in XML configuration
as well. Am I wrong?

On Mon, Oct 12, 2015 at 1:44 PM, Sergi Vladykin 
wrote:

> Guys,
>
> I'm about to finish distributed SQL joins feature for Ignite and we need to
> decide
> what to do with configuration because currently I don't see how to change
> it and
> not to brake backward compatibility.
>
> Currently we declare SQL schema for cache using indexedTypes with
> annotations
> or CacheTypeMetadata. Now we will need to specify affinity key field name
> for every type.
> By default if it is not specified we will assume that it is a *_KEY* field
> which is actual
> cache key at SQL layer.
>
> I know about two ways to specify affinity keys in IgniteCache:
>
> 1. Using @AffinityKeyMapped to annotate fields or methods inside of cache
> key class.
>
> We could use it in SQL as is but there is a possible problem: if user did
> not specify
> @QuerySqlField annotation for that field. It may be was done intentionally
> or
> by mistake. The code which should start a distributed join for this case
> must
> throw exception as I understand.
>
> 2. Using custom AffinityKeyMapper which allows to extract affinity key from
> key.
>
> With it we can extract affinity key from cache key, but for SQL we need a
> affinity field name. Probably we need additional method affinityKeyName()
> or
> something like that but it will brake backward compatibility.
>
> Thoughts?
>
> Sergi
>

[jira] [Created] (IGNITE-1659) Add query history

2015-10-12 Thread Pavel Konstantinov (JIRA)

Pavel Konstantinov created IGNITE-1659:
--

 Summary: Add query history
 Key: IGNITE-1659
 URL: https://issues.apache.org/jira/browse/IGNITE-1659
 Project: Ignite
  Issue Type: Sub-task
Reporter: Pavel Konstantinov
Assignee: Vasiliy Sisko


We need to implement saving query history and restoring query text into any 
Query field.

Note: currently we have workaround - we save notebooks with all queries in it, 
but if user remove notebook he lost all queries from this notebook.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: .Net: separate methods for async operations.

2015-10-12 Thread Vladimir Ozerov

Dima,

I do not like JSR 107 v1.1 approach. First, it is not user friendly.
Second, it is prone to deadlock/starvation problems we are currently
discussing in another thread because these "Collectors" are continuations
essentially.

On Tue, Oct 13, 2015 at 12:42 AM, Dmitriy Setrakyan 
wrote:

> I just took a look at JSR 107, and looks like they are taking a different
> (in my view not as elegant) route:
>
> https://github.com/jsr107/jsr107spec/blob/async/src/main/java/javax/cache/Cache.java
>
> Any thoughts on this?
>
> D.
>
> On Mon, Oct 12, 2015 at 11:11 AM, Vladimir Ozerov 
> wrote:
>
> > Well, if we asume that the most common usage will be
> > IgniteCache.async().doSomething(), then yes - backwards transformation is
> > not necessary.
> >
> > All in all this approach seems to be the most clean among other
> suggested.
> >
> > On Mon, Oct 12, 2015 at 8:38 PM, Sergi Vladykin <
> sergi.vlady...@gmail.com>
> > wrote:
> >
> > > Dmitriy,
> > >
> > > I mostly agree with your points except naming and hierarchy:
> > >
> > > 1. I don't like CacheAsync, it is ugly.
> > >
> > > 2. If IgniteCache extends AsyncCache then we can't use the same names
> for
> > > methods, we will be forced to use *blaAsync(...)* format
> > > which is ugly for me as well. Also this will pollute sync API with
> async
> > > one, while we are trying to avoid that.
> > >
> > > Sergi
> > >
> > > 2015-10-12 20:28 GMT+03:00 Dmitriy Setrakyan :
> > >
> > > > On Mon, Oct 12, 2015 at 10:15 AM, Vladimir Ozerov <
> > voze...@gridgain.com>
> > > > wrote:
> > > >
> > > >
> > > > > The problem with this approach is that not all methods are async.
> > E.g.
> > > > > name(), lock(K), iterator(), etc.. So you should either mix sync
> and
> > > > async
> > > > > methods in AsyncCache still, or expose only async methods.
> > > >
> > > >
> > > > I think AsyncCache, or rather CacheAsync, should expose only async
> > > methods.
> > > > Moreover, should IgniteCache simply extend CacheAsync API?
> > > >
> > > >
> > > > > In the latter case we will require backwards
> > > > > transformation: IgniteCache AsyncCache.sync().
> > > > >
> > > >
> > > > Not sure this is needed.
> > > >
> > > >
> > > > > Consistency between platforms should have minimal priority. .Net
> and
> > > Java
> > > > > are very different. For example we cannot even have "V Get(K)"
> method
> > > in
> > > > > .Net cache. Instead we have "V TryGet(K, out bool)" because .Net
> > > supports
> > > > > structs and have full generics support and naive Java approach
> simply
> > > > > doesn't work here. Base concepts must be the same across platforms,
> > but
> > > > > methods signatures and grouping will be different.
> > > > >
> > > >
> > > > I disagree here. I think consistency matters. Moreover, based on the
> > > > previous .NET examples you have provided, I do not see much
> difference
> > > > between .NET and Java, other than different syntax. I think the same
> > > > CacheAsync design can be applied to both.
> > > >
> > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Oct 12, 2015 at 7:53 PM, Sergi Vladykin <
> > > > sergi.vlady...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > In my view we should not pollute sync APIs with all async
> methods,
> > > > > > definitely we have to separate them
> > > > > > for better UX.
> > > > > >
> > > > > > Currently on Java we have IgniteAsyncSupport with method
> > withAsync()
> > > > > which
> > > > > > returns the same sync API
> > > > > > but that API works in broken manner. Instead it should look like
> > the
> > > > > > following IMO
> > > > > >
> > > > > > interface AsyncSupport {
> > > > > > X async();
> > > > > > }
> > > > > >
> > > > > > Where X will be an interface with respective async API.  For
> > example
> > > > for
> > > > > > IngneCache we will have AsyncCache
> > > > > > with all the respective async variants of all methods. Like this
> > > > > >
> > > > > > interface IgniteCache extends AsyncSupport>
> {
> > > > > > V get(K key);
> > > > > > }
> > > > > >
> > > > > >
> > > > > > interface AsyncCache {
> > > > > > IgniteFuture get(K key);
> > > > > > }
> > > > > >
> > > > > > From implementation standpoint both interfaces can be implemented
> > by
> > > > the
> > > > > > same class but for user API
> > > > > > they will be conveniently separated. Implementation of every sync
> > > > method
> > > > > is
> > > > > > trivial if we have
> > > > > > async counterpart: just call get() on received future.
> > > > > >
> > > > > > From documentation point of view we just have to write on each
> > method
> > > > > that
> > > > > > it is a async variant of some
> > > > > > method on main API like following:
> > > > > >
> > > > > >/**
> > > > > >  * Asynchronous variant of method {@link
> > > IgniteCache#get(Object)}.
> > > > > >  */
> > > > > >
> > > > > > This way we will not need to maintain the same docs for all sync
> > and

Re: IGFS concurrency issue

2015-10-12 Thread Konstantin Boudnik

Thanks Vladimir.  In the case of HDFS, the global lock for conflicting
simultaneous changes ie

mkdir /a/b/c 
together with 
mv /a/b /a/d

will trigger the NN global lock on the file system so the latter operation
won't start until the former is completed. Hence, the point of my confusion is
why IGFS is adding an additional lock on top of what the secondary file system
does anyway. Sorry, if I haven't expressed myself in a better way from the
beginning.

Cos

On Thu, Oct 08, 2015 at 11:33AM, Vladimir Ozerov wrote:
> Cos,
> 
> Initially IGFS was designed to support concurrent structural updates. E.g.,
> creation of directories /A/C/D and /E/F/G can be performed concurrently.
> Now we revealed that it might cause concurrency issues in case of
> conflicting updates.
> 
> To fix this problem we now perform every update holding a kind of global
> file system lock. This doesn't affect file read/write operations - they
> still can be performed concurrently of course. The question was whether
> this could cause severe performance degradation or not. If we assume that
> in average Hadoop jobs file operations dominate over structural operations
> on directories, then it should not cause any significant performance issues.
> 
> Vladimir.
> 
> On Wed, Oct 7, 2015 at 10:38 PM, Konstantin Boudnik  wrote:
> 
> > On Wed, Oct 07, 2015 at 09:11AM, Vladimir Ozerov wrote:
> > > Cos,
> > > Yes, no long-time locking is expected here.
> >
> > Sorry, I musta be still dense from the jet-lag: could you put in a bit more
> > details? Thanks in advance!
> >
> > Cos
> >
> > > On Wed, Oct 7, 2015 at 6:57 AM, Konstantin Boudnik 
> > wrote:
> > >
> > > > IIRC NN should be locking on these ops anyway, shouldn't it? The
> > situation
> > > > is
> > > > no different if multiple clients are doing these operations
> > > > near-simultaneously. Unless I missed something here...
> > > >
> > > > On Thu, Sep 24, 2015 at 11:28AM, Sergi Vladykin wrote:
> > > > > May be just check that they are not parent-child within the tx?
> > > > >
> > > > > Sergi
> > > > > Igniters,
> > > > >
> > > > > We revealed concurrency problem in IGFS and I would like to discuss
> > > > > possible solutions to it.
> > > > >
> > > > > Consider the following file system structure:
> > > > > root
> > > > > |-- A
> > > > > |   |-- B
> > > > > |   |   |-- C
> > > > > |   |-- D
> > > > >
> > > > > ... two concurrent operations in different threads:
> > > > > T1: move(/A/B, /A/D);
> > > > > T2: move(/A/D, /A/B/C);
> > > > >
> > > > > ... and how IGFS handles it now:
> > > > > T1: verify that "/A/B" and "/A/D" exist, they are not child-parent to
> > > > each
> > > > > other, etc. -> OK.
> > > > > T2: do the same for "A/D" and "A/B/C" -> OK.
> > > > > T1: get IDs of "/A", "/A/B" and "/A/D" to lock them later inside tx.
> > > > > T2: get IDs of "/A", "/A/D", "/A/B" and "/A/B/C" to lock them later
> > > > inside
> > > > > tx.
> > > > >
> > > > > T1: Start pessimistic tx, lock IDs of "/A", "/A/B", "/A/D", perform
> > move
> > > > ->
> > > > > OK.
> > > > > root
> > > > > |-- A
> > > > > |   |-- D
> > > > > |   |   |-- B
> > > > > |   |   |   |-- C
> > > > >
> > > > > T2: Start pessimistic tx, lock IDs of "/A", "/A/D", "/A/B" and
> > > > > "/A/B/C" (*directory
> > > > > structure already changed at this time!*), perform move -> OK.
> > > > > root
> > > > > |-- A
> > > > > B
> > > > > |-- D
> > > > > |   |-- C
> > > > > |   |   |-- B (loop!)
> > > > >
> > > > > File system is corrupted. Folders B, C and D are not reacheable from
> > > > root.
> > > > >
> > > > > To fix this now we additionaly check if directory structure is still
> > > > > valid *inside
> > > > > transaction*. It works, no more corruptions. But it requres taking
> > locks
> > > > on
> > > > > the whole paths *including root*. So move, delete and mkdirs
> > opeartions
> > > > *can
> > > > > no longer be concurrent*.
> > > > >
> > > > > Probably there is a way to relax this while still ensuring
> > consistency,
> > > > but
> > > > > I do not see how. One idea is to store real path inside each entry.
> > This
> > > > > way we will be able to ensure that it is still at a valid location
> > > > without
> > > > > blocking parents, so concurrnecy will be restored. But we will have
> > to
> > > > > propagate strucutral changes to children. E.g. move of a folder with
> > 100
> > > > > items will lead to update of >100 cache entries. Not so good.
> > > > >
> > > > > Any other ideas?
> > > > >
> > > > > Vladimir.
> > > >
> >


signature.asc
Description: Digital signature

[GitHub] ignite pull request: IGNITE-1617: Strings usage adjusted

[jira] [Created] (IGNITE-1653) Need to remove lgpl-dependent examples from Ignite binaries

[jira] [Created] (IGNITE-1650) Add ability to specify thread pool for IgniteFuture listen/chain methods.

[jira] [Created] (IGNITE-1651) Add version to portable object header.

Re: .Net: separate methods for async operations.

Re: .Net: separate methods for async operations.

Re: Which thread is used to run IgniteFuture continuations?

Re: Getting rid of md5 and sha1

Re: Which thread is used to run IgniteFuture continuations?

Re: .Net: separate methods for async operations.

[jira] [Created] (IGNITE-1655) Decouple Date and Timestamp types.

[jira] [Created] (IGNITE-1654) .Net: Ensure consistent "keepPortable" behavior.

[jira] [Created] (IGNITE-1656) Get rid of md5 and sha1

Re: .Net: separate methods for async operations.

Re: .Net: separate methods for async operations.

Re: Which thread is used to run IgniteFuture continuations?

Re: .Net: separate methods for async operations.

Re: .Net: separate methods for async operations.

Re: .Net: separate methods for async operations.

Distributed SQL joins affinity key configuration

Re: .Net: separate methods for async operations.

Re: Distributed SQL joins affinity key configuration

[jira] [Created] (IGNITE-1659) Add query history

Re: .Net: separate methods for async operations.

Re: IGFS concurrency issue

25 matches

Site Navigation

Mail list logo

Footer information