Re: Micronaut/Ignite

2020-08-18 Thread Denis Magda 
>
> oh, so we probably don't need to work with multiple instances. This is what
> I have in the current master branch.


In most cases, people start a single instance of a thick or thin client per
application. The clients are multi-threaded and can utilize all the CPUs
effectively. However, it's not harmful to have the ability to configure
several clients per application. As far as I understand, Micronaut
distinguishes clients per the "IgniteClientConfiguration.name" property,
right?

So what defaults are set for IgniteConfiguration?


Does it matter to Micronaut what those defaults are? By looking at the
IgniteThinClientConfiguration
,
that defines org.apache.ignite.configuration.ClientConfiguration property
(under the name of "configuration"), I see that Micronaut could introspect
all the fields of the ClientConfiguration and prepared these properties
table
.
For me, it means that whenever I am configuring the thin client in a YAML
file, Micronaut will create an instance of the ClientConfiguration (Ignite
sets the defaults), and then I can override some settings such as
"addresses" or "enablePartitionAwareness". Does this sound accurate
concerning Micronaut?

Jumping back to the IgniteConfiguration, I would just swap the "path" that
is the String with the "config" that is IgniteConfiguration. Then let
Ignite take care of the IgniteConfiguration defaults and allow a developer
to override some defaults (such as discoverySPI.ipFinder or memory
settings). Just in case, you can find IgniteConfiguration defaults here

.

-
Denis


On Tue, Aug 18, 2020 at 8:59 PM Michael Pollind  wrote:

> oh, so we probably don't need to work with multiple instances. This is what
> I have in the current master branch. I believe I was originally trying to
> set-up the configuration with the default ignite instance but found I
> couldn't cover enough of the configuration. So what defaults are set for
> IgniteConfiguration? some of those factory items can't be covered with how
> micronaut sets up configurations. @ConfigurationProperty can only be
> defined on a known factory, there are ways to have multiple factories and
> label them optional but that easily gets overwhelming.  In this situation
> providing your own bean would probably be more ideal in this situation when
> I think about it.  I was worrying that I wouldn't be able to cover enough
> of the configuration with
>
> ignite:  enabled: true  thin-clients:default:  address:  -
> "127.0.0.1:10800"  - "127.0.0.1:10801"
>
> thin-client-2:
>   address:  - "127.0.0.1:10800"  - "127.0.0.1:10801"
>
>
> you can see it in the current snapshot documentation:
> https://micronaut-projects.github.io/micronaut-ignite/snapshot/guide/
>
> On Tue, Aug 18, 2020 at 4:16 PM Denis Magda  wrote:
>
> > Michael, thanks for filling out the wiki page.
> >
> > I'm looking at the Auto-Configuration wiki section and the current
> version
> > of the io.micronaut.ignite.configuration.IgniteClientConfiguration
> > <
> >
> https://github.com/micronaut-projects/micronaut-ignite/blob/master/ignite-core/src/main/java/io/micronaut/ignite/configuration/IgniteClientConfiguration.java
> > >
> > class,
> > and wonder if we can perform the following changes:
> >
> >1. Rename the IgniteClientConfiguration to IgniteConfiguration (or, to
> >avoid ambiguity, even to DefaultIgniteConfiguration as it's done for
> the
> >Mongo driver
> ><
> >
> https://micronaut-projects.github.io/micronaut-mongodb/latest/api/io/micronaut/configuration/mongo/reactive/DefaultMongoConfiguration.html
> > >).
> >The rationale for this change is that the developers might need to
> > start an embedded
> >Ignite server node
> ><
> >
> https://www.gridgain.com/docs/latest/installation-guide/deployment-modes#embedded-deployment
> > >.
> >So, I would not limit the integration scope to the Ignite clients
> only.
> >2. Presently,
> > io.micronaut.ignite.configuration.IgniteClientConfiguration
> >supports two parameters - the "name" and "path". I would replace the
> > "path"
> >parameter with the "config" one that instantiates
> >org.apache.ignite.IgniteConfiguration. If we do that, then the
> > developers
> >will be able to set any property of the IgniteConfiguration straight
> in
> > the
> >main YAML file. See how it's done for the Ignite Spring Boot
> >Auto-Configuration
> ><
> >
> https://apacheignite-mix.readme.io/docs/spring-boot#set-ignite-up-via-spring-boot-configuration
> > >.
> >Guess, we can do the same with Micronaut.
> >3. If the previous

Re: Stuck in solving class not found exception

2020-08-18 Thread Denis Magda 
Hi Belal,

The following class wasn't found in the classpath of the node that failed
with the exception

Caused by: java.lang.ClassNotFoundException:
com.globant.ignite.config.CutomStoreSessionListener
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at

You need to add it to the classpath of the node.

-
Denis


On Tue, Aug 18, 2020 at 9:45 PM Belal Ahmed Khan 
wrote:

> Hi Team,
>
> I am getting an exception when my client tries to connect to the cluster, i
> have copied my application jar in the libs folder of the cluster still did
> not get why this issue is coming, please help me !
> I have written my application code in the spring boot gradle project which
> will act as a client. I can't share the project as it has some privacy of
> company
> Thanks in advance !
>
>
> centralizedAff=false, forceAffReassignment=false, exchangeLocE=null,
> cacheChangeFailureMsgSent=false, done=true, state=CRD,
> registerCachesFuture=null, partitionsSent=false, partitionsReceived=false,
> delayedLatestMsg=null, afterLsnrCompleteFut=GridFutureAdapter
> [ignoreInterrupts=false, state=DONE, res=null, hash=154939842],
> timeBag=o.a.i.i.util.TimeBag@1facd339, startTime=83336384510500,
> initTime=1597765529826, rebalanced=false, evtLatch=0, remaining=HashSet [],
> mergedJoinExchMsgs=null, awaitMergedMsgs=0, super=GridFutureAdapter
> [ignoreInterrupts=false, state=DONE, res=class o.a.i.IgniteException:
> Failed to enrich cache configuration [cacheName=positionCache],
> hash=1194223146]]
> class org.apache.ignite.IgniteCheckedException: Failed to enrich cache
> configuration [cacheName=positionCache]
> at
> org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7507)
> at
>
> org.apache.ignite.internal.util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:260)
> at
>
> org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:209)
> at
>
> org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:160)
> at
>
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3242)
> at
>
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:3063)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: class org.apache.ignite.IgniteException: Failed to enrich cache
> configuration [cacheName=positionCache]
> at
>
> org.apache.ignite.internal.processors.cache.CacheConfigurationEnricher.enrich(CacheConfigurationEnricher.java:129)
> at
>
> org.apache.ignite.internal.processors.cache.CacheConfigurationEnricher.enrich(CacheConfigurationEnricher.java:62)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheContext(GridCacheProcessor.java:1958)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheStart(GridCacheProcessor.java:1926)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.lambda$prepareStartCaches$55a0e703$1(GridCacheProcessor.java:1801)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.lambda$prepareStartCaches$8(GridCacheProcessor.java:1754)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareStartCaches(GridCacheProcessor.java:1798)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareStartCaches(GridCacheProcessor.java:1753)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.startReceivedCaches(GridCacheProcessor.java:1734)
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:840)
> at
>
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3214)
> ... 3 more
> Caused by: class org.apache.ignite.IgniteException: Failed to deserialize
> field storeSesLsnrs
> at
>
> org.apache.ignite.internal.processors.cache.CacheConfigurationEnricher.deserialize(CacheConfigurationEnricher.java:154)
> at
>
> org.apache.ignite.internal.processors.cache.CacheConfigurationEnricher.enrich(CacheConfigurationEnricher.java:122)
> ... 13 more
> Caused by: class org.apache.ignite.IgniteCheckedException: Failed to find
> class with given class loader for unmarshalling (make sure

Stuck in solving class not found exception

2020-08-18 Thread Belal Ahmed Khan 
Hi Team,

I am getting an exception when my client tries to connect to the cluster, i
have copied my application jar in the libs folder of the cluster still did
not get why this issue is coming, please help me !
I have written my application code in the spring boot gradle project which
will act as a client. I can't share the project as it has some privacy of
company
Thanks in advance !


centralizedAff=false, forceAffReassignment=false, exchangeLocE=null,
cacheChangeFailureMsgSent=false, done=true, state=CRD,
registerCachesFuture=null, partitionsSent=false, partitionsReceived=false,
delayedLatestMsg=null, afterLsnrCompleteFut=GridFutureAdapter
[ignoreInterrupts=false, state=DONE, res=null, hash=154939842],
timeBag=o.a.i.i.util.TimeBag@1facd339, startTime=83336384510500,
initTime=1597765529826, rebalanced=false, evtLatch=0, remaining=HashSet [],
mergedJoinExchMsgs=null, awaitMergedMsgs=0, super=GridFutureAdapter
[ignoreInterrupts=false, state=DONE, res=class o.a.i.IgniteException:
Failed to enrich cache configuration [cacheName=positionCache],
hash=1194223146]]
class org.apache.ignite.IgniteCheckedException: Failed to enrich cache
configuration [cacheName=positionCache]
at
org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7507)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:260)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:209)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:160)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3242)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:3063)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.IgniteException: Failed to enrich cache
configuration [cacheName=positionCache]
at
org.apache.ignite.internal.processors.cache.CacheConfigurationEnricher.enrich(CacheConfigurationEnricher.java:129)
at
org.apache.ignite.internal.processors.cache.CacheConfigurationEnricher.enrich(CacheConfigurationEnricher.java:62)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheContext(GridCacheProcessor.java:1958)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheStart(GridCacheProcessor.java:1926)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.lambda$prepareStartCaches$55a0e703$1(GridCacheProcessor.java:1801)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.lambda$prepareStartCaches$8(GridCacheProcessor.java:1754)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareStartCaches(GridCacheProcessor.java:1798)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareStartCaches(GridCacheProcessor.java:1753)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.startReceivedCaches(GridCacheProcessor.java:1734)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:840)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3214)
... 3 more
Caused by: class org.apache.ignite.IgniteException: Failed to deserialize
field storeSesLsnrs
at
org.apache.ignite.internal.processors.cache.CacheConfigurationEnricher.deserialize(CacheConfigurationEnricher.java:154)
at
org.apache.ignite.internal.processors.cache.CacheConfigurationEnricher.enrich(CacheConfigurationEnricher.java:122)
... 13 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to find
class with given class loader for unmarshalling (make sure same versions of
all classes are available on all nodes or enable peer-class-loading)
[clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6,
cls=com.globant.ignite.config.CutomStoreSessionListener]
at
org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:129)
at
org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:139)
at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:81)
at
org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10304)
at
org.apache.ignite.internal.processors.cache.CacheConfigurationEnricher.deserialize(CacheConfigurationEnricher.java:151)
... 14 more
Caused by: java.lang.ClassNotFoundException:
com.globant.ignite.config.CutomStoreSessionListener
at

Re: Micronaut/Ignite

2020-08-18 Thread Michael Pollind 
oh, so we probably don't need to work with multiple instances. This is what
I have in the current master branch. I believe I was originally trying to
set-up the configuration with the default ignite instance but found I
couldn't cover enough of the configuration. So what defaults are set for
IgniteConfiguration? some of those factory items can't be covered with how
micronaut sets up configurations. @ConfigurationProperty can only be
defined on a known factory, there are ways to have multiple factories and
label them optional but that easily gets overwhelming.  In this situation
providing your own bean would probably be more ideal in this situation when
I think about it.  I was worrying that I wouldn't be able to cover enough
of the configuration with

ignite:  enabled: true  thin-clients:default:  address:  -
"127.0.0.1:10800"  - "127.0.0.1:10801"

thin-client-2:
  address:  - "127.0.0.1:10800"  - "127.0.0.1:10801"


you can see it in the current snapshot documentation:
https://micronaut-projects.github.io/micronaut-ignite/snapshot/guide/

On Tue, Aug 18, 2020 at 4:16 PM Denis Magda  wrote:

> Michael, thanks for filling out the wiki page.
>
> I'm looking at the Auto-Configuration wiki section and the current version
> of the io.micronaut.ignite.configuration.IgniteClientConfiguration
> <
> https://github.com/micronaut-projects/micronaut-ignite/blob/master/ignite-core/src/main/java/io/micronaut/ignite/configuration/IgniteClientConfiguration.java
> >
> class,
> and wonder if we can perform the following changes:
>
>1. Rename the IgniteClientConfiguration to IgniteConfiguration (or, to
>avoid ambiguity, even to DefaultIgniteConfiguration as it's done for the
>Mongo driver
><
> https://micronaut-projects.github.io/micronaut-mongodb/latest/api/io/micronaut/configuration/mongo/reactive/DefaultMongoConfiguration.html
> >).
>The rationale for this change is that the developers might need to
> start an embedded
>Ignite server node
><
> https://www.gridgain.com/docs/latest/installation-guide/deployment-modes#embedded-deployment
> >.
>So, I would not limit the integration scope to the Ignite clients only.
>2. Presently,
> io.micronaut.ignite.configuration.IgniteClientConfiguration
>supports two parameters - the "name" and "path". I would replace the
> "path"
>parameter with the "config" one that instantiates
>org.apache.ignite.IgniteConfiguration. If we do that, then the
> developers
>will be able to set any property of the IgniteConfiguration straight in
> the
>main YAML file. See how it's done for the Ignite Spring Boot
>Auto-Configuration
><
> https://apacheignite-mix.readme.io/docs/spring-boot#set-ignite-up-via-spring-boot-configuration
> >.
>Guess, we can do the same with Micronaut.
>3. If the previous modification is feasible, then I would rework the
>Ignite thin client configuration similarly, taking our Spring Boot
>integration for the thin client
><
> https://apacheignite-mix.readme.io/docs/spring-boot#set-thin-client-up-via-spring-boot-configuration
> >
>as a reference. As I see, the current version of
>IgniteThinClientConfiguration
><
> https://github.com/micronaut-projects/micronaut-ignite/blob/master/ignite-core/src/main/java/io/micronaut/ignite/configuration/IgniteThinClientConfiguration.java
> >
> already
>adopts this approach. I would only rename "configuration" to "config",
> and
>remove the "transaction" field since you can pass the transactional
>settings via the YAML following the format below:
>
> ignite-thin-client:
> name:
> config:
> addresses: 
> partitionAwarenessEnabled: true
> transactionConfiguration:
> defaultTxConcurrency:...
> defaultTxTimeout
>
> -
> Denis
>
>
> On Mon, Aug 17, 2020 at 6:50 PM Michael Pollind 
> wrote:
>
> > oh, that makes more sense. so those methods get wrapped into a
> > micronaut-aop intercept. Below I've listed the relevant sections of code
> > that would handle each annotation along with the methods that get called
> > from the ignite branch I'm working from. Hopefully this helps. The key is
> > specified from the CacheConfig annotation but this can be changed if
> there
> > is a better way to represent the key. By default it uses this
> > DefaultCacheKeyGenerator(
> >
> >
> https://github.com/micronaut-projects/micronaut-cache/blob/master/cache-core/src/main/java/io/micronaut/cache/interceptor/DefaultCacheKeyGenerator.java
> > ).
> >
> >
> > I also finished up this document on sunday:
> > https://cwiki.apache.org/confluence/display/IGNITE/Micronaut+Integration
> .
> > Any suggestions with what I could expand on and how this could be
> adjusted.
> >
> > Cacheable:
> >
> > For Cacheable it will run a get and issue a put if the value is not
> present
> > in the cache.
> >
> > -> micronaut-cache:
> >
> >
>

Re: BEAM-9045 Implement an Ignite runner using Apache Ignite compute grid

2020-08-18 Thread Saikat Maitra 
Hi Val,

Thank you for your response. I like the idea of reactive event based
processing engine for fault tolerance. As you mentioned it will be upto
underlying system to manage job execution and offer fault tolerance and we
will need to build it in Ignite compute execution model.

I looked into Flink and Samza runners and they both offer fault
tolerance using checkpointing mechanism.

Flink - Fault-tolerance with *exactly-once* processing guarantees [1]
Samza - Fault-tolerance with support for incremental checkpointing of state
instead of full snapshots. This enables Samza to scale to applications with
very large state. [2]

I will look into it further how we can implement checkpointing[3] for
Ignite compute job when running beam pipeline.

[1] https://beam.apache.org/documentation/runners/flink/
[2] https://beam.apache.org/documentation/runners/samza/
[3] https://apacheignite.readme.io/docs/checkpointing

Regards,
Saikat





On Tue, Aug 18, 2020 at 7:29 PM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Hi Saikat,
>
> Thanks for clarifying. Is there a Beam component that monitors the state,
> or this is up to the application? If something fails, will the application
> have to retry the whole pipeline?
>
> My concern is that Ignite compute actually provides very limited
> guarantees, especially for the async execution. There are some failover
> mechanisms, but overall it's up to the application to track the state and
> retry. Moreover, if the application fails, all jobs it has submitted are
> canceled.
>
> I'm thinking that Ignite should have a reactive event-based processing
> engine. The basic idea is this:
> - an application submits an event into the cluster
> - the event is persisted in Ignite to be eventually processed
> - a processed event may result in some new events that are submitted in
> the similar fashion
>
> Ignite will provide the at-least-once guarantee (or even exactly-once
> under certain assumptions) for all the event handlers, so a user can create
> a whole chain by submitting a single event, and they don't have to worry
> about failures - it's up to Ignite to handle them.
>
> It seems to me that it might be beneficial for the Beam runner to have
> such an engine under the hood. What do you think?
>
> -Val
>
> On Mon, Aug 17, 2020 at 6:00 PM Saikat Maitra 
> wrote:
>
>> Hi,
>>
>> Luke - Thank you for sharing the details for the portability layer for
>> Flink, Samza and Spark. I will look into them and will reach out if I have
>> any questions.
>>
>> Val - Thank you for your response, yes I am planning to run the beam
>> pipeline using Ignite compute engine in async run. Here is a sample code
>> for the run method.
>>
>> IgnitePipelineResult pipelineResult = new IgnitePipelineResult(job,
>> metricsAccumulator);
>> ComputeTaskFuture computeTaskFuture =
>> ignite.compute().withAsync().run(
>> (r, f) -> {
>>   pipelineResult.freeze(f);
>>   metricsAccumulator.destroy();
>>   ignite.shutdown();
>> });
>> pipelineResult.setComputeFuture(asyncCompute.future());
>>
>> return pipelineResult;
>>
>>
>> My understanding is for failover scenarios we will need to map the job
>> state from Ignite known state to Beam Job state, an example like in
>> JetPipelineResult
>>
>> https://github.com/apache/beam/blob/master/runners/jet/src/main/java/org/apache/beam/runners/jet/JetPipelineResult.java#L68-L90
>>
>> Regards,
>> Saikat
>>
>>
>>
>>
>>
>>
>> On Mon, Aug 17, 2020 at 2:27 PM Valentin Kulichenko <
>> valentin.kuliche...@gmail.com> wrote:
>>
>> > Hi Saikat,
>> >
>> > This sounds very interesting - I've been thinking about how Ignite
>> compute
>> > engine could be enhanced, and integration with Apache Beam is one of the
>> > options I have in mind. Can you please describe how you plan to
>> implement
>> > this? Will it run on top of the Ignite Compute Grid? How are you going
>> to
>> > handle the failover, especially in the case of async pipeline execution?
>> >
>> > -Val
>> >
>> > On Sat, Aug 15, 2020 at 12:50 PM Saikat Maitra > >
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > I have been working on implementing the Apache Ignite Runner to run
>> > Apache
>> > > Beam pipeline. I have created IgniteRunner and IgnitePipelineOptions.
>> I
>> > > have implemented the normalize pipeline method and currently working
>> on
>> > run
>> > > method implementation for Pipeline and IgnitePipelineTranslator.
>> > >
>> > > Jira : https://issues.apache.org/jira/browse/BEAM-9045
>> > >
>> > > PR : https://github.com/apache/beam/pull/12593
>> > >
>> > > Please review and feel free to share any feedback or questions.
>> > >
>> > > Regards,
>> > > Saikat
>> > >
>> >
>>
>

Re: BEAM-9045 Implement an Ignite runner using Apache Ignite compute grid

2020-08-18 Thread Valentin Kulichenko 
Hi Saikat,

Thanks for clarifying. Is there a Beam component that monitors the state,
or this is up to the application? If something fails, will the application
have to retry the whole pipeline?

My concern is that Ignite compute actually provides very limited
guarantees, especially for the async execution. There are some failover
mechanisms, but overall it's up to the application to track the state and
retry. Moreover, if the application fails, all jobs it has submitted are
canceled.

I'm thinking that Ignite should have a reactive event-based processing
engine. The basic idea is this:
- an application submits an event into the cluster
- the event is persisted in Ignite to be eventually processed
- a processed event may result in some new events that are submitted in the
similar fashion

Ignite will provide the at-least-once guarantee (or even exactly-once under
certain assumptions) for all the event handlers, so a user can create a
whole chain by submitting a single event, and they don't have to worry
about failures - it's up to Ignite to handle them.

It seems to me that it might be beneficial for the Beam runner to have such
an engine under the hood. What do you think?

-Val

On Mon, Aug 17, 2020 at 6:00 PM Saikat Maitra 
wrote:

> Hi,
>
> Luke - Thank you for sharing the details for the portability layer for
> Flink, Samza and Spark. I will look into them and will reach out if I have
> any questions.
>
> Val - Thank you for your response, yes I am planning to run the beam
> pipeline using Ignite compute engine in async run. Here is a sample code
> for the run method.
>
> IgnitePipelineResult pipelineResult = new IgnitePipelineResult(job,
> metricsAccumulator);
> ComputeTaskFuture computeTaskFuture =
> ignite.compute().withAsync().run(
> (r, f) -> {
>   pipelineResult.freeze(f);
>   metricsAccumulator.destroy();
>   ignite.shutdown();
> });
> pipelineResult.setComputeFuture(asyncCompute.future());
>
> return pipelineResult;
>
>
> My understanding is for failover scenarios we will need to map the job
> state from Ignite known state to Beam Job state, an example like in
> JetPipelineResult
>
> https://github.com/apache/beam/blob/master/runners/jet/src/main/java/org/apache/beam/runners/jet/JetPipelineResult.java#L68-L90
>
> Regards,
> Saikat
>
>
>
>
>
>
> On Mon, Aug 17, 2020 at 2:27 PM Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
> > Hi Saikat,
> >
> > This sounds very interesting - I've been thinking about how Ignite
> compute
> > engine could be enhanced, and integration with Apache Beam is one of the
> > options I have in mind. Can you please describe how you plan to implement
> > this? Will it run on top of the Ignite Compute Grid? How are you going to
> > handle the failover, especially in the case of async pipeline execution?
> >
> > -Val
> >
> > On Sat, Aug 15, 2020 at 12:50 PM Saikat Maitra 
> > wrote:
> >
> > > Hi,
> > >
> > > I have been working on implementing the Apache Ignite Runner to run
> > Apache
> > > Beam pipeline. I have created IgniteRunner and IgnitePipelineOptions. I
> > > have implemented the normalize pipeline method and currently working on
> > run
> > > method implementation for Pipeline and IgnitePipelineTranslator.
> > >
> > > Jira : https://issues.apache.org/jira/browse/BEAM-9045
> > >
> > > PR : https://github.com/apache/beam/pull/12593
> > >
> > > Please review and feel free to share any feedback or questions.
> > >
> > > Regards,
> > > Saikat
> > >
> >
>

Re: Micronaut/Ignite

2020-08-18 Thread Denis Magda 
Michael, thanks for filling out the wiki page.

I'm looking at the Auto-Configuration wiki section and the current version
of the io.micronaut.ignite.configuration.IgniteClientConfiguration

class,
and wonder if we can perform the following changes:

   1. Rename the IgniteClientConfiguration to IgniteConfiguration (or, to
   avoid ambiguity, even to DefaultIgniteConfiguration as it's done for the
   Mongo driver
   
).
   The rationale for this change is that the developers might need to
start an embedded
   Ignite server node
   
.
   So, I would not limit the integration scope to the Ignite clients only.
   2. Presently, io.micronaut.ignite.configuration.IgniteClientConfiguration
   supports two parameters - the "name" and "path". I would replace the "path"
   parameter with the "config" one that instantiates
   org.apache.ignite.IgniteConfiguration. If we do that, then the developers
   will be able to set any property of the IgniteConfiguration straight in the
   main YAML file. See how it's done for the Ignite Spring Boot
   Auto-Configuration
   
.
   Guess, we can do the same with Micronaut.
   3. If the previous modification is feasible, then I would rework the
   Ignite thin client configuration similarly, taking our Spring Boot
   integration for the thin client
   

   as a reference. As I see, the current version of
   IgniteThinClientConfiguration
   

already
   adopts this approach. I would only rename "configuration" to "config", and
   remove the "transaction" field since you can pass the transactional
   settings via the YAML following the format below:

ignite-thin-client:
name:
config:
addresses: 
partitionAwarenessEnabled: true
transactionConfiguration:
defaultTxConcurrency:...
defaultTxTimeout

-
Denis


On Mon, Aug 17, 2020 at 6:50 PM Michael Pollind  wrote:

> oh, that makes more sense. so those methods get wrapped into a
> micronaut-aop intercept. Below I've listed the relevant sections of code
> that would handle each annotation along with the methods that get called
> from the ignite branch I'm working from. Hopefully this helps. The key is
> specified from the CacheConfig annotation but this can be changed if there
> is a better way to represent the key. By default it uses this
> DefaultCacheKeyGenerator(
>
> https://github.com/micronaut-projects/micronaut-cache/blob/master/cache-core/src/main/java/io/micronaut/cache/interceptor/DefaultCacheKeyGenerator.java
> ).
>
>
> I also finished up this document on sunday:
> https://cwiki.apache.org/confluence/display/IGNITE/Micronaut+Integration.
> Any suggestions with what I could expand on and how this could be adjusted.
>
> Cacheable:
>
> For Cacheable it will run a get and issue a put if the value is not present
> in the cache.
>
> -> micronaut-cache:
>
> https://github.com/micronaut-projects/micronaut-cache/blob/master/cache-core/src/main/java/io/micronaut/cache/interceptor/CacheInterceptor.java#L163-L170
> -> ignite-cache:
>   get:
>
> https://github.com/pollend/micronaut-ignite/blob/feature/rework/ignite-cache/src/main/java/io/micronaut/ignite/IgniteSyncCache.java#L60-L70
>
> CachePut:
>
> For cache put it will invalidate if the return is null else it will issue a
> put. I think there might be a mistake in my code because I use putIfAbsent
> for both cases. I need to investigate that closer and write some additional
> test cases to verify the behaviour.
>
> --> micronaut-cache:
>
> https://github.com/micronaut-projects/micronaut-cache/blob/master/cache-core/src/main/java/io/micronaut/cache/interceptor/CacheInterceptor.java#L510-L525
> -> ignite-cache:
> put:
>
> https://github.com/pollend/micronaut-ignite/blob/feature/rework/ignite-cache/src/main/java/io/micronaut/ignite/IgniteSyncCache.java#L83-L88
> invalidate:
>
> https://github.com/pollend/micronaut-ignite/blob/feature/rework/ignite-cache/src/main/java/io/micronaut/ignite/IgniteSyncCache.java#L91-L95
>
> CacheInvalidate:
>
> for cache invalidation it will just be removed by the key.
>
> --> micronaut-cache:
>
> https://github.com/micronaut-projects/micronaut-cache/blob/master/cache-core/src/main/java/io/micronaut/cache/interceptor/CacheInterceptor.java#L590-L596
> -> ignite-cache:

Re: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]

2020-08-18 Thread Pavel Tupitsyn 
Ok, I'm just asking, there is no rush for this ticket.
If the release is postponed anyway, I thought that we can include more
fixes.

On Tue, Aug 18, 2020 at 8:10 PM Alex Plehanov 
wrote:

> Pavel,
>
> We still can't find the root cause of performance drop.
>
> Ticket IGNITE-13369 still in progress. When it will be resolved? Is it
> really critical bug? According to the user-list thread attached to the
> ticket, there is a workaround exists for this problem and looks like it's
> not so critical.
>
> вт, 18 авг. 2020 г. в 12:30, Pavel Tupitsyn :
>
> > Alex,
> >
> > What's the status of the release?
> > Can we include a bug fix there [1]?
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-13369
> >
> > On Fri, Aug 14, 2020 at 11:14 AM Alex Plehanov 
> > wrote:
> >
> > > Hello,
> > >
> > > > What is the release date for 2.9, from the cwiki it still says
> August 7
> > > We have a performance drop on the new release and still trying to find
> > the
> > > problematic commit. I hope we will find it in a week + there are one or
> > two
> > > weeks needed for voting and further release steps.
> > >
> > > пт, 14 авг. 2020 г. в 11:01, gaurav.ag...@gmail.com <
> > > gaurav.ag...@gmail.com
> > > >:
> > >
> > > > What is the release date for 2.9, from the cwiki it still says August
> > 7 :
> > > > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.9
> > > >
> > > >
> > > >
> > > > --
> > > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> > > >
> > >
> >
>

Re: Operation block on Cluster recovery/rebalance.

2020-08-18 Thread Denis Magda 
John, thanks for filing. Back up the Ilya's proposal. I've changed the
ticket from the improvement to the bug.

-
Denis


On Tue, Aug 18, 2020 at 10:06 AM Ilya Kasnacheev 
wrote:

> Hello!
>
> My opinion is that it should be filed as a bug and then fixed.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> сб, 15 авг. 2020 г. в 01:22, Denis Magda :
>
>> @Evgenii Zhuravlev , @Ilya Kasnacheev
>> , any thoughts on this?
>>
>> As a dirty workaround, you can update your cache references on client
>> reconnect events. You will be getting an exception by calling
>> ignite.cache(cacheName) in the time when the cluster is not activated yet.
>> Does this work for you?
>>
>> -
>> Denis
>>
>>
>> On Fri, Aug 14, 2020 at 3:12 PM John Smith 
>> wrote:
>>
>>> Is there any work around? I can't have an HTTP server block on all
>>> requests.
>>>
>>> 1- I need to figure out why I lose a server nodes every few weeks, which
>>> when rebooting the nodes cause the inactive state until they are back
>>>
>>> 2- Implement some kind of logic on the client side not to block the HTTP
>>> part...
>>>
>>> Can IgniteCache instance be notified of disconnected events so I can
>>> maybe tell the repository class I have to set a flag to skip the operation?
>>>
>>>
>>> On Fri., Aug. 14, 2020, 5:17 p.m. Denis Magda, 
>>> wrote:
>>>
 My guess that it's standard behavior for all operations (SQL,
 key-value, compute, etc.). But I'll let the maintainers of those modules
 clarify.

 -
 Denis


 On Fri, Aug 14, 2020 at 1:44 PM John Smith 
 wrote:

> Hi Denis, so to understand it's all operations or just the query?
>
> On Fri., Aug. 14, 2020, 12:53 p.m. Denis Magda, 
> wrote:
>
>> John,
>>
>> Ok, we nailed it. That's the current expected behavior. Generally, I
>> agree with you that the platform should support an option when operations
>> fail if the cluster is deactivated. Could you propose the change by
>> starting a discussion on the dev list? You can refer to this user list
>> discussion for reference. Let me know if you need help with this.
>>
>> -
>> Denis
>>
>>
>> On Thu, Aug 13, 2020 at 5:55 PM John Smith 
>> wrote:
>>
>>> No I, reuse the instance. The cache instance is created once at
>>> startup of the application and I pass it to my "repository" class
>>>
>>> public abstract class AbstractIgniteRepository implements 
>>> CacheRepository {
>>> public final long DEFAULT_OPERATION_TIMEOUT = 2000;
>>>
>>> private Vertx vertx;
>>> private IgniteCache cache;
>>>
>>> AbstractIgniteRepository(Vertx vertx, IgniteCache cache) {
>>> this.vertx = vertx;
>>> this.cache = cache;
>>> }
>>>
>>> ...
>>>
>>> Future> query(final String sql, final long 
>>> timeoutMs, final Object... args) {
>>> final Promise> promise = Promise.promise();
>>>
>>> vertx.setTimer(timeoutMs, l -> {
>>> promise.tryFail(new TimeoutException("Cache operation did 
>>> not complete within: " + timeoutMs + " Ms.")); // THIS FIRE IF THE BLOE 
>>> DOESN"T COMPLETE IN TIME.
>>> });
>>>
>>> vertx.>executeBlocking(code -> {
>>> SqlFieldsQuery query = new 
>>> SqlFieldsQuery(sql).setArgs(args);
>>> query.setTimeout((int) timeoutMs, TimeUnit.MILLISECONDS);
>>>
>>>
>>> try (QueryCursor> cursor = cache.query(query)) { // 
>>> <--- BLOCKS HERE.
>>> List rows = new ArrayList<>();
>>> Iterator> iterator = cursor.iterator();
>>>
>>> while(iterator.hasNext()) {
>>> List currentRow = iterator.next();
>>> JsonArray row = new JsonArray();
>>>
>>> currentRow.forEach(o -> row.add(o));
>>>
>>> rows.add(row);
>>> }
>>>
>>> code.complete(rows);
>>> } catch(Exception ex) {
>>> code.fail(ex);
>>> }
>>> }, result -> {
>>> if(result.succeeded()) {
>>> promise.tryComplete(result.result());
>>> } else {
>>> promise.tryFail(result.cause());
>>> }
>>> });
>>>
>>> return promise.future();
>>> }
>>>
>>> public  T cache() {
>>> return (T) cache;
>>> }
>>> }
>>>
>>>
>>>
>>> On Thu, 13 Aug 2020 at 16:29, Denis Magda  wrote:
>>>
 I've created a simple test and always getting the exception below
 on an attempt to get a reference to an IgniteCache instance in cases 
 when
 the cluster is not activated:

 *Exception in thread "main" class

Re: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]

2020-08-18 Thread Alex Plehanov 
Pavel,

We still can't find the root cause of performance drop.

Ticket IGNITE-13369 still in progress. When it will be resolved? Is it
really critical bug? According to the user-list thread attached to the
ticket, there is a workaround exists for this problem and looks like it's
not so critical.

вт, 18 авг. 2020 г. в 12:30, Pavel Tupitsyn :

> Alex,
>
> What's the status of the release?
> Can we include a bug fix there [1]?
>
> [1] https://issues.apache.org/jira/browse/IGNITE-13369
>
> On Fri, Aug 14, 2020 at 11:14 AM Alex Plehanov 
> wrote:
>
> > Hello,
> >
> > > What is the release date for 2.9, from the cwiki it still says August 7
> > We have a performance drop on the new release and still trying to find
> the
> > problematic commit. I hope we will find it in a week + there are one or
> two
> > weeks needed for voting and further release steps.
> >
> > пт, 14 авг. 2020 г. в 11:01, gaurav.ag...@gmail.com <
> > gaurav.ag...@gmail.com
> > >:
> >
> > > What is the release date for 2.9, from the cwiki it still says August
> 7 :
> > > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.9
> > >
> > >
> > >
> > > --
> > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> > >
> >
>

Re: Operations block when cache is not active.

2020-08-18 Thread John Smith 
For everyone's reference: https://issues.apache.org/jira/browse/IGNITE-13372

On Tue, 18 Aug 2020 at 07:15, Ilya Kasnacheev 
wrote:

> Hello!
>
> Please file an issue against Apache Ignite JIRA.
>
> https://issues.apache.org/jira/projects/IGNITE/issues
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> сб, 15 авг. 2020 г. в 18:07, John Smith :
>
> > Hi I was told by Dennis to post this here. But basically he said to
> propose
> > a change not to make operations block when the cluster is in inactive
> > state. The operation should throw an exception to let the application
> > continue working.
> >
> > Here is the original thread.
> >
> >
> >
> http://apache-ignite-users.70518.x6.nabble.com/Operation-block-on-Cluster-recovery-rebalance-td33579.html
> >
>

[jira] [Created] (IGNITE-13372) Operations block when cluster becomes inactive.

2020-08-18 Thread None none (Jira) 
None none created IGNITE-13372:
--

 Summary: Operations block when cluster becomes inactive.
 Key: IGNITE-13372
 URL: https://issues.apache.org/jira/browse/IGNITE-13372
 Project: Ignite
  Issue Type: Improvement
Affects Versions: 2.8, 2.7
 Environment: 3 server nodes using I.P discovery and multiple client 
nodes (client =true)
Server nodes are 32GB total of which each is configure with 6GB heap and 10GB 
native persistence.
Reporter: None none


Operations block indefinitely when the cluster state becomes inactive. Initial 
discussion is here: 
[http://apache-ignite-users.70518.x6.nabble.com/Operation-block-on-Cluster-recovery-rebalance-td33579.html]
 # Start a server cluster with persistence.
 # Start thick client (client = true) application with Ignition.start(), create 
IgniteCache instance.
 # IgniteCache instance is initialized once at beginning of application and 
shared between "threads". This particular application is a REST API and uses 
the IgniteCache instance when HTTP request is made.
 ## When HTTP request comes in first execute cache.query()
 ## If successful do cache.put() 
 ## Reply back to client.
 # Shut off some nodes and bring back some to ensure that the cluster is in 
inactive state.
 # As the application continues to operate and since the IgniteCache instance 
was initialized once at the beginning of the application, the operation blocks 
indefinitely and renders the application unresponsive.
 # As per the discussion thread it was advised to listen for node events and 
re-initialize the IgniteCache instance. This is not user friendly or intuitive 
as the client should handle all failover scenarios it's better that the 
operation fail with exception so at least the application can respond and the 
IgniteCache instance should somehow recover when the cluster is back to active.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13371) Sporadic partition inconsistency after historical rebalancing of updates with same key put-remove pattern

2020-08-18 Thread Ivan Rakov (Jira) 
Ivan Rakov created IGNITE-13371:
---

 Summary: Sporadic partition inconsistency after historical 
rebalancing of updates with same key put-remove pattern
 Key: IGNITE-13371
 URL: https://issues.apache.org/jira/browse/IGNITE-13371
 Project: Ignite
  Issue Type: Bug
Reporter: Ivan Rakov
Assignee: Ivan Rakov
 Fix For: 2.10


h4. scenario
# start 3 servers 3 clients, create caches
# clients start combined put + 1% remove of data in transactions 
PESSIMISTIC/REPEATABLE_READ
## kill one node
## restart one node
# ensure all transactions completed
# run idle_verify

Expected: no conflicts found
Actual:
{noformat}
[12:03:18][:55 :230] Control utility --cache idle_verify --skip-zeros 
--cache-filter PERSISTENT
[12:03:20][:55 :230] Control utility [ver. 8.7.13#20200228-sha1:7b016d63]
[12:03:20][:55 :230] 2020 Copyright(C) GridGain Systems, Inc. and Contributors
[12:03:20][:55 :230] User: prtagent
[12:03:20][:55 :230] Time: 2020-03-03T12:03:19.836
[12:03:20][:55 :230] Command [CACHE] started
[12:03:20][:55 :230] Arguments: --host 172.25.1.11 --port 11211 --cache 
idle_verify --skip-zeros --cache-filter PERSISTENT 
[12:03:20][:55 :230] 

[12:03:20][:55 :230] idle_verify task was executed with the following args: 
caches=[], excluded=[], cacheFilter=[PERSISTENT]
[12:03:20][:55 :230] idle_verify check has finished, found 1 conflict 
partitions: [counterConflicts=0, hashConflicts=1]
[12:03:20][:55 :230] Hash conflicts:
[12:03:20][:55 :230] Conflict partition: PartitionKeyV2 [grpId=1338167321, 
grpName=cache_group_3_088_1, partId=24]
[12:03:20][:55 :230] Partition instances: [PartitionHashRecordV2 
[isPrimary=false, consistentId=node_1_2, updateCntr=172349, 
partitionState=OWNING, size=6299, partHash=157875238], PartitionHashRecordV2 
[isPrimary=true, consistentId=node_1_1, updateCntr=172349, 
partitionState=OWNING, size=6299, partHash=157875238], PartitionHashRecordV2 
[isPrimary=false, consistentId=node_1_4, updateCntr=172349, 
partitionState=OWNING, size=6300, partHash=-944532882]]
[12:03:20][:55 :230] Command [CACHE] finished with code: 0
[12:03:20][:55 :230] Control utility has completed execution at: 
2020-03-03T12:03:20.593
[12:03:20][:55 :230] Execution time: 757 ms
{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Operations block when cache is not active.

2020-08-18 Thread Ilya Kasnacheev 
Hello!

Please file an issue against Apache Ignite JIRA.

https://issues.apache.org/jira/projects/IGNITE/issues

Regards,
-- 
Ilya Kasnacheev


сб, 15 авг. 2020 г. в 18:07, John Smith :

> Hi I was told by Dennis to post this here. But basically he said to propose
> a change not to make operations block when the cluster is in inactive
> state. The operation should throw an exception to let the application
> continue working.
>
> Here is the original thread.
>
>
> http://apache-ignite-users.70518.x6.nabble.com/Operation-block-on-Cluster-recovery-rebalance-td33579.html
>

Re: Apache Ignite 3.0

2020-08-18 Thread Ilya Kasnacheev 
Hello!

Maybe we can implement messaging via compute? Share underlying code but
have a different API for convenience (maybe in extras).

Regards,
-- 
Ilya Kasnacheev


вт, 18 авг. 2020 г. в 11:59, Pavel Tupitsyn :

> Val,
>
> > Is see the ".NET Modernization for Ignite 3.0" item in the roadmap, but
> it
> > doesn't provide much detail. What are the actual changes in .NET that you
> > propose for 3.0?
>
> I've updated the roadmap with a brief description.
> We want to move away from an old, long unsupported .NET version,
> and remove integrations with legacy technologies (old ASP.NET and EF).
>
> > As for the messaging, so far I haven't seen use cases that would require
> > this API. Sometimes users attempt to use it for remote code invocation,
> but
> > compute is usually a better option for this. Do you have any examples
> where
> > messaging is a better fit then compute?
>
> I guess you are right, Compute can replace Messaging in most scenarios.
>
> Messaging can be more convenient, since it is pub-sub - subscriber controls
> whether it receives messages on the given topic. But this can be achieved
> with a little more work with Compute as well.
>
> Thanks,
> Pavel
>
>
>
> On Tue, Aug 18, 2020 at 3:16 AM Saikat Maitra 
> wrote:
>
> > Hi Pavel,
> >
> > Awesome, thank you.
> >
> > Yes, I remember having .Net modernization as part of Apache Ignite 3.0
> > roadmap.
> >
> > Regards,
> > Saikat
> >
> > On Sun, Aug 16, 2020 at 11:04 AM Pavel Tupitsyn 
> > wrote:
> >
> > > Saikat, yes, most definitely.
> > > This is mentioned in the wishlist under ".NET: Target .NET Standard
> 2.0,
> > > discontinue .NET 4.0 support".
> > > I'm already working towards this goal by making more code and tests
> work
> > > properly under .NET Core,
> > > so when the time for breaking changes comes, it will be simpler.
> > >
> > > On Sat, Aug 15, 2020 at 10:06 PM Saikat Maitra <
> saikat.mai...@gmail.com>
> > > wrote:
> > >
> > > > Hi Val,
> > > >
> > > > Thank you for adding the Cleanup section and Removals list.
> > > >
> > > > Pavel, As part of Apache Ignite Roadmap we had mentioned we will add
> > > > modernization of .NET. Are we still targeting it in Apache Ignite 3.0
> > > > release?
> > > >
> > > >
> > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+Roadmap
> > > >
> > > > Regards,
> > > > Saikat
> > > >
> > > >
> > > >
> > > > On Fri, Aug 14, 2020 at 10:04 AM Carbone, Adam <
> > > > adam.carb...@bottomline.com>
> > > > wrote:
> > > >
> > > > > If you want to make is simpler to have the components that you
> want,
> > > but
> > > > > have that be immutable at install time you could take an approach
> > > similar
> > > > > to the way spring does it with their initializer (
> > > > > https://start.spring.io/ )  as an example... Basically the concept
> > > being
> > > > > something that produces a set of configurations that are used to
> > define
> > > > > what the environment looks like ( these could be k8s objects ) they
> > > could
> > > > > be spring-configuration objects? They could be something that you
> > > develop
> > > > > all upon ignite ( probably wouldn’t recommend this approach )
> there
> > > > seems
> > > > > to be plenty of these types of things already
> > > > >
> > > > > Example
> > > > > * https://spring.io/guides/gs/centralized-configuration
> > > > > *
> > > > >
> > > >
> > >
> >
> https://cocoon.apache.org/subprojects/configuration/spring-configurator/index.html
> > > > >
> > > > > And I'm by no means saying to use spring these are just examples
> > that I
> > > > > came across
> > > > >
> > > > > I'm thinking the outcome needs to be a platform config of source (
> > that
> > > > > can be checked in for those doing gitops ) and maybe ends up as a
> > > config
> > > > > map for those doing k8s, and some other fashion for those doing
> > > something
> > > > > else.
> > > > >
> > > > > Honestly I am not deep enough into the internals of ignite to know
> > how
> > > > > this might work for the platform, was more describing what I'm
> seeing
> > > > from
> > > > > a bigger picture trend
> > > > >
> > > > > Regards
> > > > >
> > > > > Adam Carbone | Director of Innovation – Intelligent Platform Team |
> > > > > Bottomline Technologies
> > > > > Office: 603-501-6446 | Mobile: 603-570-8418
> > > > > www.bottomline.com
> > > > >
> > > > >
> > > > >
> > > > > On 8/13/20, 7:55 PM, "Valentin Kulichenko" <
> > > > valentin.kuliche...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > Hi Ilya,
> > > > >
> > > > > Can you please describe your vision of how it should work?
> > > > >
> > > > > Let's say, I want to set up a cluster of several standalone
> > server
> > > > > nodes
> > > > > with a couple of optional modules enabled. What are my steps?
> > > > >
> > > > > -Val
> > > > >
> > > > > On Thu, Aug 13, 2020 at 6:03 AM Carbone, Adam <
> > > > > adam.carb...@bottomline.com>
> > > > > wrote:
> > > > >
> > > > > > Good Morning from the EastCoast
> > > > > >
> > > > >

[jira] [Created] (IGNITE-13370) Missed README.txt in ignite-control-utility module

2020-08-18 Thread Kirill Tkalenko (Jira) 
Kirill Tkalenko created IGNITE-13370:


 Summary: Missed README.txt in ignite-control-utility module
 Key: IGNITE-13370
 URL: https://issues.apache.org/jira/browse/IGNITE-13370
 Project: Ignite
  Issue Type: Improvement
  Components: control.sh
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
 Fix For: 2.10


Missed README.txt in ignite-control-utility module



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]

2020-08-18 Thread Pavel Tupitsyn 
Alex,

What's the status of the release?
Can we include a bug fix there [1]?

[1] https://issues.apache.org/jira/browse/IGNITE-13369

On Fri, Aug 14, 2020 at 11:14 AM Alex Plehanov 
wrote:

> Hello,
>
> > What is the release date for 2.9, from the cwiki it still says August 7
> We have a performance drop on the new release and still trying to find the
> problematic commit. I hope we will find it in a week + there are one or two
> weeks needed for voting and further release steps.
>
> пт, 14 авг. 2020 г. в 11:01, gaurav.ag...@gmail.com <
> gaurav.ag...@gmail.com
> >:
>
> > What is the release date for 2.9, from the cwiki it still says August 7 :
> > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.9
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> >
>

[jira] [Created] (IGNITE-13369) .NET: Local node info is not updated on client reconnect

2020-08-18 Thread Pavel Tupitsyn (Jira) 
Pavel Tupitsyn created IGNITE-13369:
---

 Summary: .NET: Local node info is not updated on client reconnect
 Key: IGNITE-13369
 URL: https://issues.apache.org/jira/browse/IGNITE-13369
 Project: Ignite
  Issue Type: Bug
  Components: platforms
Reporter: Pavel Tupitsyn
Assignee: Pavel Tupitsyn
 Fix For: 2.10


{{Ignite._locNode}} field caches local node information, and this cache info is 
not updated on client reconnect. We should remove cached info on every 
disconnect.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Apache Ignite 3.0

2020-08-18 Thread Pavel Tupitsyn 
Val,

> Is see the ".NET Modernization for Ignite 3.0" item in the roadmap, but it
> doesn't provide much detail. What are the actual changes in .NET that you
> propose for 3.0?

I've updated the roadmap with a brief description.
We want to move away from an old, long unsupported .NET version,
and remove integrations with legacy technologies (old ASP.NET and EF).

> As for the messaging, so far I haven't seen use cases that would require
> this API. Sometimes users attempt to use it for remote code invocation,
but
> compute is usually a better option for this. Do you have any examples
where
> messaging is a better fit then compute?

I guess you are right, Compute can replace Messaging in most scenarios.

Messaging can be more convenient, since it is pub-sub - subscriber controls
whether it receives messages on the given topic. But this can be achieved
with a little more work with Compute as well.

Thanks,
Pavel



On Tue, Aug 18, 2020 at 3:16 AM Saikat Maitra 
wrote:

> Hi Pavel,
>
> Awesome, thank you.
>
> Yes, I remember having .Net modernization as part of Apache Ignite 3.0
> roadmap.
>
> Regards,
> Saikat
>
> On Sun, Aug 16, 2020 at 11:04 AM Pavel Tupitsyn 
> wrote:
>
> > Saikat, yes, most definitely.
> > This is mentioned in the wishlist under ".NET: Target .NET Standard 2.0,
> > discontinue .NET 4.0 support".
> > I'm already working towards this goal by making more code and tests work
> > properly under .NET Core,
> > so when the time for breaking changes comes, it will be simpler.
> >
> > On Sat, Aug 15, 2020 at 10:06 PM Saikat Maitra 
> > wrote:
> >
> > > Hi Val,
> > >
> > > Thank you for adding the Cleanup section and Removals list.
> > >
> > > Pavel, As part of Apache Ignite Roadmap we had mentioned we will add
> > > modernization of .NET. Are we still targeting it in Apache Ignite 3.0
> > > release?
> > >
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+Roadmap
> > >
> > > Regards,
> > > Saikat
> > >
> > >
> > >
> > > On Fri, Aug 14, 2020 at 10:04 AM Carbone, Adam <
> > > adam.carb...@bottomline.com>
> > > wrote:
> > >
> > > > If you want to make is simpler to have the components that you want,
> > but
> > > > have that be immutable at install time you could take an approach
> > similar
> > > > to the way spring does it with their initializer (
> > > > https://start.spring.io/ )  as an example... Basically the concept
> > being
> > > > something that produces a set of configurations that are used to
> define
> > > > what the environment looks like ( these could be k8s objects ) they
> > could
> > > > be spring-configuration objects? They could be something that you
> > develop
> > > > all upon ignite ( probably wouldn’t recommend this approach )  there
> > > seems
> > > > to be plenty of these types of things already
> > > >
> > > > Example
> > > > * https://spring.io/guides/gs/centralized-configuration
> > > > *
> > > >
> > >
> >
> https://cocoon.apache.org/subprojects/configuration/spring-configurator/index.html
> > > >
> > > > And I'm by no means saying to use spring these are just examples
> that I
> > > > came across
> > > >
> > > > I'm thinking the outcome needs to be a platform config of source (
> that
> > > > can be checked in for those doing gitops ) and maybe ends up as a
> > config
> > > > map for those doing k8s, and some other fashion for those doing
> > something
> > > > else.
> > > >
> > > > Honestly I am not deep enough into the internals of ignite to know
> how
> > > > this might work for the platform, was more describing what I'm seeing
> > > from
> > > > a bigger picture trend
> > > >
> > > > Regards
> > > >
> > > > Adam Carbone | Director of Innovation – Intelligent Platform Team |
> > > > Bottomline Technologies
> > > > Office: 603-501-6446 | Mobile: 603-570-8418
> > > > www.bottomline.com
> > > >
> > > >
> > > >
> > > > On 8/13/20, 7:55 PM, "Valentin Kulichenko" <
> > > valentin.kuliche...@gmail.com>
> > > > wrote:
> > > >
> > > > Hi Ilya,
> > > >
> > > > Can you please describe your vision of how it should work?
> > > >
> > > > Let's say, I want to set up a cluster of several standalone
> server
> > > > nodes
> > > > with a couple of optional modules enabled. What are my steps?
> > > >
> > > > -Val
> > > >
> > > > On Thu, Aug 13, 2020 at 6:03 AM Carbone, Adam <
> > > > adam.carb...@bottomline.com>
> > > > wrote:
> > > >
> > > > > Good Morning from the EastCoast
> > > > >
> > > > > I have to agree that the larger industry is tending towards
> > > > immutability,
> > > > > and that you build once and test, then you promote/migrate that
> > > > immutable
> > > > > binary object, be is a library or a docker image etc... however
> > > > there are
> > > > > still patterns that allow you to determine at install/or
> > deployment
> > > > time (
> > > > > helm as an example, you choose based on your values what the
> > > package
> > > > > installs/provides ) It just isn't decided at runtime but
>

Re: [MTCGA]: new failures in builds [5539465] needs to be handled

2020-08-18 Thread Anton Kalashnikov 
Hi,

It's my fail. I will fix it.
https://issues.apache.org/jira/browse/IGNITE-13368

-- 
Best regards,
Anton Kalashnikov



17.08.2020, 22:52, "dpavlov.ta...@gmail.com" :
> Hi Igniters,
>
>  I've detected some new issue on TeamCity to be handled. You are more than 
> welcomed to help.
>
>  If your changes can lead to this failure(s): We're grateful that you were a 
> volunteer to make the contribution to this project, but things change and you 
> may no longer be able to finalize your contribution.
>  Could you respond to this email and indicate if you wish to continue and fix 
> test failures or step down and some committer may revert you commit.
>
>  * New test failure in master PagesWriteThrottleSmokeTest.testThrottle 
> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=2808794487465215609=%3Cdefault%3E=testDetails
>  Changes may lead to failure were done by
>  - ivan rakov  
> https://ci.ignite.apache.org/viewModification.html?modId=905838
>  - ymolochkov  
> https://ci.ignite.apache.org/viewModification.html?modId=905833
>  - anton kalashnikov  
> https://ci.ignite.apache.org/viewModification.html?modId=905850
>  - ibessonov  
> https://ci.ignite.apache.org/viewModification.html?modId=905840
>
>  - Here's a reminder of what contributors were agreed to do 
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute
>  - Should you have any questions please contact dev@ignite.apache.org
>
> Best Regards,
> Apache Ignite TeamCity Bot
> https://github.com/apache/ignite-teamcity-bot
> Notification generated at 22:44:23 17-08-2020

[jira] [Created] (IGNITE-13368) Speed base throttling unexpectedly degraded to zero

2020-08-18 Thread Anton Kalashnikov (Jira) 
Anton Kalashnikov created IGNITE-13368:
--

 Summary: Speed base throttling unexpectedly degraded to zero
 Key: IGNITE-13368
 URL: https://issues.apache.org/jira/browse/IGNITE-13368
 Project: Ignite
  Issue Type: Bug
Reporter: Anton Kalashnikov
Assignee: Anton Kalashnikov


New test failure in master PagesWriteThrottleSmokeTest.testThrottle 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=2808794487465215609=%3Cdefault%3E=testDetails

Throttling degraded to zero.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13367) meta --remove command usage improvements

2020-08-18 Thread Sergey Chugunov (Jira) 
Sergey Chugunov created IGNITE-13367:


 Summary: meta --remove command usage improvements
 Key: IGNITE-13367
 URL: https://issues.apache.org/jira/browse/IGNITE-13367
 Project: Ignite
  Issue Type: Improvement
Reporter: Sergey Chugunov
Assignee: Sergey Chugunov
 Fix For: 2.10


Command for removing metadata has the following issues:
# In 'Type not found' scenario it prints long stack traces to console instead 
of short information about requested type.
# When used it registers some internal classes which are not supposed to go 
through binary metadata registration protocol.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)