[jira] [Created] (IGNITE-13013) Thick client must not open server sockets when used by serverless functions

[Denis A. Magda (Jira)]

Denis A. Magda created IGNITE-13013:
---

 Summary: Thick client must not open server sockets when used by 
serverless functions
 Key: IGNITE-13013
 URL: https://issues.apache.org/jira/browse/IGNITE-13013
 Project: Ignite
  Issue Type: Improvement
  Components: networking
Affects Versions: 2.8
Reporter: Denis A. Magda
 Fix For: 2.9


A thick client fails to start if being used inside of a serverless function 
such as AWS Lamda or Azure Functions. Cloud providers prohibit opening network 
ports to accept connections on the function's end. In short, the function can 
only connect to a remote address.

The thick client needs to support a mode when the communication SPI doesn't 
create a server socket if the client is used for serverless computing. This 
improvement looks like an extra task of this initiative: 
https://issues.apache.org/jira/browse/IGNITE-12438



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13012) Make node connection checking rely on the configuration. Simplify node ping routine.

[Vladimir Steshin (Jira)]

Vladimir Steshin created IGNITE-13012:
-

 Summary: Make node connection checking rely on the configuration. 
Simplify node ping routine.
 Key: IGNITE-13012
 URL: https://issues.apache.org/jira/browse/IGNITE-13012
 Project: Ignite
  Issue Type: Improvement
Reporter: Vladimir Steshin
Assignee: Vladimir Steshin



Current noted-to-node connection checking has several drawbacks:
1)  Minimal connection checking interval is not bound to failure detection 
parameters: 
static int ServerImpls.CON_CHECK_INTERVAL = 500;
2)  Connection checking is made as ability of periodical message sending 
(TcpDiscoveryConnectionCheckMessage). It is bound to own time (ServerImpl. 
RingMessageWorker.lastTimeConnCheckMsgSent), not to common time of last sent 
message. This is weird because any discovery message actually checks 
connection. And TpDiscoveryConnectionCheckMessage is just an addition when 
message queue is empty for a long time.
3)  Period of Node-to-Node connection checking can be sometimes shortened 
for strange reason: if no sent or received message appears within 
failureDetectionTimeout. Here, despite we have minimal period of connection 
checking (ServerImpls.CON_CHECK_INTERVAL), we can also send 
TpDiscoveryConnectionCheckMessage before this period exhausted. Moreover, this 
premature node ping relies also on time of last received message. Imagine: if 
node 2 receives no message from node 1 within some time it decides to do extra 
ping node 3 not waiting for regular ping interval. Such behavior makes 
confusion and gives no additional guaranties.
4)  If #3 happens, node writes in the log on INFO: “Local node seems to be 
disconnected from topology …” whereas it is not actually disconnected. User can 
see this message if he typed failureDetectionTimeout < 500ms. I wouldn’t like 
seeing INFO in a log saying a node is might be disconnected. This sounds like 
some troubles raised in network. But not as everything is OK. 

Suggestions:
1)  Make connection check interval be based on failureDetectionTimeout or 
similar params.
2)  Make connection check interval rely on common time of last sent 
message. Not on dedicated time.
3)  Remove additional, random, quickened connection checking.
4)  Do not worry user with “Node disconnected” when everything is OK.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Apache URL for TC bot

[Dmitriy Pavlov]

Thank you, it works perfectly!

чт, 14 мая 2020 г. в 11:09, Ivan Pavlukhin :

> Petr, Ivan,
>
> Good stuff! Thank you!
>
> Best regards,
> Ivan Pavlukhin
>
> чт, 14 мая 2020 г. в 10:59, Petr Ivanov :
> >
> > https://mtcga.ignite.apache.org  is
> set.
> >
> >
> > > On 12 May 2020, at 17:12, Ivan Pavlukhin  wrote:
> > >
> > > Yes, it sounds reasonable at the first place to research how much
> > > computing power we can get for free.
> > >
> > > Best regards,
> > > Ivan Pavlukhin
> > >
> > > вт, 12 мая 2020 г. в 16:36, Dmitriy Pavlov :
> > >>
> > >> Hi Igniters,
> > >>
> > >> Open clouds for TC Bot sounds really good, but only concern here if
> there
> > >> will be enough storage space and capabilities to run an Apache Ignite
> > >> instance there.
> > >>
> > >> Currently, TC Bot requires a lot of resources, e.g. its DB size
> >100Gb and
> > >> RAM allocated is approx 32Gb.
> > >>
> > >> Sincerely,
> > >> Dmitriy Pavlov
> > >>
> > >> вт, 12 мая 2020 г. в 15:35, Ivan Rakov :
> > >>
> > >>> Ivan,
> > >>>
> > >>> Agree.
> > >>> Mail notifications can be temporarily turned off in configuration of
> the
> > >>> new bot.
> > >>>
> > >>> On Tue, May 12, 2020 at 3:12 PM Ivan Pavlukhin 
> > >>> wrote:
> > >>>
> >  Having bot deployed in open/free (and reliable) infrastructure
> sounds
> >  great! One precaution which seems important to me though is
> avoidance
> >  of duplicate (or even controversial) notifications from 2 bots at
> the
> >  same time.
> > 
> >  Best regards,
> >  Ivan Pavlukhin
> > 
> >  вт, 12 мая 2020 г. в 15:06, Ivan Rakov :
> > >
> > > Hi,
> > >
> > > I've created an INFRA ticket [1] for forwarding requests from "
> > > mtcga.ignite.apache.org" to the server where TC bot is hosted [1].
> > > Definitely, I wouldn't object if anyone will deploy TC bot to the
> > >>> public
> > > cloud. We can live with two bots for a while, and then start using
> a
> >  public
> > > bot after it accumulates enough build history to grant VISAs. If
> anyone
> >  is
> > > interested, please check TC bot homepage on github with setup guide
> > >>> [2].
> > > 
> > >
> > > [1]: https://issues.apache.org/jira/browse/INFRA-20257
> > > [2]: https://github.com/apache/ignite-teamcity-bot
> > >
> > > --
> > > Best Regards,
> > > Ivan Rakov
> > >
> > > On Tue, May 12, 2020 at 12:44 PM Ilya Kasnacheev <
> >  ilya.kasnach...@gmail.com>
> > > wrote:
> > >
> > >> Hello!
> > >>
> > >> It would be nice if somebody would try to bring up a parallel
> >  deployment of
> > >> MTCGA bot on Apache domain.
> > >>
> > >> This way people will have a choice of using "old" or "new" bot,
> and
> >  they we
> > >> may decide of sticking to one of them.
> > >>
> > >> Regards,
> > >> --
> > >> Ilya Kasnacheev
> > >>
> > >>
> > >> пн, 11 мая 2020 г. в 18:37, Maxim Muzafarov :
> > >>
> > >>> Ivan,
> > >>>
> > >>>
> > >>> Good idea.
> > >>> +1 to have the right domain name.
> > >>>
> > >>> I can imagine that we can go even further and completely move
> > >>> TC.Bot
> > >>> to some public cloud storage. For example, Amazon can provide
> > >>> promotional credits for open source projects [1].
> > >>>
> > >>>
> > >>> [1]
> > >>>
> > >>
> > 
> > >>>
> https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/
> > >>>
> > >>> On Mon, 11 May 2020 at 11:35, Ivan Pavlukhin <
> vololo...@gmail.com>
> > >> wrote:
> > 
> >  Igniters,
> > 
> >  As you might know currently TC bot has a domain name in a
> > >>> GridGain
> >  domain [1]. What do you think should we assign a name in an
> > >>> Apache
> >  domain to the bot?
> > 
> >  [1] https://mtcga.gridgain.com/
> > 
> >  Best regards,
> >  Ivan Pavlukhin
> > >>>
> > >>
> > 
> > >>>
> >
>

Inconsistent behavior of IgniteMessaging

[Denis Garus]

Hello, Igniters!

IgniteMessaging has inconsistent behavior when a remote listener throws an
exception.

A remote listener throws an exception, if this listener registered on a
node that sends a message,
the sender gets this exception. But if the sender node and the node with
the remote listener aren't the same,
no one will get this exception.
I think the right behavior is to write an exception into a log and ignore
the exception.

WDYT?

The reproducer:
```

public class InconsistentBehaviorOfIgniteMessagingReproducer extends
GridCommonAbstractTest {
@Test
public void test() throws Exception {
startGrids(3);

String topic = "test_topic";

grid(0).message(grid(0).cluster().forNodeId(grid(2).localNode().id()))
.remoteListen(topic, (uuid, msg) -> {
throw new RuntimeException("Ops!");
});

try {
// This line doesn't throw an exception.
grid(1).message().send(topic, "Hello!");
}
catch (Exception e) {
fail("Shouldn't be any exception");
}

// This line throws java.lang.RuntimeException: Ops!
grid(2).message().send(topic, "Hello!");
}
}

```

[jira] [Created] (IGNITE-13011) .NET: Thin client Kubernetes discovery

[Pavel Tupitsyn (Jira)]

Pavel Tupitsyn created IGNITE-13011:
---

 Summary: .NET: Thin client Kubernetes discovery
 Key: IGNITE-13011
 URL: https://issues.apache.org/jira/browse/IGNITE-13011
 Project: Ignite
  Issue Type: New Feature
  Components: platforms
Reporter: Pavel Tupitsyn
Assignee: Pavel Tupitsyn


Thin clients should be able to discover servers from within Kubernetes pod 
through k8s API, without specifying any IP addresses.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Extensions for control.sh

[Anton Vinogradov]

Denis,

> Do you mean, that external plugins should be able to configure the
> connection that is used to communicate with a cluster?
Yes

> Could you give an example ...
The same case I mentioned before.
We may need binary rest attributes to be set.
Some code should get them somewhere (eg. from file, system property, JMV
option, from usb-devce or keyboard) and set then via
clientCfg.setUserAttributes(userAttr); inside control.sh code.

On Thu, May 14, 2020 at 1:20 PM Denis Mekhanikov 
wrote:

> Anton,
>
> Do you mean, that external plugins should be able to configure the
> connection that is used to communicate with a cluster? Could you give an
> example, what kind of plugin would benefit from it?
>
> If there are some connection-specific properties that can change the way
> how control.sh communicates with a cluster, then it makes sense to
> donate such configuration to Ignite. Or am I missing something?
>
> Denis
>
> On 14.05.2020 11:43, Anton Vinogradov wrote:
> > Denis,
> >
> > In addition to extending the features list it's also important to find
> some
> > way to allow customization of control.sh connection configuration/code.
> > For example, it may be useful to set some attributes to binary rest
> client.
> >
> > On Thu, May 14, 2020 at 2:09 AM Denis Magda  wrote:
> >
> >> Perfect idea to use this the tool for configuration and addition of
> >> extensions!
> >>
> >> -
> >> Denis
> >>
> >>
> >> On Wed, May 13, 2020 at 11:43 AM Denis Mekhanikov <
> dmekhani...@gmail.com>
> >> wrote:
> >>
> >>> Hi everyone!
> >>>
> >>> Control.sh is a command-line management tool that you can use to manage
> >>> your grid and check its vital parameters like topology version or
> >>> availability of baseline nodes. It has is good set of commands which
> are
> >>> suitable to work with vanilla Ignite.
> >>>
> >>> There is also a way to extend functionality of Ignite by implementing a
> >>> 3rd-party plugin or a module. Any plugin or external module should have
> >>> some kind of API to manage and monitor its activity.
> >>> If a command-line management command needs to be added, then the only
> >>> way to achieve that is to provide an additional script, separate from
> >>> control.sh. If you use multiple such plugins, then the set of required
> >>> tools may grow and lead to confusion, which script should be used to
> >>> configure which extension. Instead of doing that it would be convenient
> >>> for users to have ability to use the same script, but with an extended
> >>> set of options. It should make lifes of 3rd-party vendors easier.
> >>>
> >>> Currently many integrations and community-supported modules are being
> >>> moved outside of the core product:
> >>>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization
> >>> I think it makes sense to provide a possibility to configure extensions
> >>> using control.sh, since their number will grow over time, and some of
> >>> them will require some runtime configuration.
> >>>
> >>> What do you think?
> >>>
> >>> Denis
> >>>
> >>>
>

Re: Github pull requests are not linked automatically to jira tickets.

[Pavel Pereslegin]

Folks,

Now, it seems that the problem has been fixed.

вт, 12 мая 2020 г. в 13:57, Pavel Pereslegin :
>
> Ivan,
>
> unfortunately this did not help, I left a comment about it in the
> latest INFRA ticket on this subject [1].
>
> [1] https://issues.apache.org/jira/browse/INFRA-20253
>
> вт, 12 мая 2020 г. в 13:47, Ivan Pavlukhin :
> >
> > Pavel, folks,
> >
> > Does anyone have an evidence that PR linking works fine now? I noticed
> > that for a ticket [1] a PR link had not been added automatically.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-12994
> >
> > Best regards,
> > Ivan Pavlukhin
> >
> > пн, 11 мая 2020 г. в 11:55, Ivan Pavlukhin :
> > >
> > > Pavel,
> > >
> > > Thank you! I merged the PR.
> > >
> > > Let's see that PR linking to JIRA and everything else work fine.
> > >
> > > Best regards,
> > > Ivan Pavlukhin
> > >
> > > пт, 8 мая 2020 г. в 14:45, Pavel Pereslegin :
> > > >
> > > > Igniters,
> > > >
> > > > ticket created [1], please take a look at the proposed changes [2].
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-12993
> > > > [2] https://github.com/apache/ignite/pull/7784/files
> > > >
> > > > пт, 8 мая 2020 г. в 12:02, Maxim Muzafarov :
> > > > >
> > > > > Pavel,
> > > > >
> > > > >
> > > > > I have no objections. Let's file a ticket.
> > > > >
> > > > > On Fri, 8 May 2020 at 08:04, Ivan Pavlukhin  
> > > > > wrote:
> > > > > >
> > > > > > Pavel,
> > > > > >
> > > > > > Thank you for starting this discussion!
> > > > > >
> > > > > > Controlling all this repo options on our side sounds very 
> > > > > > attractive.
> > > > > >
> > > > > > Personally I do not have any arguments about the proposal.
> > > > > >
> > > > > > Best regards,
> > > > > > Ivan Pavlukhin
> > > > > >
> > > > > > чт, 7 мая 2020 г. в 17:39, Pavel Pereslegin :
> > > > > > >
> > > > > > > Igniters,
> > > > > > >
> > > > > > > recent github pull requests are not automatically linked to the 
> > > > > > > jira
> > > > > > > tickets.
> > > > > > >
> > > > > > > Infra advises creating a yaml configuration in the root of the 
> > > > > > > repository
> > > > > > > with the settings for the github bot [1].
> > > > > > > Examples of such configuration in Hive [2] and HBase [3].
> > > > > > > I suggest the following settings for Ignite [4].
> > > > > > >
> > > > > > > If there are no objections, I'll file a ticket for this.
> > > > > > >
> > > > > > > [1] https://issues.apache.org/jira/browse/INFRA-20177
> > > > > > > [2] https://github.com/apache/hbase/blob/master/.asf.yaml
> > > > > > > [3] https://github.com/apache/hive/blob/master/.asf.yaml
> > > > > > > [4] https://gist.github.com/xtern/572cbc7d4a7916f6e933fbafe5034492

Re: Extensions for control.sh

[Denis Mekhanikov]

Anton,

Do you mean, that external plugins should be able to configure the 
connection that is used to communicate with a cluster? Could you give an 
example, what kind of plugin would benefit from it?


If there are some connection-specific properties that can change the way 
how control.sh communicates with a cluster, then it makes sense to 
donate such configuration to Ignite. Or am I missing something?


Denis

On 14.05.2020 11:43, Anton Vinogradov wrote:

Denis,

In addition to extending the features list it's also important to find some
way to allow customization of control.sh connection configuration/code.
For example, it may be useful to set some attributes to binary rest client.

On Thu, May 14, 2020 at 2:09 AM Denis Magda  wrote:


Perfect idea to use this the tool for configuration and addition of
extensions!

-
Denis


On Wed, May 13, 2020 at 11:43 AM Denis Mekhanikov 
wrote:


Hi everyone!

Control.sh is a command-line management tool that you can use to manage
your grid and check its vital parameters like topology version or
availability of baseline nodes. It has is good set of commands which are
suitable to work with vanilla Ignite.

There is also a way to extend functionality of Ignite by implementing a
3rd-party plugin or a module. Any plugin or external module should have
some kind of API to manage and monitor its activity.
If a command-line management command needs to be added, then the only
way to achieve that is to provide an additional script, separate from
control.sh. If you use multiple such plugins, then the set of required
tools may grow and lead to confusion, which script should be used to
configure which extension. Instead of doing that it would be convenient
for users to have ability to use the same script, but with an extended
set of options. It should make lifes of 3rd-party vendors easier.

Currently many integrations and community-supported modules are being
moved outside of the core product:


https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization

I think it makes sense to provide a possibility to configure extensions
using control.sh, since their number will grow over time, and some of
them will require some runtime configuration.

What do you think?

Denis

[jira] [Created] (IGNITE-13010) A local listener for cache events with type EVT_CACHE_STOPPED does not get a cache event from a remote node.

[Denis Garus (Jira)]

Denis Garus created IGNITE-13010:


 Summary: A local listener for cache events with type 
EVT_CACHE_STOPPED does not get a cache event from a remote node.
 Key: IGNITE-13010
 URL: https://issues.apache.org/jira/browse/IGNITE-13010
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.8
Reporter: Denis Garus


A local listener for cache events with type EVT_CACHE_STOPPED does not get a 
cache event from a remote node. 
That occurs due to NPE on a remote node:
{code:java}
[2020-05-14 
12:07:25,623][ERROR][sys-#206%security.NpeGridEventConsumeHandlerReproducer2%][GridEventConsumeHandler]
 Failed to send event notification to node: 
55671ec1-dad9-452b-8ab2-4b7916c0[2020-05-14 
12:07:25,623][ERROR][sys-#206%security.NpeGridEventConsumeHandlerReproducer2%][GridEventConsumeHandler]
 Failed to send event notification to node: 
55671ec1-dad9-452b-8ab2-4b7916c0java.lang.NullPointerException at 
org.apache.ignite.internal.GridEventConsumeHandler$2$1.run(GridEventConsumeHandler.java:238)
 at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
 at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 at java.base/java.lang.Thread.run(Thread.java:834)
{code}
The reproducer:


{code:java}
public class NpeGridEventConsumeHandlerReproducer extends 
GridCommonAbstractTest {

private static AtomicInteger rmtCounter = new AtomicInteger();
private static AtomicInteger locCounter = new AtomicInteger();

@Override protected IgniteConfiguration getConfiguration(String 
igniteInstanceName) throws Exception {
return 
super.getConfiguration(igniteInstanceName).setIncludeEventTypes(EVT_CACHE_STOPPED);
}

@Test
public void test() throws Exception {
startGrids(3);

grid(1).createCache(new CacheConfiguration<>("test_cache"));

grid(0).events().remoteListen((uuid, evt) ->{
 locCounter.incrementAndGet();
 return true;
}, evt->{
rmtCounter.incrementAndGet();
return true;
}, EVT_CACHE_STOPPED);

grid(1).destroyCache("test_cache");

TimeUnit.SECONDS.sleep(10);

assertEquals(rmtCounter.get(), locCounter.get());
}
}
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Extensions for control.sh

[Anton Vinogradov]

Denis,

In addition to extending the features list it's also important to find some
way to allow customization of control.sh connection configuration/code.
For example, it may be useful to set some attributes to binary rest client.

On Thu, May 14, 2020 at 2:09 AM Denis Magda  wrote:

> Perfect idea to use this the tool for configuration and addition of
> extensions!
>
> -
> Denis
>
>
> On Wed, May 13, 2020 at 11:43 AM Denis Mekhanikov 
> wrote:
>
> > Hi everyone!
> >
> > Control.sh is a command-line management tool that you can use to manage
> > your grid and check its vital parameters like topology version or
> > availability of baseline nodes. It has is good set of commands which are
> > suitable to work with vanilla Ignite.
> >
> > There is also a way to extend functionality of Ignite by implementing a
> > 3rd-party plugin or a module. Any plugin or external module should have
> > some kind of API to manage and monitor its activity.
> > If a command-line management command needs to be added, then the only
> > way to achieve that is to provide an additional script, separate from
> > control.sh. If you use multiple such plugins, then the set of required
> > tools may grow and lead to confusion, which script should be used to
> > configure which extension. Instead of doing that it would be convenient
> > for users to have ability to use the same script, but with an extended
> > set of options. It should make lifes of 3rd-party vendors easier.
> >
> > Currently many integrations and community-supported modules are being
> > moved outside of the core product:
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization
> > I think it makes sense to provide a possibility to configure extensions
> > using control.sh, since their number will grow over time, and some of
> > them will require some runtime configuration.
> >
> > What do you think?
> >
> > Denis
> >
> >
>

Re: [DISCUSS] Best way to re-encrypt existing data (TDE cache key rotation).

[Anton Vinogradov]

+1 to "In place re-encryption".

- It has a simple design.
- Clusters under load may require just load to re-encrypt the data.
(Friendly to load).
- Easy to throttle.
- Easy to continue.
- Design compatible with the multi-key architecture.
- It can be optimized to use own WAL buffer and to re-encrypt pages without
restoring them to on-heap.

On Thu, May 14, 2020 at 1:54 AM Pavel Pereslegin  wrote:

> Hello Igniters.
>
> Recently, master key rotation for Apache Ignite Transparent Data
> Encryption was implemented [1], but some security standards (PCI DSS
> at least) require rotation of all encryption keys [2]. Currently,
> encryption occurs when reading/writing pages to disk, cache encryption
> keys are stored in metastore.
>
> I'm going to contribute cache encryption key rotation and want to
> consult what is the best way to re-encrypting existing data, I see two
> different strategies.
>
> 1. In place re-encryption:
> Using the old key, sequentially read all the pages from the datastore,
> mark as dirty and log them into the WAL. After checkpoint pages will
> be stored to disk encrypted with the new key (as usual, along with
> updates). This strategy requires store the identifier (number) of the
> encryption key into the encrypted page.
> pros:
>   - can work in the background with minimal performance impact (this
> impact can be managed).
> cons:
>   - page duplication in the WAL may affect performance and historical
> rebalance.
>
> 2. Copy partition with re-encryption.
> This strategy is similar to partition snapshotting [3] - create
> partition copy encrypted with the new key and then replace the
> original partition file with the new one (see details [4]).
> pros:
>   - should work faster than "in place" re-encryption.
> cons:
>   - re-encryption in active cluster (and on unstable topology) can be
> difficult to implement.
>
> (See more detailed comparison [5])
>
> Re-encryption of existing data is a long and rare procedure (It is
> recommended to change the key every 6 months, but at least once every
> 2 years). Thus, re-encryption can be implemented for maintenance mode
> (for example, on a stable topology in a read-only cluster) and in such
> case the approach with partition copying seems simpler and faster.
>
> So, what do you think - do we need "online" re-encryption and which of
> the proposed options is best suited for this?
>
> [1] https://issues.apache.org/jira/browse/IGNITE-12186
> [2] https://www.pcisecuritystandards.org/documents/PCI_DSS_v3-2-1.pdf
> [3]
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-43%3A+Cluster+snapshots#IEP-43:Clustersnapshots-Partitionscopystrategy
> [4]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652384#TDE.Phase-3.Cachekeyrotation.-Copywithre-encryptiondesign
> .
> [5]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652384#TDE.Phase-3.Cachekeyrotation.-Comparison
>

Re: [DISCUSS] Apache URL for TC bot

[Ivan Pavlukhin]

Petr, Ivan,

Good stuff! Thank you!

Best regards,
Ivan Pavlukhin

чт, 14 мая 2020 г. в 10:59, Petr Ivanov :
>
> https://mtcga.ignite.apache.org  is set.
>
>
> > On 12 May 2020, at 17:12, Ivan Pavlukhin  wrote:
> >
> > Yes, it sounds reasonable at the first place to research how much
> > computing power we can get for free.
> >
> > Best regards,
> > Ivan Pavlukhin
> >
> > вт, 12 мая 2020 г. в 16:36, Dmitriy Pavlov :
> >>
> >> Hi Igniters,
> >>
> >> Open clouds for TC Bot sounds really good, but only concern here if there
> >> will be enough storage space and capabilities to run an Apache Ignite
> >> instance there.
> >>
> >> Currently, TC Bot requires a lot of resources, e.g. its DB size >100Gb and
> >> RAM allocated is approx 32Gb.
> >>
> >> Sincerely,
> >> Dmitriy Pavlov
> >>
> >> вт, 12 мая 2020 г. в 15:35, Ivan Rakov :
> >>
> >>> Ivan,
> >>>
> >>> Agree.
> >>> Mail notifications can be temporarily turned off in configuration of the
> >>> new bot.
> >>>
> >>> On Tue, May 12, 2020 at 3:12 PM Ivan Pavlukhin 
> >>> wrote:
> >>>
>  Having bot deployed in open/free (and reliable) infrastructure sounds
>  great! One precaution which seems important to me though is avoidance
>  of duplicate (or even controversial) notifications from 2 bots at the
>  same time.
> 
>  Best regards,
>  Ivan Pavlukhin
> 
>  вт, 12 мая 2020 г. в 15:06, Ivan Rakov :
> >
> > Hi,
> >
> > I've created an INFRA ticket [1] for forwarding requests from "
> > mtcga.ignite.apache.org" to the server where TC bot is hosted [1].
> > Definitely, I wouldn't object if anyone will deploy TC bot to the
> >>> public
> > cloud. We can live with two bots for a while, and then start using a
>  public
> > bot after it accumulates enough build history to grant VISAs. If anyone
>  is
> > interested, please check TC bot homepage on github with setup guide
> >>> [2].
> > 
> >
> > [1]: https://issues.apache.org/jira/browse/INFRA-20257
> > [2]: https://github.com/apache/ignite-teamcity-bot
> >
> > --
> > Best Regards,
> > Ivan Rakov
> >
> > On Tue, May 12, 2020 at 12:44 PM Ilya Kasnacheev <
>  ilya.kasnach...@gmail.com>
> > wrote:
> >
> >> Hello!
> >>
> >> It would be nice if somebody would try to bring up a parallel
>  deployment of
> >> MTCGA bot on Apache domain.
> >>
> >> This way people will have a choice of using "old" or "new" bot, and
>  they we
> >> may decide of sticking to one of them.
> >>
> >> Regards,
> >> --
> >> Ilya Kasnacheev
> >>
> >>
> >> пн, 11 мая 2020 г. в 18:37, Maxim Muzafarov :
> >>
> >>> Ivan,
> >>>
> >>>
> >>> Good idea.
> >>> +1 to have the right domain name.
> >>>
> >>> I can imagine that we can go even further and completely move
> >>> TC.Bot
> >>> to some public cloud storage. For example, Amazon can provide
> >>> promotional credits for open source projects [1].
> >>>
> >>>
> >>> [1]
> >>>
> >>
> 
> >>> https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/
> >>>
> >>> On Mon, 11 May 2020 at 11:35, Ivan Pavlukhin 
> >> wrote:
> 
>  Igniters,
> 
>  As you might know currently TC bot has a domain name in a
> >>> GridGain
>  domain [1]. What do you think should we assign a name in an
> >>> Apache
>  domain to the bot?
> 
>  [1] https://mtcga.gridgain.com/
> 
>  Best regards,
>  Ivan Pavlukhin
> >>>
> >>
> 
> >>>
>

Re: Apache Ignite 2.8.1 RELEASE [Time, Scope, Manager]

[Юрий]

Nikolay,

Release 2.8.1 are delayed and announced dates [1] at release page is not
actual. Could you update it to reflect current vision of release date?

[1]. https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8.1

чт, 7 мая 2020 г. в 17:23, Nikolay Izhikov :

> Done.
>
>
> > 7 мая 2020 г., в 13:20, Denis Garus  написал(а):
> >
> > Nikolay,
> > could we add the simple improvement [1] to 2.8.1 scope?
> >
> > 1. https://issues.apache.org/jira/browse/IGNITE-12983
> >
> > чт, 30 апр. 2020 г. в 11:26, Alex Plehanov :
> >
> >> Nikolay,
> >>
> >> TC results: [1], [2]
> >> I've cherry-picked IGNITE-12933 and IGNITE-12855 to 2.8.1
> >>
> >> [1]:
> >>
> >>
> https://mtcga.gridgain.com/pr.html?serverId=apache=IgniteTests24Java8_RunAll=pull%2F7754%2Fhead=Latest=ignite-2.8.1
> >> [2]:
> >>
> >>
> https://mtcga.gridgain.com/pr.html?serverId=apache=IgniteTests24Java8_RunAll=pull%2F7755%2Fhead=Latest=ignite-2.8.1
> >>
> >> вт, 28 апр. 2020 г. в 15:19, Nikolay Izhikov :
> >>
> >>> Hello, Alex.
> >>>
> >>> +1 from me.
> >>>
>  28 апр. 2020 г., в 15:03, Alex Plehanov 
> >>> написал(а):
> 
>  Hello guys,
> 
>  While we are still waiting for some tickets to resolve I propose to
>  cherry-pick to 2.8.1 two more bugfixes:
>  IGNITE-12933 Fixed node failure after put incorrect key class for
> cache
>  with indexed types
>  IGNITE-12855 Fixed node failure with concurrent get operation and
> entry
>  expiration when persistent is enabled
>  Both fixes prevent node failure in some circumstances, both fixed
> >> already
>  merged to master.
> 
>  WDYT?
> 
>  пн, 27 апр. 2020 г. в 11:53, Nikolay Izhikov :
> 
> > Taras.
> >
> > Thank you, very much!
> > You changes merged to 2.8.1 branch.
> >
> > Igniters,
> >
> > We have 10 tickets scheduled for 2.8.1 release:
> >
> > OPEN:
> >
> > IGNITE-11687Concurrent WAL replay & log may fail with CRC error
> on
> > read - Dmitriy Govorukhin
> > IGNITE-12346.NET: Platform error:System.NullReferenceException -
> > Pavel Tupitsyn
> >
> > IN PROGRESS:
> >
> > IGNITE-12637IgniteSparkSession doesn't start the clients on
> really
> > distributed cluster - Yaroslav Molochkov
> > IGNITE-12788Cluster achieved fully rebalanced (PME-free ready)
> >> state
> > metric - Nikolay Izhikov
> >
> > PATCH AVAILABLE:
> >
> > IGNITE-10417notifyDiscoveryListener() call can be lost. - Pavel
> > Voronkin
> > IGNITE-12852Comma in field is not supported by COPY command -
> >> YuJue
> >>> Li
> > IGNITE-12252Unchecked exceptions during rebalancing should be
> >>> handled
> > - Nikolai Kulagin
> > IGNITE-12905QueryKeyValueIterable missing custom spliterator()
> > implementation - Johnny Galatikitis
> > IGNITE-12801Possible extra page release when throttling and
> >>> checkpoint
> > thread store its concurrently - Anton Kalashnikov
> > IGNITE-12794Scan query fails with an assertion error: Unexpected
> >> row
> > key - Denis Mekhanikov
> >
> >
> >> 27 апр. 2020 г., в 11:08, Taras Ledkov 
> > написал(а):
> >>
> >> Hi,
> >>
> >> Nikolay, i've created PR [1] that contains the SQL-related tickets
> to
> > port into 2.8.1:
> >>
> >> IGNITE-12790 Introduce distributed SQL configuration and ability to
> > disable SQL functions.
> >> IGNITE-12887 Fix handle type mismatch exception on compare values
> >> while
> > traversing index tree.
> >> IGNITE-12848 fix H2Connection leaks on INSERT
> >> IGNITE-12800  SQL: local queries cursors must be closed or full read
> >> to
> > unlock the GridH2Table.
> >>
> >> TC test are OK. Please take a look at the TC bot report [2].
> >> Please review & merge the patch into ignite-2.8.1.
> >>
> >> [1]. https://github.com/apache/ignite/pull/7703
> >> [2].
> >
> >>>
> >>
> https://mtcga.gridgain.com/pr.html?serverId=apache=IgniteTests24Java8_RunAll=pull%2F7703%2Fhead=Latest=ignite-2.8.1
> >>
> >>
> >> On 23.04.2020 13:25, Mikhail Petrov wrote:
> >>> Hello, Igniters.
> >>>
> >>> I propose to cherry-pick to 2.8.1 ticket [1].
> >>>
> >>>
> >>> In addition to adding a new metric, it fixes a bug when, after
> > deactivation, GridDhtPartitionsExchangeFuture#rebalanced flag was not
> > reset. And therefore, it can be different on nodes that are already
> in
> >>> the
> > cluster from newly joined ones.
> >>>
> >>> [1] - https://issues.apache.org/jira/browse/IGNITE-12788
> >>>
> >>> Regards,
> >>> Mikhail.
> >>>
> >>> On 22.04.2020 14:03, Nikolay Izhikov wrote:
>  Hello, Ivan.
> 
>  I think we can include this improvements.
>  Please, go ahead.
> 
> > 22 апр. 2020 г., в 10:21, Ivan Bessonov 
> > написал(а):
> >
> > 

Re: [DISCUSS] Apache URL for TC bot

[Petr Ivanov]

https://mtcga.ignite.apache.org  is set.


> On 12 May 2020, at 17:12, Ivan Pavlukhin  wrote:
> 
> Yes, it sounds reasonable at the first place to research how much
> computing power we can get for free.
> 
> Best regards,
> Ivan Pavlukhin
> 
> вт, 12 мая 2020 г. в 16:36, Dmitriy Pavlov :
>> 
>> Hi Igniters,
>> 
>> Open clouds for TC Bot sounds really good, but only concern here if there
>> will be enough storage space and capabilities to run an Apache Ignite
>> instance there.
>> 
>> Currently, TC Bot requires a lot of resources, e.g. its DB size >100Gb and
>> RAM allocated is approx 32Gb.
>> 
>> Sincerely,
>> Dmitriy Pavlov
>> 
>> вт, 12 мая 2020 г. в 15:35, Ivan Rakov :
>> 
>>> Ivan,
>>> 
>>> Agree.
>>> Mail notifications can be temporarily turned off in configuration of the
>>> new bot.
>>> 
>>> On Tue, May 12, 2020 at 3:12 PM Ivan Pavlukhin 
>>> wrote:
>>> 
 Having bot deployed in open/free (and reliable) infrastructure sounds
 great! One precaution which seems important to me though is avoidance
 of duplicate (or even controversial) notifications from 2 bots at the
 same time.
 
 Best regards,
 Ivan Pavlukhin
 
 вт, 12 мая 2020 г. в 15:06, Ivan Rakov :
> 
> Hi,
> 
> I've created an INFRA ticket [1] for forwarding requests from "
> mtcga.ignite.apache.org" to the server where TC bot is hosted [1].
> Definitely, I wouldn't object if anyone will deploy TC bot to the
>>> public
> cloud. We can live with two bots for a while, and then start using a
 public
> bot after it accumulates enough build history to grant VISAs. If anyone
 is
> interested, please check TC bot homepage on github with setup guide
>>> [2].
> 
> 
> [1]: https://issues.apache.org/jira/browse/INFRA-20257
> [2]: https://github.com/apache/ignite-teamcity-bot
> 
> --
> Best Regards,
> Ivan Rakov
> 
> On Tue, May 12, 2020 at 12:44 PM Ilya Kasnacheev <
 ilya.kasnach...@gmail.com>
> wrote:
> 
>> Hello!
>> 
>> It would be nice if somebody would try to bring up a parallel
 deployment of
>> MTCGA bot on Apache domain.
>> 
>> This way people will have a choice of using "old" or "new" bot, and
 they we
>> may decide of sticking to one of them.
>> 
>> Regards,
>> --
>> Ilya Kasnacheev
>> 
>> 
>> пн, 11 мая 2020 г. в 18:37, Maxim Muzafarov :
>> 
>>> Ivan,
>>> 
>>> 
>>> Good idea.
>>> +1 to have the right domain name.
>>> 
>>> I can imagine that we can go even further and completely move
>>> TC.Bot
>>> to some public cloud storage. For example, Amazon can provide
>>> promotional credits for open source projects [1].
>>> 
>>> 
>>> [1]
>>> 
>> 
 
>>> https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/
>>> 
>>> On Mon, 11 May 2020 at 11:35, Ivan Pavlukhin 
>> wrote:
 
 Igniters,
 
 As you might know currently TC bot has a domain name in a
>>> GridGain
 domain [1]. What do you think should we assign a name in an
>>> Apache
 domain to the bot?
 
 [1] https://mtcga.gridgain.com/
 
 Best regards,
 Ivan Pavlukhin
>>> 
>> 
 
>>> 

Re: Crash recovery speed-up #3, Cellular Switch

[Anton Vinogradov]

Folks,

It seems, we have tacit agreement here.
Going to merge fix May 15.

On Tue, May 12, 2020 at 10:08 AM Anton Vinogradov  wrote:

> Denis,
>
> Rebalance is not expected here since this optimization works only on a
> fully rebalanced cluster with baseline.
>
> On Sat, May 9, 2020 at 12:48 AM Denis Magda  wrote:
>
>> Hi Anton,
>>
>> Generally, it means that Ignite will keep executing
>> operations/transactions
>> that are mapped into the partitions of those cells that won't be
>> rebalanced, is that correct?
>>
>> -
>> Denis
>>
>>
>> On Wed, May 6, 2020 at 3:24 AM Anton Vinogradov  wrote:
>>
>> > Igniters,
>> >
>> > PME-free switch [1] (since 2.8) skips PME on node left when possible
>> > (baseline + fully rebalanced cluster).
>> > This means we already wait for nothing (except recovery) to perform the
>> > switch.
>> > This optimization allows continuing already started operations during or
>> > after the switch if they are not affected by failed primary.
>> > But upcoming operations still can't be started until the switch is
>> finished
>> > cluster-wide.
>> >
>> > Let me propose an additional optimization - Cellular switch.
>> > Cellular Affinity [2] means that nodes combined into virtual cells
>> where,
>> > for each partition, backups located at the same cell with primaries.
>> > The simplest way to gain Cellular Affinity is to use backup filters [3].
>> >
>> > Cellular Affinity allows to finish the switch outside the affected cell
>> > instantly with the following assumptions:
>> > - Replicated caches should be recovered first since every node affected
>> (as
>> > a backup) by any failed primary.
>> >   But, it is expected that replicated caches effectively read-only (has
>> > extremely rare updates), so, nothing to wait here.
>> > - Upcoming replicated transactions (with non-failed primaries) can be
>> > started but can't be committed until switch finished cluster-wide.
>> > - Upcoming transactions related to the broken cell will wait for cell
>> > recovery (cluster-wide switch finish).
>> >
>> > ... and this means:
>> > In addition to PME-free switch, where we able to continue already
>> started
>> > operations during or after the switch, now we also able to perform most
>> of
>> > the upcoming operations during the switch.
>> >
>> > In other words, Cellular switch has little effect on the operation's
>> > latency, when operation not related to the failed cell.
>> >
>> > According to benchmark [4] which checks "how fast upcoming transactions
>> > (started after switch start) can be committed when we have thousands of
>> > prepared transactions (prepared before switch start)", we have 5326 ms
>> [5]
>> > operation's latency on master and 65 ms [6] with the proposed fix,
>> which is
>> > ~100 times faster.
>> >
>> > Fix [7] (as a part of IEP-45 [8]) ready to be reviewed.
>> > Waiting for your review!
>> >
>> >
>> > [1]
>> >
>> >
>> http://apache-ignite-developers.2346864.n4.nabble.com/Non-blocking-PME-Phase-One-Node-fail-tp43531p44586.html
>> > [2]
>> >
>> >
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up#IEP-45:CrashRecoverySpeed-Up-Cellularswitch
>> > [3]
>> >
>> >
>> https://gist.github.com/anton-vinogradov/c50f9d0ce3e3e2997646f84ba7eba5f5#file-bench-java-L417
>> > [4]
>> >
>> https://gist.github.com/anton-vinogradov/c50f9d0ce3e3e2997646f84ba7eba5f5
>> > [5]
>> >
>> >
>> https://gist.github.com/anton-vinogradov/a35a3a8151b7494aa84b83f58cb75889#file-master-txt-L15
>> > [6]
>> >
>> >
>> https://gist.github.com/anton-vinogradov/a35a3a8151b7494aa84b83f58cb75889#file-fix-txt-L15
>> > [7] https://issues.apache.org/jira/browse/IGNITE-12617
>> > [8]
>> >
>> >
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up
>> >
>>
>