Re: Thin Client Kubernetes discovery

[Denis Magda]

Max,

That improvement automates the cluster discovery if both the cluster and
thin clients are deployed *inside* of Kubernetes. That's my reading of the
ticket's description. While you're referring to the case when the client is
deployed outside of a K8S environment.

@Vladimir Pligin , could you please join the thread.
I've seen you joined the discussion in JIRA.

-
Denis


On Wed, Aug 12, 2020 at 12:06 PM Max Timonin 
wrote:

> Hi, Igniters
>
> I'm now discovering the issue
> https://issues.apache.org/jira/browse/IGNITE-13204 and wondering what is a
> case that requires a client to be able to discover a cluster?
>
> I believe that discovery is a goal of kubernetes itself. We could assign a
> DNS name for a Service [1] and then any request to the Service will be
> forwarded to any of Ignite pods behind it. So there is no need to extract
> pods IP to provide the connection. For example, I've configured a  k8s
> service with name "ignite" and port mapping 10800:9888. Then just connect
> to it with sqlline by a string "ignite:9888".
>
> I think that it's OK that clients should be configured with those settings
> as they aren't changed during cluster live.
>
> Also k8s provides SessionAffinity [2] to guarantee forwarding traffic from
> one client to the same pod for every request. So there is no need to
> provide IP for a specific pod if a state is required.
>
> Maybe I'm missing something about this feature? Could somebody provide more
> details for this task?
>
> Below is the example config and connection string:
>
> apiVersion: v1
> kind: Service
> metadata:
>   name: ignite
> spec:
>   ports:
> - port: 9888
>   targetPort: 10800
>   selector:
> app: ignite
>
> ./bin/sqlline.sh --verbose=true -u jdbc:ignite:thin://ignite:9888
> 0: jdbc:ignite:thin://ignite:9888>
>
> [1] https://kubernetes.io/docs/concepts/services-networking/service/
> [2]
>
> https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-userspace
>

Re: Deploying an app to an ignite cache cluster

[emmathornber]

Apache Ignite allows you to store the most frequently accessed data in
memory. It evenly distributes the data across a cluster of computers in
either partitioned or replicated manner. Ignite allows you to access the
data from any underlying data store – RDBMS, NoSQL, or HDFS. you can
dynamically add nodes to the Ignite cluster without restarting the entire
cluster. Ignite has virtually unlimited scale.



-
Developer at Cokernutx 
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Thin Client Kubernetes discovery

[Max Timonin]

Hi, Igniters

I'm now discovering the issue
https://issues.apache.org/jira/browse/IGNITE-13204 and wondering what is a
case that requires a client to be able to discover a cluster?

I believe that discovery is a goal of kubernetes itself. We could assign a
DNS name for a Service [1] and then any request to the Service will be
forwarded to any of Ignite pods behind it. So there is no need to extract
pods IP to provide the connection. For example, I've configured a  k8s
service with name "ignite" and port mapping 10800:9888. Then just connect
to it with sqlline by a string "ignite:9888".

I think that it's OK that clients should be configured with those settings
as they aren't changed during cluster live.

Also k8s provides SessionAffinity [2] to guarantee forwarding traffic from
one client to the same pod for every request. So there is no need to
provide IP for a specific pod if a state is required.

Maybe I'm missing something about this feature? Could somebody provide more
details for this task?

Below is the example config and connection string:

apiVersion: v1
kind: Service
metadata:
  name: ignite
spec:
  ports:
- port: 9888
  targetPort: 10800
  selector:
app: ignite

./bin/sqlline.sh --verbose=true -u jdbc:ignite:thin://ignite:9888
0: jdbc:ignite:thin://ignite:9888>

[1] https://kubernetes.io/docs/concepts/services-networking/service/
[2]
https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-userspace

Re: Apache Ignite 3.0

[Valentin Kulichenko]

Hi Petr,

Got it, that makes sense. I think we should rely on Maven for this.
Basically, like the mvnw wrapper, but specific for Ignite purposes
(enable/disable a module, start/stop a node, etc.). BTW, do you think
RPM/DEB packages can use the same approach? Or they require that binaries
are stored in specific repositories?

-Val

On Tue, Aug 11, 2020 at 11:48 PM Petr Ivanov  wrote:

> Hi, Val.
>
> On 12 Aug 2020, at 01:31, Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
> Hi Petr,
>
> I agree -- we should better modularize the platform. The current way if
> very error-prone, especially during upgrades -- any changes made within
> IGNITE_HOME (configs, scripts, modules, etc.) must be merged with a new
> version of the package. There is no standard way of doing this.
>
> However, I'm a bit concerned with your suggestion regarding custom
> dependency management. Can you please elaborate on how you think it should
> work? Are there tools we can reuse for this purpose? I would try to avoid
> reinventing the wheel.
>
>
> I see it as a a2enmod | 2dismod analog of Apache2.
>
> We build and store Apache Ignite and its modules as separate binaries
> (binary per module) then use custom script that will know where to download
> necessary module. Or possibly use modified ignite.sh to specify required
> optional libs in run command while ignite.sh will download everything
> missing from known storage.
>
> The whole idea is in storing everything remotely and download on demand,
> not have all libs locally from the start.
>
>
>
> -Val
>
> On Sun, Aug 9, 2020 at 11:25 PM Petr Ivanov  wrote:
>
>> Hi, Val!
>> Thanks for your efforts on this endeavour!
>>
>>
>> I would like to suggest deliveries changes in Apache Ignite 3.0:
>>  — modularised  binary delivery — single minimal binary for starting
>> Ignite and all other modules and parts of the project (benchmarks,
>> examples, etc.) packed in their own binary which can be added via custom
>> dependency management tool (i.e. modules.sh)
>>  — same distribution for RPM and DEB packages but with modules packed as
>> separate ones (PHP for example)
>>  — separate thin client release cycle with custom versioning
>> Possibly, we can we add additional section to the document you introduced
>> for this part.
>>
>> Also, it seems that full JDK11 support (including building) would be a
>> huge milestone and a sign of healthy modern project that tends to be on the
>> verge of mainstream technologies and not the stockpile of legacy leftovers
>> (fully support Iliya in removing all that was deprecated and/or marked as
>> unused anymore).
>>
>>
>> > On 8 Aug 2020, at 02:00, Valentin Kulichenko <
>> valentin.kuliche...@gmail.com> wrote:
>> >
>> > Igniters,
>> >
>> > I've created the page:
>> > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0
>> >
>> > That's not everything I have in mind, but I believe there is already a
>> lot
>> > to talk about :)
>> >
>> > Please take a look let me know if you have any concerns, objections, or
>> > questions. Once we reach the consensus on the proposed changes, I will
>> > start creating tickets in Jira and a more detailed plan.
>> >
>> > -Val
>> >
>> > On Thu, Aug 6, 2020 at 6:28 PM Saikat Maitra 
>> > wrote:
>> >
>> >> Hi Denis, Val
>> >>
>> >> Thank you for your reply and really appreciate it. It will be very
>> cool to
>> >> be able to connect and plan release together and learn more about
>> Ignite in
>> >> the process :)
>> >>
>> >> Regards
>> >> Saikat
>> >>
>> >>
>> >>
>> >> On Thu, Aug 6, 2020 at 7:12 PM Valentin Kulichenko <
>> >> valentin.kuliche...@gmail.com> wrote:
>> >>
>> >>> Hi Saikat,
>> >>>
>> >>> That surely is a great idea. We will work together with Denis on
>> setting
>> >>> this up in the nearest future.
>> >>>
>> >>> -Val
>> >>>
>> >>> On Thu, Aug 6, 2020 at 10:21 AM Denis Magda 
>> wrote:
>> >>>
>>  Saikat,
>> 
>>  Fully support your idea on a virtual meetup! Once Val collects and
>> >>> outlines
>>  the main changes with directions on wiki, we’ll go ahead and schedule
>> >> the
>>  meetup to talk things out in a bit more detail. We’ll use our new
>> >> Virtual
>>  Ignite Meetup group for that inviting both Ignite contributors and
>>  application developers.
>> 
>>  Denis
>> 
>>  On Thursday, August 6, 2020, Saikat Maitra 
>>  wrote:
>> 
>> > Hi Valentin
>> >
>> > Thank you for sharing and starting the thread. I am thinking if it
>> >> will
>>  be
>> > a good idea to have a virtual meet setup to discuss on the release
>> > planning.
>> >
>> > It will help to learn more individual features to be added and also
>> >> to
>> > understand about features that have been deprecated and scheduled
>> for
>> > removal in Ignite 3.0 release. Also it will help community member to
>> > connect in real time and ask questions and share feedback.
>> >
>> > Regards,
>> > Saikat
>> >
>> > On Thu, 

[jira] [Created] (IGNITE-13354) Add ClusterMetrics to the new framework

[Nikolay Izhikov (Jira)]

Nikolay Izhikov created IGNITE-13354:


 Summary: Add ClusterMetrics to the new framework
 Key: IGNITE-13354
 URL: https://issues.apache.org/jira/browse/IGNITE-13354
 Project: Ignite
  Issue Type: Improvement
Reporter: Nikolay Izhikov
Assignee: Nikolay Izhikov


We need to provide to the user information about cluster topology such as:

* TopologyVersion
* TotalNodes
* TotalBaselineNodes
* TotalServerNodes
* TotalClientNodes
* ActiveBaselineNodes




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Cpp thin client transactions support ready for review.

[Zhenya Stanilovsky]

 
Igniters, seems i complete with transactions in thin cpp client implementation 
[1], part of iep-34 [2].
Can anyone review my decision ?
Failed test seems not mine, looks like after fresh master rebase it will gone.
 
[1]  https://issues.apache.org/jira/browse/IGNITE-13308
[2] 
https://cwiki.apache.org/confluence/display/IGNITE/IEP-34+Thin+client%3A+transactions+support
 
 
 

Re[2]: new connection event

[Zhenya Stanilovsky]

may be jmx would be enough here ?

 
>Hello!
>
>Why not the Failure Handler then?
>
>(I'm only half-joking).
>
>Regards,
>
>--
>Ilya Kasnacheev
>
>
>ср, 12 авг. 2020 г. в 09:54, Oleg Ostanin < oleg.alex.osta...@gmail.com >:
> 
>> Thank you for the response. Yes, we have a simple warning in log, but in
>> case SSL error needs an immediate human attention it would be better to set
>> a listener for such events.
>>
>> вт, 11 авг. 2020 г. в 17:59, Ilya Kasnacheev < ilya.kasnach...@gmail.com >:
>>
>> > Hello!
>> >
>> > I'm not sure this is a good event. Events are computer-processable, while
>> > SSL error either needs attention from a human or none at all.
>> >
>> > Then again, sometimes it's not an SSL error (come to think or it,
>> > unauthorized people tend to not use SSL since success is not an option
>> > here) but any unexpected input. Such input will mean that something (or
>> > somebody) wrong is trying to connect, possibly using a wrong port or
>> hoping
>> > that SSL is disabled.
>> >
>> > Maybe we need a simple warning. I guess we already have it, actually.
>> Don't
>> > we?
>> >
>> > Regards,
>> > --
>> > Ilya Kasnacheev
>> >
>> >
>> > вт, 11 авг. 2020 г. в 14:27, Oleg Ostanin < oleg.alex.osta...@gmail.com >:
>> >
>> > > Hello igniters! I wonder if we can add a new event for a failed ssl
>> > > connection? Considering security hazards it can be quite helpful to get
>> > > notifications in case someone unauthorized is trying to connect to the
>> > > cluster. I've created a small pull-request to illustrate a possible
>> > > solution.
>> > >  https://github.com/apache/ignite/pull/8139
>> > > Please tell me what do you think.
>> > >
>> >
>> 
 
 
 
 

Re: new connection event

[Ilya Kasnacheev]

Hello!

Why not the Failure Handler then?

(I'm only half-joking).

Regards,

-- 
Ilya Kasnacheev


ср, 12 авг. 2020 г. в 09:54, Oleg Ostanin :

> Thank you for the response. Yes, we have a simple warning in log, but in
> case SSL error needs an immediate human attention it would be better to set
> a listener for such events.
>
> вт, 11 авг. 2020 г. в 17:59, Ilya Kasnacheev :
>
> > Hello!
> >
> > I'm not sure this is a good event. Events are computer-processable, while
> > SSL error either needs attention from a human or none at all.
> >
> > Then again, sometimes it's not an SSL error (come to think or it,
> > unauthorized people tend to not use SSL since success is not an option
> > here) but any unexpected input. Such input will mean that something (or
> > somebody) wrong is trying to connect, possibly using a wrong port or
> hoping
> > that SSL is disabled.
> >
> > Maybe we need a simple warning. I guess we already have it, actually.
> Don't
> > we?
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > вт, 11 авг. 2020 г. в 14:27, Oleg Ostanin :
> >
> > > Hello igniters! I wonder if we can add a new event for a failed ssl
> > > connection? Considering security hazards it can be quite helpful to get
> > > notifications in case someone unauthorized is trying to connect to the
> > > cluster. I've created a small pull-request to illustrate a possible
> > > solution.
> > > https://github.com/apache/ignite/pull/8139
> > > Please tell me what do you think.
> > >
> >
>

Re: [DISCUSSION] Add index rebuild time metrics

[Ivan Rakov]

I seem to be in the minority here :)
Fine, let's make it as clear as possible which metric method
(localCacheSize) should be called in order to retrieve a 100% progress
milestone.
I've left comments in the PR.

On Tue, Aug 11, 2020 at 4:31 PM Nikolay Izhikov  wrote:

> > I propose to stick with a cache-group level metric (e.g.
> getIndexBuildProgress)
>
> +1
>
> > that returns a float from 0 to 1, which is calculated as [processedKeys]
> / [localCacheSize].
>
> From my point of view, we shouldn’t do calculations on the Ignite side if
> we can avoid it.
> I’d rather provide two separate metrics - processedKeys and localCacheSize.
>
> > 11 авг. 2020 г., в 16:26, Ivan Rakov  написал(а):
> >
> >>
> >> As a compromise, I can add jmx methods (rebuilding indexes in the
> process
> >> and the percentage of rebuilding) for the entire node, but I tried to
> find
> >> a suitable place and did not find it, tell me where to add it?
> >
> > I have checked existing JMX beans. To be honest, I struggle to find a
> > suitable place as well.
> > We have ClusterMetrics that may represent the state of a local node, but
> > this class is also used for aggregated cluster metrics. I can't propose a
> > reasonable way to merge percentages from different nodes.
> > On the other hand, total index rebuild for all caches isn't a common
> > scenario. It's either performed after manual index.bin removal or after
> > index creation, both operations are performed on cache / cache-group
> level.
> > Also, all other similar metrics are provided on cache-group level.
> >
> > I propose to stick with a cache-group level metric (e.g.
> > getIndexBuildProgress) that returns a float from 0 to 1, which is
> > calculated as [processedKeys] / [localCacheSize]. Even if a user handles
> > metrics through Zabbix, I anticipate that he'll perform this calculation
> on
> > his own in order to estimate progress. Let's help him a bit and perform
> it
> > on the system side.
> > If a per-group percentage metric is present, I
> > think getIndexRebuildKeyProcessed becomes redundant.
> >
> > On Tue, Aug 11, 2020 at 8:20 AM ткаленко кирилл 
> > wrote:
> >
> >> Hi, Ivan!
> >>
> >> What precision would be sufficient?
> >>> If the progress is very slow, I don't see issues with tracking it if
> the
> >>> percentage float has enough precision.
> >>
> >> I think we can add a mention getting cache size.
> >>> 1. Gain an understanding that local cache size
> >>> (CacheMetricsImpl#getCacheSize) should be used as a 100% milestone (it
> >>> isn't mentioned neither in javadoc nor in JMX method description).
> >>
> >> Do you think users collect metrics with their hands? I think this is
> done
> >> by other systems, such as zabbix.
> >>> 2. Manually calculate sum of all metrics and divide to sum of all cache
> >>> sizes.
> >>
> >> As a compromise, I can add jmx methods (rebuilding indexes in the
> process
> >> and the percentage of rebuilding) for the entire node, but I tried to
> find
> >> a suitable place and did not find it, tell me where to add it?
> >>> On the other hand, % of index rebuild progress is self-descriptive. I
> >> don't
> >>> understand why we tend to make user's life harder.
> >>
> >> 10.08.2020, 21:57, "Ivan Rakov" :
>  This metric can be used only for local node, to get size of cache use
> 
> >>
> org.apache.ignite.internal.processors.cache.CacheMetricsImpl#getCacheSize.
> >>>
> >>> Got it, agree.
> >>>
> >>> If there is a lot of data in node that can be rebuilt, percentage may
>  change very rarely and may not give an estimate of how much time is
> >> left.
>  If we see for example that 50_000 keys are rebuilt once a minute, and
> >> we
>  have 1_000_000_000 keys, then we can have an approximate estimate.
> >> What do
>  you think of that?
> >>>
> >>> If the progress is very slow, I don't see issues with tracking it if
> the
> >>> percentage float has enough precision.
> >>> Still, usability of the metric concerns me. In order to estimate
> >> remaining
> >>> time of index rebuild, user should:
> >>> 1. Gain an understanding that local cache size
> >>> (CacheMetricsImpl#getCacheSize) should be used as a 100% milestone (it
> >>> isn't mentioned neither in javadoc nor in JMX method description).
> >>> 2. Manually calculate sum of all metrics and divide to sum of all cache
> >>> sizes.
> >>> On the other hand, % of index rebuild progress is self-descriptive. I
> >> don't
> >>> understand why we tend to make user's life harder.
> >>>
> >>> --
> >>> Best regards,
> >>> Ivan
> >>>
> >>> On Mon, Aug 10, 2020 at 8:53 PM ткаленко кирилл 
> >>> wrote:
> >>>
>  Hi, Ivan!
> 
>  For this you can use
>  org.apache.ignite.cache.CacheMetrics#IsIndexRebuildInProgress
> > How can a local number of processed keys can help us to understand
> >> when
> > index rebuild will be finished?
> 
>  This metric can be used only for local node, to get size of cache use
> 
> >>
> 

Re: new connection event

[Oleg Ostanin]

Thank you for the response. Yes, we have a simple warning in log, but in
case SSL error needs an immediate human attention it would be better to set
a listener for such events.

вт, 11 авг. 2020 г. в 17:59, Ilya Kasnacheev :

> Hello!
>
> I'm not sure this is a good event. Events are computer-processable, while
> SSL error either needs attention from a human or none at all.
>
> Then again, sometimes it's not an SSL error (come to think or it,
> unauthorized people tend to not use SSL since success is not an option
> here) but any unexpected input. Such input will mean that something (or
> somebody) wrong is trying to connect, possibly using a wrong port or hoping
> that SSL is disabled.
>
> Maybe we need a simple warning. I guess we already have it, actually. Don't
> we?
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> вт, 11 авг. 2020 г. в 14:27, Oleg Ostanin :
>
> > Hello igniters! I wonder if we can add a new event for a failed ssl
> > connection? Considering security hazards it can be quite helpful to get
> > notifications in case someone unauthorized is trying to connect to the
> > cluster. I've created a small pull-request to illustrate a possible
> > solution.
> > https://github.com/apache/ignite/pull/8139
> > Please tell me what do you think.
> >
>

Re: Apache Ignite 3.0

[Petr Ivanov]

Hi, Val.

> On 12 Aug 2020, at 01:31, Valentin Kulichenko  
> wrote:
> 
> Hi Petr,
> 
> I agree -- we should better modularize the platform. The current way if very 
> error-prone, especially during upgrades -- any changes made within 
> IGNITE_HOME (configs, scripts, modules, etc.) must be merged with a new 
> version of the package. There is no standard way of doing this.
> 
> However, I'm a bit concerned with your suggestion regarding custom dependency 
> management. Can you please elaborate on how you think it should work? Are 
> there tools we can reuse for this purpose? I would try to avoid reinventing 
> the wheel.

I see it as a a2enmod | 2dismod analog of Apache2.

We build and store Apache Ignite and its modules as separate binaries (binary 
per module) then use custom script that will know where to download necessary 
module. Or possibly use modified ignite.sh to specify required optional libs in 
run command while ignite.sh will download everything missing from known storage.

The whole idea is in storing everything remotely and download on demand, not 
have all libs locally from the start.


> 
> -Val
> 
> On Sun, Aug 9, 2020 at 11:25 PM Petr Ivanov  > wrote:
> Hi, Val!
> Thanks for your efforts on this endeavour!
> 
> 
> I would like to suggest deliveries changes in Apache Ignite 3.0:
>  — modularised  binary delivery — single minimal binary for starting Ignite 
> and all other modules and parts of the project (benchmarks, examples, etc.) 
> packed in their own binary which can be added via custom dependency 
> management tool (i.e. modules.sh)
>  — same distribution for RPM and DEB packages but with modules packed as 
> separate ones (PHP for example)
>  — separate thin client release cycle with custom versioning
> Possibly, we can we add additional section to the document you introduced for 
> this part.
> 
> Also, it seems that full JDK11 support (including building) would be a huge 
> milestone and a sign of healthy modern project that tends to be on the verge 
> of mainstream technologies and not the stockpile of legacy leftovers (fully 
> support Iliya in removing all that was deprecated and/or marked as unused 
> anymore).
> 
> 
> > On 8 Aug 2020, at 02:00, Valentin Kulichenko  > > wrote:
> > 
> > Igniters,
> > 
> > I've created the page:
> > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0 
> > 
> > 
> > That's not everything I have in mind, but I believe there is already a lot
> > to talk about :)
> > 
> > Please take a look let me know if you have any concerns, objections, or
> > questions. Once we reach the consensus on the proposed changes, I will
> > start creating tickets in Jira and a more detailed plan.
> > 
> > -Val
> > 
> > On Thu, Aug 6, 2020 at 6:28 PM Saikat Maitra  > >
> > wrote:
> > 
> >> Hi Denis, Val
> >> 
> >> Thank you for your reply and really appreciate it. It will be very cool to
> >> be able to connect and plan release together and learn more about Ignite in
> >> the process :)
> >> 
> >> Regards
> >> Saikat
> >> 
> >> 
> >> 
> >> On Thu, Aug 6, 2020 at 7:12 PM Valentin Kulichenko <
> >> valentin.kuliche...@gmail.com > 
> >> wrote:
> >> 
> >>> Hi Saikat,
> >>> 
> >>> That surely is a great idea. We will work together with Denis on setting
> >>> this up in the nearest future.
> >>> 
> >>> -Val
> >>> 
> >>> On Thu, Aug 6, 2020 at 10:21 AM Denis Magda  >>> > wrote:
> >>> 
>  Saikat,
>  
>  Fully support your idea on a virtual meetup! Once Val collects and
> >>> outlines
>  the main changes with directions on wiki, we’ll go ahead and schedule
> >> the
>  meetup to talk things out in a bit more detail. We’ll use our new
> >> Virtual
>  Ignite Meetup group for that inviting both Ignite contributors and
>  application developers.
>  
>  Denis
>  
>  On Thursday, August 6, 2020, Saikat Maitra   >
>  wrote:
>  
> > Hi Valentin
> > 
> > Thank you for sharing and starting the thread. I am thinking if it
> >> will
>  be
> > a good idea to have a virtual meet setup to discuss on the release
> > planning.
> > 
> > It will help to learn more individual features to be added and also
> >> to
> > understand about features that have been deprecated and scheduled for
> > removal in Ignite 3.0 release. Also it will help community member to
> > connect in real time and ask questions and share feedback.
> > 
> > Regards,
> > Saikat
> > 
> > On Thu, Aug 6, 2020 at 3:51 AM Ilya Kasnacheev <
>  ilya.kasnach...@gmail.com >
> > wrote:
> > 
> >> Hello!
> >> 
> >> I hope to see Apache Ignite release 3.0 as API trimming release.
> >> Let