Re: Please a reviewer for the case IGNITE-12518

2020-01-23 Thread Luis Arce
Hi Saikat,
Thanks for you response:

My system need a web server for work. i work correctly with Wildfly and i
use Ignite as Backend for database.
My problem is the combination of both is around 6 GB RAM for work in the
VPS.
For this reason i modified the unique plugin with webserver inside
(rest-http module).
The change enabled the possibility for use only 3GB RAM in a VPS (
vpsserver.com) saving money and enabling horizontal scalling.
I shared this change in the Jira ticket touching the two classes of
rest-http plugin.
The change has less impact (programaticly), the point is the library
dependencies (quantity of jars).

It change has sense for you?, i think is a util because that technologies
dont has present inside of ignite currently and is friendly with the code
of rest-ignite.




Best regards,


*Luis Arce Martínez*Licenciado e Ingeniero en Informática y Gestión
09-57861903
Linkedin:
https://cl.linkedin.com/in/luisalejandroarcemartinez





El mié., 22 ene. 2020 a las 0:38, Saikat Maitra ()
escribió:

> Hello Luis,
>
> Thank you for your email. You can plan to create a separate application for
> jaxws service and use any build tools like gradle or maven to define your
> dependencies.
>
> Please find below some of the performance tips related to Ignite
>
> https://apacheignite.readme.io/docs/durable-memory-tuning
>
> You can use IgniteClient in your service and can connect to remote cluster
> of Apache Ignite for data persistence.
>
> Can you please correct my understanding on the usage of ignite-rest-http
> in IGNITE-12518, I see the dependencies you have mentioned are related to
> your project and my understanding is you are trying to use ignite-rest-http
> jetty server for running your application. My understanding is this change
> will make ignite-rest-http very large jar file with dependencies
> like tomcat-servlet-api-9.0.10.jar may not needed outside of your project
> scope.
>
> Please let me know your thoughts.
>
> Regards,
> Saikat
>
>
> On Sun, Jan 19, 2020 at 9:47 PM Luis Arce  wrote:
>
> > Hi Saikat,
> > I agree, the impact of changes is bigger on the module.
> > I have a question: If i need create a jaxws service what is your
> > recomendation?
> > My motivation for the changes is the next:
> > *Introduction.*
> > A few time ago i design a ABB for traceability for Oracle Service Bus
> with
> > the objective of detecting failures points in many processes of a
> customer.
> > In first instance my team worked with rest-http module in ignite 2.4 with
> > poor results, the quantity of TPS was 4.Then we make a implementation of
> > Rest service inside Apache Tomcat and call to Apache Ignite directly to
> > Database with persistence activated. This change, enabled the possibility
> > for work with 4 environment of the customer (Development, Testing, QA,
> > Production) with 8GB of RAM in the machine, the configuration of the
> client
> > had a Oracle Portal for the View layer, EJB for composition of the
> > controller layer, and OSB for the integration the TPS of the client are
> > biggest.
> >
> > *AS-IS*
> >
> > [image: imagen.png]
> >
> > *To Be roadmap*
> >
> > [image: imagen.png]
> >
> > With the ignite modification published in Jira is possible run JSF for
> run
> > my reports and forms, JaxWS for the service SOAP and Jersey for Rest (i
> > start modification in this task).
> > The code published in Jira have capabilites for work with Primefaces
> > (tested ok), JaxWS (tested ok), but jersey is not included yet.
> >
> > Best regards,
> >
> >
> > *Luis Arce Martínez*Licenciado e Ingeniero en Informática y Gestión
> > 09-57861903
> > Linkedin:
> > https://cl.linkedin.com/in/luisalejandroarcemartinez
> >
> >
> >
> >
> >
> > El jue., 16 ene. 2020 a las 0:25, Saikat Maitra (<
> saikat.mai...@gmail.com>)
> > escribió:
> >
> >> Hi Luis,
> >>
> >> Thank you for sharing the details on the changes. I reviewed the
> >> dependencies that you shared in the jira issue and wanted to discuss on
> >> the
> >> changes.
> >>
> >> The purpose of ignite-rest-http is to provide a web based interface to
> >> easily access and use the ignite features and the changes you suggested
> >> can
> >> be built as part of separate application and ignite-rest-http can be
> used
> >> as an add on dependency. This will help keep ignite-rest-http module as
> >> minimal and thin as possible.
> >>
> >> Please review and share your thoughts.
> >>
> >> Regards,
> >> Saikat
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Mon, Jan 13, 2020 at 7:24 PM Luis Arce  wrote:
> >>
> >> > Hi Saikat,
> >> > I add information for evidence for the changes of ignite rest-http
> >> module.
> >> >
> >> > 1. Can you please share more information on the issue that will be
> >> resolved
> >> > with this change?
> >> > R:  This change add the possibility for publish war files with
> webpages
> >> in
> >> > JSF or webservices JaxWS inside of the jetty server embedded.
> >> > When the WAR file is loade automatically attached the JNDI for lockup
> to

Re: Ignite-spring-boot-autoconfigurer

2020-01-23 Thread Saikat Maitra
Hi Nikolay,

Thank you for updating the PR, the changes looks good.

Regards,
Saikat

On Wed, Jan 22, 2020 at 1:33 PM Николай Ижиков  wrote:

> Hello, Saikat.
>
> Thank you so much for the review.
>
> I answered your questions and resolve all the comments.
> Please, take a look, one more time.
>
> > 22 янв. 2020 г., в 07:58, Saikat Maitra 
> написал(а):
> >
> > Hi Nikolay,
> >
> > I have reviewed the PR and shared comments.
> >
> > Please let me know if you have any feedback.
> >
> > Regards,
> > Saikat
> >
> > On Mon, Jan 20, 2020 at 2:42 PM Николай Ижиков 
> wrote:
> >
> >> Hello, Saikat.
> >>
> >> Thanks, for feedback.
> >>
> >> I raised a PR [1] to `ignite-extensions`.
> >>
> >> You can find description of the new module below(examples can be found
> at
> >> [2]):
> >>
> >> Module provides the ability to integrate `Ignite` into you spring-boot
> >> application with zero(or minimal) configuration.
> >>
> >> After you add this module as a dependency to your spring-boot
> application
> >> `Ignite` node will be configured and injected into `BeanFactory`.
> >>
> >> Algorithm to configure `Ignite` is the following:
> >>  1. If `IgniteConfiguration` bean exists in the `BeanFactory` it will be
> >> used.
> >>  2. If `IgniteConfiguration` bean doesn't exist following rules are
> >> applied:
> >>2.1. Default `Ignite` configuration created.
> >>2.2. If `IgniteConfigurer` bean exists in `BeanFactory` it will be
> >> used to customize `IgniteConfiguration`.
> >> If a user wants to set custom SPI instances or similar hardcoded
> >> values
> >> one should do it with `IgniteConfigurer` implementation.
> >>2.3  Application properties applied to `IgniteConfiguration`. Prefix
> >> for the properties is `ignite`.
> >>
> >>
> >> [1] https://github.com/apache/ignite-extensions/pull/6
> >> [2]
> https://github.com/nizhikov/ignite-spring-boot-autoconfigure-example
> >>
> >>
> >>> 18 янв. 2020 г., в 06:44, Saikat Maitra 
> >> написал(а):
> >>>
> >>> Hi Nikolay,
> >>>
> >>> Thank you for your email. As part of Ignite Extensions migration we are
> >> migrating Ignite Extensions to following repo.
> >>>
> >>> https://github.com/apache/ignite-extensions
> >>>
> >>> We have added flink and pub-sub modules and few additional modules are
> >> open in PR.
> >>>
> >>> You can refer to this PR to see how we are migrating the modules
> >> https://github.com/apache/ignite-extensions/pull/5
> >>>
> >>> I wanted to connect and discuss the changes to understand the spring
> >> boot auto configure feature. We currently have an ignite spring module
> that
> >> allows resource injection capabilities and provides a parser for Spring
> >> based xml configuration files. Can you please review and share if the
> >> changes you are proposing can be added as part of Ignite spring module
> or
> >> it make sense to make it a separate spring boot auto configure module.
> >>>
> >>> https://github.com/apache/ignite/tree/master/modules/spring
> >>>
> >>> Regards,
> >>> Saikat
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Fri, Jan 17, 2020 at 3:12 AM Николай Ижиков 
> >> wrote:
> >>> Tests added.
> >>> Please, review.
> >>>
> >>> Saikat, can you help with this PR [1]?
> >>>
> >>> I think it should be added as a separate module as we do with the flink
> >> integration.
> >>> Can you help me with it?
> >>> Do we have some how-to for it?
> >>>
> >>> [1] https://github.com/apache/ignite/pull/7237
> >>>
>  16 янв. 2020 г., в 16:51, Николай Ижиков 
> >> написал(а):
> 
>  Hello, Denis.
> 
>  Thanks, for the feedback.
> 
>  Alexey, it seems, PR is ready to be reviewed, but I need some time(a
> >> day or two) to write tests.
>  You can start with the core code review if you wish.
> 
>  Here are autoconfigurer requirements:
> 
>  1. Start usage of Ignite with minimal(or zero) configuration.
>  2. Configure Ignite configuration properties with the standard spring
> >> boot application properties.
>  3. Configure Ignite SPI implementation and so on that can’t be
> >> configured via #2.
> 
>  After some consultation with the Spring experts from the
> >> community(Maxim Stepachev thanks for the idea)
>  I updated the PR with the logic described below:
> 
>  1. To enable Ignite auto-configuration user should add
> >> `org.apache.ignite:spring-boot-ignite-autoconfigure:2.9.0` to
> dependencies.
>    After it Ignite node will be started during spring-boot application
> >> startup.
> 
>  2. IgniteConfiguration initialization logic:
> 
>  2.1 If {@link IgniteConfiguration} bean exists in {@link BeanFactory}
> >> it will be used for the node start.
>  2.2 If {@link IgniteConfiguration} bean doesn't exist following rules
> >> are applied:
>  * Newly introducer IgniteConfigurer bean will be used to customize an
> >> empty IgniteConfiguration instance.
>    If a user wants to set custom SPI instances or similar hardcoded
> >> values one 

Re: Internal classes are exposed in public API

2020-01-23 Thread Николай Ижиков
> You want say didn't discuss?

Yes.

> Secondly, yes it took a lot of time due to a lot of changes. Is it problem?

No, of course.
My point - that your contribution that took a long time, already, can’t be an 
argument to postpone creation of the API in the current release.


> 23 янв. 2020 г., в 18:11, Andrey Gura  написал(а):
> 
>> We don’t discuss your changes and your proposals for the Metric API.
> 
> You want say didn't discuss? Actually we did it [1] but you told ok
> let's see code :)
> 
>> So I don’t think we can make the existence of some PR is an argument to 
>> introduce(or not introduce) this facade.
> 
> You definitely right in case of competition development. But I think
> that collaborative development is more effective. Isn't it?
> 
>> Moreover, As far as I know, you developing changes for the Metric API for 5 
>> months without public discussion, for now. Let's start it.
> 
> Firsty, with discussion. See above.
> Secondly, yes it took a lot of time due to a lot of changes. Is it problem?
> 
>> Let’s do the following:
>> 1. Review `IgniteMetric` facade and introduce it to the users as an 
>> experimental API.
> 
> It just doesn't make sense. Just commit for commit.
> 
>> 2. Discuss your proposal to the Metric API in the separate thread you are 
>> referencing.
> 
> You a re welcome to thread [1] again. But please, take into account.
> because discussion is postponed by you it's late enough to discuss
> proposed solution. But of course I'll answer on your questions and
> will be glade to your comments and suggestions.
> 
>> I think we should start a discussion from the simplified explanation of:
>> 1. API intentions - What we want to gain with it? What is our focus?
>> 2. Simple, minimal example of API main interfaces and desired usages.
> 
> All this already described at [1]. You also can take a look on PR (see
> MetricSource implementations, there are complex and simple ones).
> 
> [1] 
> http://apache-ignite-developers.2346864.n4.nabble.com/IEP-35-Metrics-management-in-Ignite-tp43243.html
> 
> On Thu, Jan 23, 2020 at 5:42 PM Николай Ижиков  wrote:
>> 
>> Andrey.
>> 
>>> IGNITE-11927 implementation so your changes are incompatible with my changes
>> 
>> We don’t discuss your changes and your proposals for the Metric API.
>> So I don’t think we can make the existence of some PR is an argument to 
>> introduce(or not introduce) this facade.
>> Moreover, As far as I know, you developing changes for the Metric API for 5 
>> months without public discussion, for now. Let's start it.
>> 
>> Let’s do the following:
>> 
>> 1. Review `IgniteMetric` facade and introduce it to the users as an 
>> experimental API.
>> 
>> 2. Discuss your proposal to the Metric API in the separate thread you are 
>> referencing.
>> 
>> I think we should start a discussion from the simplified explanation of:
>> 
>> 1. API intentions - What we want to gain with it? What is our focus?
>> 2. Simple, minimal example of API main interfaces and desired usages.
>> 
>> This will help to the community and me personally better understand your 
>> idea and share feedback.
>> 
>> Thanks.
>> 
>>> 23 янв. 2020 г., в 17:15, Andrey Gura  написал(а):
>>> 
>>> Nikolay,
>>> 
>>> as I wrote early in this thread API significantly changed during
>>> IGNITE-11927 implementation so your changes are incompatible with my
>>> changes.
>>> 
>>> ReadOnlyMetricRegistry will be removed at all (still exists in a
>>> couple of places where it uses).
>>> 
>>> Also I don't want to introduce IgniteMetric facade in this rush. In
>>> current implementation this interface just provides access to the
>>> ReadOnlyMetricManager instance (which will be removed) but it is not
>>> enough.
>>> 
>>> I agree with Denis. We should mark current API as experimental and
>>> continue IEP-35 development. See my process proposal [1] described
>>> early this thread. We can release Apache Ignite 2.8.1 specially for
>>> this changes.
>>> Public APIs require deeper thinking in order to provide comprehensive,
>>> consistent and convenient way of metrics management for end users.
>>> 
>>> Let's add IgniteExperimental, release 2.8 and finish metrics related
>>> issues after it.
>>> 
>>> [1] 
>>> http://apache-ignite-developers.2346864.n4.nabble.com/Internal-classes-are-exposed-in-public-API-tp45146p45185.html
>>> 
>>> 
>>> On Wed, Jan 22, 2020 at 5:21 PM Николай Ижиков  wrote:
 
 Hello, Igniters.
 
 * IGNITE-12552: Move ReadOnlyMetricRegistry to public API merged to the 
 master and cherry-picked to the 2.8.
 So the main issue with the Metric API solved.
 
 * I raised the PR [1] to fix second issue with the new Metric API: absence 
 of the public Java API to get metrics.
 This PR introduces the following changes:
 
 1. IgniteMetric interface created: it provides Java API to access Ignite 
 metrics created with the new Metric API.
 
 ```
 public interface IgniteMetric extends Iterable {
   @Nullable 

Re: Internal classes are exposed in public API

2020-01-23 Thread Andrey Gura
> We don’t discuss your changes and your proposals for the Metric API.

You want say didn't discuss? Actually we did it [1] but you told ok
let's see code :)

> So I don’t think we can make the existence of some PR is an argument to 
> introduce(or not introduce) this facade.

You definitely right in case of competition development. But I think
that collaborative development is more effective. Isn't it?

> Moreover, As far as I know, you developing changes for the Metric API for 5 
> months without public discussion, for now. Let's start it.

Firsty, with discussion. See above.
Secondly, yes it took a lot of time due to a lot of changes. Is it problem?

> Let’s do the following:
> 1. Review `IgniteMetric` facade and introduce it to the users as an 
> experimental API.

It just doesn't make sense. Just commit for commit.

> 2. Discuss your proposal to the Metric API in the separate thread you are 
> referencing.

You a re welcome to thread [1] again. But please, take into account.
because discussion is postponed by you it's late enough to discuss
proposed solution. But of course I'll answer on your questions and
will be glade to your comments and suggestions.

> I think we should start a discussion from the simplified explanation of:
> 1. API intentions - What we want to gain with it? What is our focus?
> 2. Simple, minimal example of API main interfaces and desired usages.

All this already described at [1]. You also can take a look on PR (see
MetricSource implementations, there are complex and simple ones).

[1] 
http://apache-ignite-developers.2346864.n4.nabble.com/IEP-35-Metrics-management-in-Ignite-tp43243.html

On Thu, Jan 23, 2020 at 5:42 PM Николай Ижиков  wrote:
>
> Andrey.
>
> > IGNITE-11927 implementation so your changes are incompatible with my changes
>
> We don’t discuss your changes and your proposals for the Metric API.
> So I don’t think we can make the existence of some PR is an argument to 
> introduce(or not introduce) this facade.
> Moreover, As far as I know, you developing changes for the Metric API for 5 
> months without public discussion, for now. Let's start it.
>
> Let’s do the following:
>
> 1. Review `IgniteMetric` facade and introduce it to the users as an 
> experimental API.
>
> 2. Discuss your proposal to the Metric API in the separate thread you are 
> referencing.
>
> I think we should start a discussion from the simplified explanation of:
>
> 1. API intentions - What we want to gain with it? What is our focus?
> 2. Simple, minimal example of API main interfaces and desired usages.
>
> This will help to the community and me personally better understand your idea 
> and share feedback.
>
> Thanks.
>
> > 23 янв. 2020 г., в 17:15, Andrey Gura  написал(а):
> >
> > Nikolay,
> >
> > as I wrote early in this thread API significantly changed during
> > IGNITE-11927 implementation so your changes are incompatible with my
> > changes.
> >
> > ReadOnlyMetricRegistry will be removed at all (still exists in a
> > couple of places where it uses).
> >
> > Also I don't want to introduce IgniteMetric facade in this rush. In
> > current implementation this interface just provides access to the
> > ReadOnlyMetricManager instance (which will be removed) but it is not
> > enough.
> >
> > I agree with Denis. We should mark current API as experimental and
> > continue IEP-35 development. See my process proposal [1] described
> > early this thread. We can release Apache Ignite 2.8.1 specially for
> > this changes.
> > Public APIs require deeper thinking in order to provide comprehensive,
> > consistent and convenient way of metrics management for end users.
> >
> > Let's add IgniteExperimental, release 2.8 and finish metrics related
> > issues after it.
> >
> > [1] 
> > http://apache-ignite-developers.2346864.n4.nabble.com/Internal-classes-are-exposed-in-public-API-tp45146p45185.html
> >
> >
> > On Wed, Jan 22, 2020 at 5:21 PM Николай Ижиков  wrote:
> >>
> >> Hello, Igniters.
> >>
> >> * IGNITE-12552: Move ReadOnlyMetricRegistry to public API merged to the 
> >> master and cherry-picked to the 2.8.
> >> So the main issue with the Metric API solved.
> >>
> >> * I raised the PR [1] to fix second issue with the new Metric API: absence 
> >> of the public Java API to get metrics.
> >> This PR introduces the following changes:
> >>
> >> 1. IgniteMetric interface created: it provides Java API to access Ignite 
> >> metrics created with the new Metric API.
> >>
> >> ```
> >> public interface IgniteMetric extends Iterable {
> >>@Nullable ReadOnlyMetricRegistry registry(String name);
> >> }
> >> ```
> >>
> >> 2. All deprecation javadocs regarding metrics now reference to the public 
> >> IgniteMetric instead of internal GridMetricManager:
> >>
> >>> @deprecated Use {@link IgniteMetric} instead.
> >>
> >> 3. Tests refactored to use IgniteMetric instead of GridMetricManager when 
> >> possible.
> >>
> >> Please, review.
> >>
> >> [1]  https://github.com/apache/ignite/pull/7283
> >>
> >>> 21 янв. 2020 

[jira] [Created] (IGNITE-12573) Copy ML dependencies to other standalone pom files in examples

2020-01-23 Thread Stepan Pilschikov (Jira)
Stepan Pilschikov created IGNITE-12573:
--

 Summary: Copy  ML dependencies to other standalone pom files in 
examples
 Key: IGNITE-12573
 URL: https://issues.apache.org/jira/browse/IGNITE-12573
 Project: Ignite
  Issue Type: Task
  Components: examples
Affects Versions: 2.8
Reporter: Stepan Pilschikov


2 dependencies missed from pom-standalone.xml and pom-standalone-lgpl.xml


{color:#9876aa}<{color}{color:#e8bf6a}dependency{color}{color:#9876aa}>
{color}{color:#9876aa} 
<{color}{color:#e8bf6a}groupId{color}{color:#9876aa}>{color}org.apache.ignite{color:#9876aa}
{color}{color:#9876aa} 
<{color}{color:#e8bf6a}artifactId{color}{color:#9876aa}>{color}ignite-ml-tensorflow-model-parser{color:#9876aa}
{color}{color:#9876aa} 
<{color}{color:#e8bf6a}version{color}{color:#9876aa}>{color}${project.version}{color:#9876aa}
{color}{color:#9876aa}{color}

{color:#9876aa}<{color}{color:#e8bf6a}dependency{color}{color:#9876aa}>
{color}{color:#9876aa} 
<{color}{color:#e8bf6a}groupId{color}{color:#9876aa}>{color}org.apache.ignite{color:#9876aa}
{color}{color:#9876aa} 
<{color}{color:#e8bf6a}artifactId{color}{color:#9876aa}>{color}ignite-ml-h2o-model-parser{color:#9876aa}
{color}{color:#9876aa} 
<{color}{color:#e8bf6a}version{color}{color:#9876aa}>{color}${project.version}{color:#9876aa}
{color}{color:#9876aa}{color}

Need to copy this libraries from original pom.xml because examples in released 
build can't be build



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Internal classes are exposed in public API

2020-01-23 Thread Николай Ижиков
Andrey.

> IGNITE-11927 implementation so your changes are incompatible with my changes

We don’t discuss your changes and your proposals for the Metric API.
So I don’t think we can make the existence of some PR is an argument to 
introduce(or not introduce) this facade.
Moreover, As far as I know, you developing changes for the Metric API for 5 
months without public discussion, for now. Let's start it.

Let’s do the following:

1. Review `IgniteMetric` facade and introduce it to the users as an 
experimental API.

2. Discuss your proposal to the Metric API in the separate thread you are 
referencing.

I think we should start a discussion from the simplified explanation of:

1. API intentions - What we want to gain with it? What is our focus?
2. Simple, minimal example of API main interfaces and desired usages.

This will help to the community and me personally better understand your idea 
and share feedback.

Thanks.

> 23 янв. 2020 г., в 17:15, Andrey Gura  написал(а):
> 
> Nikolay,
> 
> as I wrote early in this thread API significantly changed during
> IGNITE-11927 implementation so your changes are incompatible with my
> changes.
> 
> ReadOnlyMetricRegistry will be removed at all (still exists in a
> couple of places where it uses).
> 
> Also I don't want to introduce IgniteMetric facade in this rush. In
> current implementation this interface just provides access to the
> ReadOnlyMetricManager instance (which will be removed) but it is not
> enough.
> 
> I agree with Denis. We should mark current API as experimental and
> continue IEP-35 development. See my process proposal [1] described
> early this thread. We can release Apache Ignite 2.8.1 specially for
> this changes.
> Public APIs require deeper thinking in order to provide comprehensive,
> consistent and convenient way of metrics management for end users.
> 
> Let's add IgniteExperimental, release 2.8 and finish metrics related
> issues after it.
> 
> [1] 
> http://apache-ignite-developers.2346864.n4.nabble.com/Internal-classes-are-exposed-in-public-API-tp45146p45185.html
> 
> 
> On Wed, Jan 22, 2020 at 5:21 PM Николай Ижиков  wrote:
>> 
>> Hello, Igniters.
>> 
>> * IGNITE-12552: Move ReadOnlyMetricRegistry to public API merged to the 
>> master and cherry-picked to the 2.8.
>> So the main issue with the Metric API solved.
>> 
>> * I raised the PR [1] to fix second issue with the new Metric API: absence 
>> of the public Java API to get metrics.
>> This PR introduces the following changes:
>> 
>> 1. IgniteMetric interface created: it provides Java API to access Ignite 
>> metrics created with the new Metric API.
>> 
>> ```
>> public interface IgniteMetric extends Iterable {
>>@Nullable ReadOnlyMetricRegistry registry(String name);
>> }
>> ```
>> 
>> 2. All deprecation javadocs regarding metrics now reference to the public 
>> IgniteMetric instead of internal GridMetricManager:
>> 
>>> @deprecated Use {@link IgniteMetric} instead.
>> 
>> 3. Tests refactored to use IgniteMetric instead of GridMetricManager when 
>> possible.
>> 
>> Please, review.
>> 
>> [1]  https://github.com/apache/ignite/pull/7283
>> 
>>> 21 янв. 2020 г., в 17:51, Николай Ижиков  
>>> написал(а):
>>> 
>>> Hello, Igniters.
>>> 
>>> Alexey approved my PR [1] regarding fixing public API for metric exporters.
>>> I’m waiting for a bot visa and merge this PR.
>>> 
>>> As we discussed, the metrics API will be marked with IgniteExperimental 
>>> annotation.
>>> 
>>> If anyone has any objection to this plan, please provide your feedback.
>>> 
>>> [1] https://github.com/apache/ignite/pull/7269
>>> 
 21 янв. 2020 г., в 13:45, Николай Ижиков  
 написал(а):
 
 Thanks, for the review Alexey.
 
 I will fix your comment and  try to implement Monitoring facade, shortly.
 
> 21 янв. 2020 г., в 13:32, Alexey Goncharuk  
> написал(а):
> 
> Nikolay,
> 
> I left a single comment in the PR about the histogram metric. I think the
> API looks much cleaner now.
> 
> I will take care of the @IgniteExperimental annotation.
> 
> пн, 20 янв. 2020 г. в 20:55, Николай Ижиков :
> 
>> Alexey.
>> 
>> PR [1] is waiting for your review.
>> Please, take a look.
>> 
>> I think we should do the following before 2.8 release
>> 
>> * Introduce new @IgniteExperimental annotation as discussed.
>> * Mark Monitoring API with it.
>> * merge «[IEP-35] Expose MetricRegistry to the public API» to 2.8
>> * merge «[IEP-35] public Java metric API» to 2.8
>> 
>> [1] https://github.com/apache/ignite/pull/7269
>> 
>>> 20 янв. 2020 г., в 17:09, Alexey Goncharuk 
>> написал(а):
>>> 
>>> Nikolay,
>>> 
>>> Should we wait for both of the tickets given that we agreed that we are
>>> putting an experimental marker on the new APIs? I'm ok to fix only the
>>> first one and add the experimental marker so that we do not delay 2.8
>>> release.
>>> 
>>> пн, 20 янв. 

Re: Feature masks for thin clients

2020-01-23 Thread Pavel Tupitsyn
Igor, great idea, I think this should be our priority for all thin clients.

Alexei, protocol version bump can be still required for major changes (e.g.
if we want to change handshake format).
But for simpler things like new features that don't affect existing ones,
feature masks allow us to keep current protocol version.

On Thu, Jan 23, 2020 at 3:55 PM Taras Ledkov  wrote:

> Alexei,
>
> After the flags is introduced we can change the flag set instead of
> change protocol version.
> Of course, we will need to up the protocol  version for introducing flags.
>
> On 23.01.2020 15:47, Alexei Scherbakov wrote:
> > Igor Sapego,
> >
> > I do not understand how feature masks can remove the necessity of having
> > protocol versioning.
> > A protocol for one feature can change from release to release.
> >
> > чт, 23 янв. 2020 г. в 15:36, Igor Sapego :
> >
> >> Hi Igniters,
> >>
> >> As we have a lot of different thin clients now, maintained by different
> >> people, the issues with our backward compatibility mechanism becomes
> >> more and more prominent.
> >>
> >> Currently, we use protocol versioning as the only approach to provide
> >> backward compatibility. The main issue of this approach is that we can
> >> not skip some change in protocol and implement i.e. protocol of version
> >> 1.5 without implementation of 1.4. There are many cases when one may
> >> want to do so: e.g. when feature provided in 1.4 is not relevant for a
> >> specific client, or when protocol version 1.5 contains urgent fix or
> >> feature
> >> which is easy to implement, but its blocked by not-so-urgent and
> >> hard-to-implement feature introduced in 1.4.
> >>
> >> So to fix this issue I propose to introduce another backward
> compatibility
> >> mechanism. The idea is to send "supported features" mask by a client to
> >> a server, which should be answered with the same mask by the server.
> >> The resulting set of enabled features is acquired with a simple logical
> >> "AND"
> >> operation on these two masks.
> >>
> >> This change has many other positive effects:
> >> 1. It improves readability and also potentially simplifies debugging.
> >> 2. It gives users the ability to enable or disable features of thin
> clients
> >> on both
> >> server and client as they desire.
> >>
> >> What are your thoughts guys?
> >>
> >> Best Regards,
> >> Igor
> >>
> >
> --
> Taras Ledkov
> Mail-To: tled...@gridgain.com
>
>


Re: Add user attributes to thin clients

2020-01-23 Thread Pavel Tupitsyn
>  Well, my understanding was a binary object with compact footer = false
is totally standalone entity and can be read without any external metadata
Not exactly: type name and field names are unknown, you only have typeId,
field ids and positions.
There is one more Marshaller "mode": UNREGISTERED_TYPE_ID. In this mode,
full type name is included in the binary object.
However, field names are still missing and the class needs to be present to
deserialize.
That's why arbitrary objects should not be allowed in the handshake: in
most cases they can't be decoded.

On Thu, Jan 23, 2020 at 2:27 PM Dmitrii Ryabov 
wrote:

> Alexei, yes, compactFooter is used only in 1 place.
>
> ```
>
> BinaryWriterExImpl.marshal0() {
>
>   BinaryClassDescriptor desc = ctx.descriptorForClass(cls);
>
>   // descriptor transportation fails here
>
>   ...
>
>   desc.write(obj, this); // compactFooter here
>
> }
>
> ```
>
> чт, 23 янв. 2020 г., 14:15 Pavel Tupitsyn :
>
> > > If we support only strings it will be necessary to encode binary values
> > to
> > > something like BASE64 which is not sounds good from usability side
> >
> > There should be no need to put binary values to attributes. What's the
> use
> > case?
> >
> > On Thu, Jan 23, 2020 at 2:08 PM Alexei Scherbakov <
> > alexey.scherbak...@gmail.com> wrote:
> >
> > > > Footer is checked in postWrite - much later class descriptor check.
> > >
> > > Well, my understanding was a binary object with compact footer = false
> is
> > > totally standalone entity and can be read without any external
> metadata.
> > > Dmitrii Ryabov can you double check ?
> > >
> > > If we support only strings it will be necessary to encode binary values
> > to
> > > something like BASE64 which is not sounds good from usability side.
> > >
> > >
> > >
> > > чт, 23 янв. 2020 г. в 13:26, Nikita Amelchev :
> > >
> > > > +1 for the hardcoded String type only
> > > >
> > > > чт, 23 янв. 2020 г. в 13:15, Pavel Tupitsyn :
> > > > >
> > > > > - Cross-platform binary objects are totally possible, all those
> thin
> > > > > clients support them.
> > > > > - User attributes can be useful, no objections here
> > > > >
> > > > > However, I don't think we should allow arbitrary objects in user
> > > > attributes.
> > > > > Let's make them string only, much less to worry about.
> > > > >
> > > > > And using attributes for authentication still seems weird and
> dirty.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Jan 23, 2020 at 12:40 PM Dmitrii Ryabov <
> > somefire...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > > Even if compact footer is disabled ?
> > > > > > Footer is checked in postWrite - much later class descriptor
> check.
> > > > > >
> > > > > > чт, 23 янв. 2020 г., 12:23 Alexei Scherbakov <
> > > > alexey.scherbak...@gmail.com
> > > > > > >:
> > > > > >
> > > > > > > чт, 23 янв. 2020 г. в 12:17, Dmitrii Ryabov <
> > somefire...@gmail.com
> > > >:
> > > > > > >
> > > > > > > > > The protocol must be language-agnostic. If we add some
> > features
> > > > > > there,
> > > > > > > > let's make sure they are usable from anywhere.
> > > > > > > >
> > > > > > > > That's why I want to allow primitives only. Any language can
> > send
> > > > > > numbers
> > > > > > > > and strings.
> > > > > > > >
> > > > > > >
> > > > > > > In general it's possible to have cross-platform complex data
> > > > structures,
> > > > > > > for example see protobuf.
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > Binary marshaller, before packing object to byte[], will try
> to
> > > use
> > > > > > > > discovery processor and send message containing class
> > descriptor.
> > > > But
> > > > > > > thin
> > > > > > > > clients don't have discovery. Furthermore, if we write binary
> > > > > > marshaller
> > > > > > > > without class descriptor synchronization, we can get objects
> > with
> > > > > > > different
> > > > > > > > class versions and corresponding exceptions.
> > > > > > > >
> > > > > > >
> > > > > > > Even if compact footer is disabled ?
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > But we can say users to declare their classes in
> > > > > > > > META-INF/classnames.properties and current binary marshaller
> > will
> > > > works
> > > > > > > > good.
> > > > > > > >
> > > > > > >
> > > > > > > This approach doesn't looks like cross-platform.
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > чт, 23 янв. 2020 г., 12:13 Alex Plehanov <
> > > plehanov.a...@gmail.com
> > > > >:
> > > > > > > >
> > > > > > > > > Hello,
> > > > > > > > >
> > > > > > > > > User attributes also (besides authentication) can be used
> to
> > > pass
> > > > > > some
> > > > > > > > info
> > > > > > > > > about an application that uses a client and then display
> this
> > > > > > > information
> > > > > > > > > in monitoring tools. Other vendors use such approach
> (Oracle
> > > DB,
> > > > for
> > > > > > > > > example, have DBMS_APPLICATION_INFO package, PostgreeSQL
> have
> > > > > > > > > 

Re: Internal classes are exposed in public API

2020-01-23 Thread Andrey Gura
Nikolay,

as I wrote early in this thread API significantly changed during
IGNITE-11927 implementation so your changes are incompatible with my
changes.

ReadOnlyMetricRegistry will be removed at all (still exists in a
couple of places where it uses).

Also I don't want to introduce IgniteMetric facade in this rush. In
current implementation this interface just provides access to the
ReadOnlyMetricManager instance (which will be removed) but it is not
enough.

I agree with Denis. We should mark current API as experimental and
continue IEP-35 development. See my process proposal [1] described
early this thread. We can release Apache Ignite 2.8.1 specially for
this changes.
Public APIs require deeper thinking in order to provide comprehensive,
consistent and convenient way of metrics management for end users.

Let's add IgniteExperimental, release 2.8 and finish metrics related
issues after it.

[1] 
http://apache-ignite-developers.2346864.n4.nabble.com/Internal-classes-are-exposed-in-public-API-tp45146p45185.html


On Wed, Jan 22, 2020 at 5:21 PM Николай Ижиков  wrote:
>
> Hello, Igniters.
>
> * IGNITE-12552: Move ReadOnlyMetricRegistry to public API merged to the 
> master and cherry-picked to the 2.8.
> So the main issue with the Metric API solved.
>
> * I raised the PR [1] to fix second issue with the new Metric API: absence of 
> the public Java API to get metrics.
> This PR introduces the following changes:
>
> 1. IgniteMetric interface created: it provides Java API to access Ignite 
> metrics created with the new Metric API.
>
> ```
> public interface IgniteMetric extends Iterable {
> @Nullable ReadOnlyMetricRegistry registry(String name);
> }
> ```
>
> 2. All deprecation javadocs regarding metrics now reference to the public 
> IgniteMetric instead of internal GridMetricManager:
>
>   > @deprecated Use {@link IgniteMetric} instead.
>
> 3. Tests refactored to use IgniteMetric instead of GridMetricManager when 
> possible.
>
> Please, review.
>
> [1]  https://github.com/apache/ignite/pull/7283
>
> > 21 янв. 2020 г., в 17:51, Николай Ижиков  
> > написал(а):
> >
> > Hello, Igniters.
> >
> > Alexey approved my PR [1] regarding fixing public API for metric exporters.
> > I’m waiting for a bot visa and merge this PR.
> >
> > As we discussed, the metrics API will be marked with IgniteExperimental 
> > annotation.
> >
> > If anyone has any objection to this plan, please provide your feedback.
> >
> > [1] https://github.com/apache/ignite/pull/7269
> >
> >> 21 янв. 2020 г., в 13:45, Николай Ижиков  
> >> написал(а):
> >>
> >> Thanks, for the review Alexey.
> >>
> >> I will fix your comment and  try to implement Monitoring facade, shortly.
> >>
> >>> 21 янв. 2020 г., в 13:32, Alexey Goncharuk  
> >>> написал(а):
> >>>
> >>> Nikolay,
> >>>
> >>> I left a single comment in the PR about the histogram metric. I think the
> >>> API looks much cleaner now.
> >>>
> >>> I will take care of the @IgniteExperimental annotation.
> >>>
> >>> пн, 20 янв. 2020 г. в 20:55, Николай Ижиков :
> >>>
>  Alexey.
> 
>  PR [1] is waiting for your review.
>  Please, take a look.
> 
>  I think we should do the following before 2.8 release
> 
>  * Introduce new @IgniteExperimental annotation as discussed.
>  * Mark Monitoring API with it.
>  * merge «[IEP-35] Expose MetricRegistry to the public API» to 2.8
>  * merge «[IEP-35] public Java metric API» to 2.8
> 
>  [1] https://github.com/apache/ignite/pull/7269
> 
> > 20 янв. 2020 г., в 17:09, Alexey Goncharuk 
>  написал(а):
> >
> > Nikolay,
> >
> > Should we wait for both of the tickets given that we agreed that we are
> > putting an experimental marker on the new APIs? I'm ok to fix only the
> > first one and add the experimental marker so that we do not delay 2.8
> > release.
> >
> > пн, 20 янв. 2020 г. в 13:32, Николай Ижиков :
> >
> >> Andrey.
> >>
> >> Let’s move from the long letters to the code.
> >> If you want to change API - please, propose this changes.
> >> I think everybody wins if we make our API better.
> >>
> >> Please, describe proposed changes.
> >> It would be great if you have some examples or PR for it.
> >>
> >> As for this release, I have plans to contribute tickets
> >> «[IEP-35] Expose MetricRegistry to the public API» [1] and
> >> «[IEP-35] public Java metric API» [2] for it.
> >>
> >> Any objections on it?
> >>
> >> [1] https://github.com/apache/ignite/pull/7269
> >> [2] https://issues.apache.org/jira/browse/IGNITE-12553
> >>
> >>
> >>> 20 янв. 2020 г., в 13:08, Andrey Gura  написал(а):
> >>>
> >>> It solves problem.
> >>>
> >>> On Mon, Jan 20, 2020 at 12:09 PM Alexey Goncharuk
> >>>  wrote:
> 
>  After giving it some thought, I agree with Denis - there is nothing
> >> wrong
>  with exposing the new APIs, we just need to make it 

[jira] [Created] (IGNITE-12572) Get rid of synchronous cache destroying in multiple caches group.

2020-01-23 Thread Alexey Scherbakov (Jira)
Alexey Scherbakov created IGNITE-12572:
--

 Summary: Get rid of synchronous cache destroying in multiple 
caches group.
 Key: IGNITE-12572
 URL: https://issues.apache.org/jira/browse/IGNITE-12572
 Project: Ignite
  Issue Type: Improvement
Affects Versions: 2.7.6
Reporter: Alexey Scherbakov
 Fix For: 2.9


Current implementation of destroying a cache belonging to multiple caches group 
is done synchronously in exchange worker and blocks exchange events processing 
[1].

This has negative impact on grid availability if topology is changed during 
this process.

Proposed solution: make cache destroying asynchronous, similar to partition 
eviction. Same mechanics can be reused for clearing partition of destroyed 
cache data.

Special attention should be given to the case then cache is recreated while 
being destroyed.
The idea is to continue clearing of old data asynchronously and insert new data 
using new cache generation. This can be achieved for example by implementing 
unique key prefixes for each cache generation.

[1] 
org.apache.ignite.internal.processors.cache.GridCacheProcessor#processCacheStopRequestOnExchangeDone



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (IGNITE-12571) Statistics of query cache statements

2020-01-23 Thread Konstantin Orlov (Jira)
Konstantin Orlov created IGNITE-12571:
-

 Summary: Statistics of query cache statements
 Key: IGNITE-12571
 URL: https://issues.apache.org/jira/browse/IGNITE-12571
 Project: Ignite
  Issue Type: Bug
  Components: sql
Reporter: Konstantin Orlov
Assignee: Konstantin Orlov


We need to collect hit/miss statistics for the query cache.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: [MTCGA]: new failures in builds [4944565] needs to be handled

2020-01-23 Thread Alex Plehanov
Ivan, yes, it's https://issues.apache.org/jira/browse/IGNITE-12562

The ticket is already fixed, tests passed. Zhenya Stanilovsky has agreed to
have a look at fix in the near future (when hi will have some time). But I
would appreciate it if someone else reviews the ticket shortly. It's just a
small test fix (+8 -4).

чт, 23 янв. 2020 г. в 16:32, Ivan Pavlukhin :

> Sorry, missed it https://issues.apache.org/jira/browse/IGNITE-12562
>
> чт, 23 янв. 2020 г. в 16:30, Ivan Pavlukhin :
> >
> > Do we have a ticket for it?
> >
> > ср, 22 янв. 2020 г. в 09:21, Alex Plehanov :
> > >
> > > It was caused by my fix. I will try to fix it shortly.
> > >
> > >
> > > ср, 22 янв. 2020 г. в 00:56, :
> > >
> > > > Hi Igniters,
> > > >
> > > >  I've detected some new issue on TeamCity to be handled. You are
> more than
> > > > welcomed to help.
> > > >
> > > >  If your changes can lead to this failure(s): We're grateful that
> you were
> > > > a volunteer to make the contribution to this project, but things
> change and
> > > > you may no longer be able to finalize your contribution.
> > > >  Could you respond to this email and indicate if you wish to
> continue and
> > > > fix test failures or step down and some committer may revert you
> commit.
> > > >
> > > >  *Recently contributed test failed in master-nightly
> > > > FreeListCachingTest.testPageListCacheLimit
> > > >
> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-4892367253112139895=%3Cdefault%3E=testDetails
> > > >  Changes may lead to failure were done by
> > > >  - aleksei scherbakov 
> > > > https://ci.ignite.apache.org/viewModification.html?modId=895725
> > > >  - aleksei scherbakov 
> > > > https://ci.ignite.apache.org/viewModification.html?modId=895727
> > > >  - ilya kasnacheev 
> > > > https://ci.ignite.apache.org/viewModification.html?modId=895710
> > > >  - aleksey plekhanov 
> > > > https://ci.ignite.apache.org/viewModification.html?modId=895739
> > > >  - anton kalashnikov 
> > > > https://ci.ignite.apache.org/viewModification.html?modId=895723
> > > >  - alexey goncharuk 
> > > > https://ci.ignite.apache.org/viewModification.html?modId=895735
> > > >  - anton kalashnikov 
> > > > https://ci.ignite.apache.org/viewModification.html?modId=895719
> > > >  - alexey goncharuk 
> > > > https://ci.ignite.apache.org/viewModification.html?modId=895729
> > > >
> > > >  - Here's a reminder of what contributors were agreed to do
> > > > https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute
> > > >  - Should you have any questions please contact
> > > > dev@ignite.apache.org
> > > >
> > > > Best Regards,
> > > > Apache Ignite TeamCity Bot
> > > > https://github.com/apache/ignite-teamcity-bot
> > > > Notification generated at 00:56:41 22-01-2020
> > > >
> >
> >
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>


Re: Re[4]: Text queries/indexes (GridLuceneIndex, @QueryTextFiled)

2020-01-23 Thread Yuriy Shuliga
Hi Ivan,

Actually I have engaged another developer to help bring TextQueries to
correctly working state.
For now we have solution that adds Ordering functionality to distributed
TextQueries .
This is developed and tested locally. I can share details here, then we can
discuss and decide whether to create a corresponding ticket.

The starting point is that by nature Lucene's documents are always ordered
by docScore:float;
So we created abstract class Ranked, implementing Comparable and
Serializable; and containing float rank value;

Each entity expected to be ordered on TextQuery merge should be
derived from this class.
All subsequent actions will be done under the hood automatically due
to new CacheQueryFutureRankedDecorator

that contain special BlockingIterator used for correct merge of distributed
responses.
Text queries with Ranked entities are automatically wrapped with this new
decorator.

This is a contour of solution. Please ask if any questions.
Or i can create ticket and link PR with already tested (yet locally)
solution to it for detailed review.

BR,
Yuriy


вт, 21 січ. 2020 о 07:29 Ivan Pavlukhin  пише:

> Hi Yuriy,
>
> Just would like to realize current state. Are you still working on
> Ignite text queries? If not, are you going to continue with it?
>
> пт, 13 дек. 2019 г. в 11:52, Ivan Pavlukhin :
> >
> > Yuriy,
> >
> > Sure, I will be glad to help.
> >
> > > - incorrect nodes/partition selection during querying?
> > Apparently this is the problem. If you feel it really complicated to
> > understand and debug then I can dig deeper and share my vision how the
> > problem can be fixed.
> >
> > ср, 11 дек. 2019 г. в 18:46, Yuriy Shuliga :
> > >
> > > I will look to the MOVING partition issue.
> > > But also need a guidance there.
> > >
> > > Ivan, don't you mind to be that person?
> > >
> > > The question is whether we have an issue with:
> > > -  wrong storing targets during indexing OR
> > > - incorrect nodes/partition selection during querying?
> > >
> > > BR,
> > > Yuriy Shluiha
> > >
> > >
> > >
> > > --
> > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> >
> >
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>


Re: AWS EBS Discovery: Contributor Wanted

2020-01-23 Thread Emmanouil Gkatziouras
Hi all!

Yes It seems possible to get some free quota for integration tests on AWS
[1] however most probably they are not gonna last forever.

[1]
https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/

King Regards
*Emmanouil Gkatziouras*
https://egkatzioura.com/ | https://www.linkedin.com/in/gkatziourasemmanouil/
https://github.com/gkatzioura


On Wed, 22 Jan 2020 at 16:48, Denis Magda  wrote:

> Hi Emmanouil,
>
> Thanks for preparing a pull-request for Application Load Balancer:
> https://issues.apache.org/jira/browse/IGNITE-8617
>
> Igniters, who is willing to step in as a primary reviewer?
>
> As for automated testing on AWS, are you aware of any sponsorship program
> of AWS for open source projects of our kind? It will be ideal to have real
> testing on AWS but someone needs to pay.
>
> -
> Denis
>
>
> On Sun, Jan 19, 2020 at 6:45 AM Emmanouil Gkatziouras <
> gkatzio...@gmail.com>
> wrote:
>
> > Hi all!
> >
> > I have spinned up an Application Load Balancer and an autoscaling group
> on
> > AWS and the Ignite discovery using TcpDiscoveryAlbIpFinder works as
> > expected.
> >
> >- On startup nodes discover each other.
> >- On ec2 node down, connection is lost and the cluster decreases.
> >- On an extra node addition the cluster size increases
> >
> > This contribution is essential since the Previous ELB based discovery
> uses
> > the Classic Load Balancer which is still available however
> > AWS advices users to use the Application one. [1]
> > While my pull request gets reviewed I will also have a look at
> > the IGNITE-12398 [2] issue which has to do with the S3 discovery.
> > Another idea would also be to implement a `TCP Load Balancer based`
> > discovery.
> >
> > In order to test this issue and future ones I implemented some terraform
> > scripts (which I shall use for other issues too) [3].
> > If some automated e2e testing on AWS is being considered they might be of
> > value.
> > I can help on implementing those tests by provisioning the infrastructure
> > in an automated way and validate the discovery.
> >
> > [1]
> >
> >
> https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/migrate-to-application-load-balancer.html
> > [2] https://issues.apache.org/jira/browse/IGNITE-12398
> > [3] https://github.com/gkatzioura/ignite-aws-deploy
> >
> > Kind regards,
> > *Emmanouil Gkatziouras*
> > https://egkatzioura.com/ |
> > https://www.linkedin.com/in/gkatziourasemmanouil/
> > https://github.com/gkatzioura
> >
> >
> > On Tue, 14 Jan 2020 at 22:22, Denis Magda  wrote:
> >
> > > Hi Emmanouil,
> > >
> > > Agree, let's check that the IP finder functions normally in the cloud
> > > environment and the mock tests can be used for regular testing on Team
> > > City. That's the way we tested other environment-specific IP finders
> > > including the Kubernetes one.
> > >
> > > Let us know once the IP finder is tested on AWS and then we can proceed
> > > with the review.
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Tue, Jan 14, 2020 at 2:47 AM Emmanouil Gkatziouras <
> > > gkatzio...@gmail.com>
> > > wrote:
> > >
> > > > Hi all!
> > > >
> > > > With regards to the `Node Discovery Using AWS Application ELB` issue
> > [1]
> > > > I made this pull request to fix the merge conflicts [2].
> > > > Will also check if the concerns described on the mailing list topic
> > > > `Volunteer needed: AWS Elastic Load Balancer IP Finders implemented`
> > are
> > > > addressed [3]
> > > > Since I did not test it on an AWS infrastructure the next step would
> be
> > > to
> > > > spin up some AWS resources and test it.
> > > > By doing so it will help me on checking the Amazon S3 Based Discovery
> > > issue
> > > > [4] .
> > > > I will try to make the deployment as infrastructure as code, and see
> if
> > > > this can be of value for the ignite project and testing.
> > > >
> > > > Kind regards,
> > > > Emmanouil
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-8617
> > > > [2]  https://github.com/apache/ignite/pull/7247
> > > > [3]
> > > >
> > > >
> > >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/Volunteer-needed-AWS-Elastic-Load-Balancer-IP-Finders-implemented-td33847.html#a39793
> > > > [4] https://issues.apache.org/jira/browse/IGNITE-12398
> > > >
> > > > >
> > > >
> > >
> >
>


Re: [MTCGA]: new failures in builds [4944565] needs to be handled

2020-01-23 Thread Ivan Pavlukhin
Sorry, missed it https://issues.apache.org/jira/browse/IGNITE-12562

чт, 23 янв. 2020 г. в 16:30, Ivan Pavlukhin :
>
> Do we have a ticket for it?
>
> ср, 22 янв. 2020 г. в 09:21, Alex Plehanov :
> >
> > It was caused by my fix. I will try to fix it shortly.
> >
> >
> > ср, 22 янв. 2020 г. в 00:56, :
> >
> > > Hi Igniters,
> > >
> > >  I've detected some new issue on TeamCity to be handled. You are more than
> > > welcomed to help.
> > >
> > >  If your changes can lead to this failure(s): We're grateful that you were
> > > a volunteer to make the contribution to this project, but things change 
> > > and
> > > you may no longer be able to finalize your contribution.
> > >  Could you respond to this email and indicate if you wish to continue and
> > > fix test failures or step down and some committer may revert you commit.
> > >
> > >  *Recently contributed test failed in master-nightly
> > > FreeListCachingTest.testPageListCacheLimit
> > > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-4892367253112139895=%3Cdefault%3E=testDetails
> > >  Changes may lead to failure were done by
> > >  - aleksei scherbakov 
> > > https://ci.ignite.apache.org/viewModification.html?modId=895725
> > >  - aleksei scherbakov 
> > > https://ci.ignite.apache.org/viewModification.html?modId=895727
> > >  - ilya kasnacheev 
> > > https://ci.ignite.apache.org/viewModification.html?modId=895710
> > >  - aleksey plekhanov 
> > > https://ci.ignite.apache.org/viewModification.html?modId=895739
> > >  - anton kalashnikov 
> > > https://ci.ignite.apache.org/viewModification.html?modId=895723
> > >  - alexey goncharuk 
> > > https://ci.ignite.apache.org/viewModification.html?modId=895735
> > >  - anton kalashnikov 
> > > https://ci.ignite.apache.org/viewModification.html?modId=895719
> > >  - alexey goncharuk 
> > > https://ci.ignite.apache.org/viewModification.html?modId=895729
> > >
> > >  - Here's a reminder of what contributors were agreed to do
> > > https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute
> > >  - Should you have any questions please contact
> > > dev@ignite.apache.org
> > >
> > > Best Regards,
> > > Apache Ignite TeamCity Bot
> > > https://github.com/apache/ignite-teamcity-bot
> > > Notification generated at 00:56:41 22-01-2020
> > >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin



-- 
Best regards,
Ivan Pavlukhin


Re: [MTCGA]: new failures in builds [4944565] needs to be handled

2020-01-23 Thread Ivan Pavlukhin
Do we have a ticket for it?

ср, 22 янв. 2020 г. в 09:21, Alex Plehanov :
>
> It was caused by my fix. I will try to fix it shortly.
>
>
> ср, 22 янв. 2020 г. в 00:56, :
>
> > Hi Igniters,
> >
> >  I've detected some new issue on TeamCity to be handled. You are more than
> > welcomed to help.
> >
> >  If your changes can lead to this failure(s): We're grateful that you were
> > a volunteer to make the contribution to this project, but things change and
> > you may no longer be able to finalize your contribution.
> >  Could you respond to this email and indicate if you wish to continue and
> > fix test failures or step down and some committer may revert you commit.
> >
> >  *Recently contributed test failed in master-nightly
> > FreeListCachingTest.testPageListCacheLimit
> > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-4892367253112139895=%3Cdefault%3E=testDetails
> >  Changes may lead to failure were done by
> >  - aleksei scherbakov 
> > https://ci.ignite.apache.org/viewModification.html?modId=895725
> >  - aleksei scherbakov 
> > https://ci.ignite.apache.org/viewModification.html?modId=895727
> >  - ilya kasnacheev 
> > https://ci.ignite.apache.org/viewModification.html?modId=895710
> >  - aleksey plekhanov 
> > https://ci.ignite.apache.org/viewModification.html?modId=895739
> >  - anton kalashnikov 
> > https://ci.ignite.apache.org/viewModification.html?modId=895723
> >  - alexey goncharuk 
> > https://ci.ignite.apache.org/viewModification.html?modId=895735
> >  - anton kalashnikov 
> > https://ci.ignite.apache.org/viewModification.html?modId=895719
> >  - alexey goncharuk 
> > https://ci.ignite.apache.org/viewModification.html?modId=895729
> >
> >  - Here's a reminder of what contributors were agreed to do
> > https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute
> >  - Should you have any questions please contact
> > dev@ignite.apache.org
> >
> > Best Regards,
> > Apache Ignite TeamCity Bot
> > https://github.com/apache/ignite-teamcity-bot
> > Notification generated at 00:56:41 22-01-2020
> >



-- 
Best regards,
Ivan Pavlukhin


Re: Apache Ignite contribution

2020-01-23 Thread Ilya Kasnacheev
Hello!

I have added you to Contributors of our project, you can now assign issues
to yourself.

Please read
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute

Regards,
-- 
Ilya Kasnacheev


чт, 23 янв. 2020 г. в 15:31, Лев Киселев :

> Hello,
> I want to take part in the development of Apache Ignite.
> Primary skills: Java SE, Java 8, Spring framework, SQL.
> Also: Multithreading (incl. FJP), Design Patterns, Algorithms and Data
> Structures.
>
> My JIRA ID: l4ndsc4pe
>
> Thanks.
>


Re: Feature masks for thin clients

2020-01-23 Thread Taras Ledkov

Alexei,

After the flags is introduced we can change the flag set instead of 
change protocol version.

Of course, we will need to up the protocol  version for introducing flags.

On 23.01.2020 15:47, Alexei Scherbakov wrote:

Igor Sapego,

I do not understand how feature masks can remove the necessity of having
protocol versioning.
A protocol for one feature can change from release to release.

чт, 23 янв. 2020 г. в 15:36, Igor Sapego :


Hi Igniters,

As we have a lot of different thin clients now, maintained by different
people, the issues with our backward compatibility mechanism becomes
more and more prominent.

Currently, we use protocol versioning as the only approach to provide
backward compatibility. The main issue of this approach is that we can
not skip some change in protocol and implement i.e. protocol of version
1.5 without implementation of 1.4. There are many cases when one may
want to do so: e.g. when feature provided in 1.4 is not relevant for a
specific client, or when protocol version 1.5 contains urgent fix or
feature
which is easy to implement, but its blocked by not-so-urgent and
hard-to-implement feature introduced in 1.4.

So to fix this issue I propose to introduce another backward compatibility
mechanism. The idea is to send "supported features" mask by a client to
a server, which should be answered with the same mask by the server.
The resulting set of enabled features is acquired with a simple logical
"AND"
operation on these two masks.

This change has many other positive effects:
1. It improves readability and also potentially simplifies debugging.
2. It gives users the ability to enable or disable features of thin clients
on both
server and client as they desire.

What are your thoughts guys?

Best Regards,
Igor




--
Taras Ledkov
Mail-To: tled...@gridgain.com



Re: Feature masks for thin clients

2020-01-23 Thread Taras Ledkov

Hi,

The great idea!

What do you think about thin JDBC and ODBC?
Should we have common feature flags set and client specific feature flag 
set?

Or each client has independent flags set even if they are duplicated?

On 23.01.2020 15:36, Igor Sapego wrote:

Hi Igniters,

As we have a lot of different thin clients now, maintained by different
people, the issues with our backward compatibility mechanism becomes
more and more prominent.

Currently, we use protocol versioning as the only approach to provide
backward compatibility. The main issue of this approach is that we can
not skip some change in protocol and implement i.e. protocol of version
1.5 without implementation of 1.4. There are many cases when one may
want to do so: e.g. when feature provided in 1.4 is not relevant for a
specific client, or when protocol version 1.5 contains urgent fix or feature
which is easy to implement, but its blocked by not-so-urgent and
hard-to-implement feature introduced in 1.4.

So to fix this issue I propose to introduce another backward compatibility
mechanism. The idea is to send "supported features" mask by a client to
a server, which should be answered with the same mask by the server.
The resulting set of enabled features is acquired with a simple logical
"AND"
operation on these two masks.

This change has many other positive effects:
1. It improves readability and also potentially simplifies debugging.
2. It gives users the ability to enable or disable features of thin clients
on both
server and client as they desire.

What are your thoughts guys?

Best Regards,
Igor


--
Taras Ledkov
Mail-To: tled...@gridgain.com



Feature masks for thin clients

2020-01-23 Thread Igor Sapego
Hi Igniters,

As we have a lot of different thin clients now, maintained by different
people, the issues with our backward compatibility mechanism becomes
more and more prominent.

Currently, we use protocol versioning as the only approach to provide
backward compatibility. The main issue of this approach is that we can
not skip some change in protocol and implement i.e. protocol of version
1.5 without implementation of 1.4. There are many cases when one may
want to do so: e.g. when feature provided in 1.4 is not relevant for a
specific client, or when protocol version 1.5 contains urgent fix or feature
which is easy to implement, but its blocked by not-so-urgent and
hard-to-implement feature introduced in 1.4.

So to fix this issue I propose to introduce another backward compatibility
mechanism. The idea is to send "supported features" mask by a client to
a server, which should be answered with the same mask by the server.
The resulting set of enabled features is acquired with a simple logical
"AND"
operation on these two masks.

This change has many other positive effects:
1. It improves readability and also potentially simplifies debugging.
2. It gives users the ability to enable or disable features of thin clients
on both
server and client as they desire.

What are your thoughts guys?

Best Regards,
Igor


Apache Ignite contribution

2020-01-23 Thread Лев Киселев
Hello,
I want to take part in the development of Apache Ignite.
Primary skills: Java SE, Java 8, Spring framework, SQL.
Also: Multithreading (incl. FJP), Design Patterns, Algorithms and Data
Structures.

My JIRA ID: l4ndsc4pe

Thanks.


Re: Add user attributes to thin clients

2020-01-23 Thread Dmitrii Ryabov
Alexei, yes, compactFooter is used only in 1 place.

```

BinaryWriterExImpl.marshal0() {

  BinaryClassDescriptor desc = ctx.descriptorForClass(cls);

  // descriptor transportation fails here

  ...

  desc.write(obj, this); // compactFooter here

}

```

чт, 23 янв. 2020 г., 14:15 Pavel Tupitsyn :

> > If we support only strings it will be necessary to encode binary values
> to
> > something like BASE64 which is not sounds good from usability side
>
> There should be no need to put binary values to attributes. What's the use
> case?
>
> On Thu, Jan 23, 2020 at 2:08 PM Alexei Scherbakov <
> alexey.scherbak...@gmail.com> wrote:
>
> > > Footer is checked in postWrite - much later class descriptor check.
> >
> > Well, my understanding was a binary object with compact footer = false is
> > totally standalone entity and can be read without any external metadata.
> > Dmitrii Ryabov can you double check ?
> >
> > If we support only strings it will be necessary to encode binary values
> to
> > something like BASE64 which is not sounds good from usability side.
> >
> >
> >
> > чт, 23 янв. 2020 г. в 13:26, Nikita Amelchev :
> >
> > > +1 for the hardcoded String type only
> > >
> > > чт, 23 янв. 2020 г. в 13:15, Pavel Tupitsyn :
> > > >
> > > > - Cross-platform binary objects are totally possible, all those thin
> > > > clients support them.
> > > > - User attributes can be useful, no objections here
> > > >
> > > > However, I don't think we should allow arbitrary objects in user
> > > attributes.
> > > > Let's make them string only, much less to worry about.
> > > >
> > > > And using attributes for authentication still seems weird and dirty.
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Jan 23, 2020 at 12:40 PM Dmitrii Ryabov <
> somefire...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > > Even if compact footer is disabled ?
> > > > > Footer is checked in postWrite - much later class descriptor check.
> > > > >
> > > > > чт, 23 янв. 2020 г., 12:23 Alexei Scherbakov <
> > > alexey.scherbak...@gmail.com
> > > > > >:
> > > > >
> > > > > > чт, 23 янв. 2020 г. в 12:17, Dmitrii Ryabov <
> somefire...@gmail.com
> > >:
> > > > > >
> > > > > > > > The protocol must be language-agnostic. If we add some
> features
> > > > > there,
> > > > > > > let's make sure they are usable from anywhere.
> > > > > > >
> > > > > > > That's why I want to allow primitives only. Any language can
> send
> > > > > numbers
> > > > > > > and strings.
> > > > > > >
> > > > > >
> > > > > > In general it's possible to have cross-platform complex data
> > > structures,
> > > > > > for example see protobuf.
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > Binary marshaller, before packing object to byte[], will try to
> > use
> > > > > > > discovery processor and send message containing class
> descriptor.
> > > But
> > > > > > thin
> > > > > > > clients don't have discovery. Furthermore, if we write binary
> > > > > marshaller
> > > > > > > without class descriptor synchronization, we can get objects
> with
> > > > > > different
> > > > > > > class versions and corresponding exceptions.
> > > > > > >
> > > > > >
> > > > > > Even if compact footer is disabled ?
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > But we can say users to declare their classes in
> > > > > > > META-INF/classnames.properties and current binary marshaller
> will
> > > works
> > > > > > > good.
> > > > > > >
> > > > > >
> > > > > > This approach doesn't looks like cross-platform.
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > чт, 23 янв. 2020 г., 12:13 Alex Plehanov <
> > plehanov.a...@gmail.com
> > > >:
> > > > > > >
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > > User attributes also (besides authentication) can be used to
> > pass
> > > > > some
> > > > > > > info
> > > > > > > > about an application that uses a client and then display this
> > > > > > information
> > > > > > > > in monitoring tools. Other vendors use such approach (Oracle
> > DB,
> > > for
> > > > > > > > example, have DBMS_APPLICATION_INFO package, PostgreeSQL have
> > > > > > > > application_name connection property and application
> > information
> > > > > > > available
> > > > > > > > later in system views).
> > > > > > > >
> > > > > > > > About allowed data types: we should definitely limit
> attribute
> > > types
> > > > > to
> > > > > > > > only primitive types. Thin client binary marshaller can't
> send
> > > > > > > information
> > > > > > > > about custom types before the handshake.
> > > > > > > >
> > > > > > > >
> > > > > > > > ср, 22 янв. 2020 г. в 21:39, Pavel Tupitsyn <
> > > ptupit...@apache.org>:
> > > > > > > >
> > > > > > > > > I've looked through the PR more closely, trying to
> understand
> > > the
> > > > > use
> > > > > > > > case,
> > > > > > > > > and there are some Java-specific things going on (left a
> > > comment).
> > > > > > > > > Please keep in mind that we have thin clients in Python,
> > > Node.js,
> > > > > > C++,
> > > > > > > > C#.
> > > > > > > 

Re: Add user attributes to thin clients

2020-01-23 Thread Pavel Tupitsyn
> If we support only strings it will be necessary to encode binary values to
> something like BASE64 which is not sounds good from usability side

There should be no need to put binary values to attributes. What's the use
case?

On Thu, Jan 23, 2020 at 2:08 PM Alexei Scherbakov <
alexey.scherbak...@gmail.com> wrote:

> > Footer is checked in postWrite - much later class descriptor check.
>
> Well, my understanding was a binary object with compact footer = false is
> totally standalone entity and can be read without any external metadata.
> Dmitrii Ryabov can you double check ?
>
> If we support only strings it will be necessary to encode binary values to
> something like BASE64 which is not sounds good from usability side.
>
>
>
> чт, 23 янв. 2020 г. в 13:26, Nikita Amelchev :
>
> > +1 for the hardcoded String type only
> >
> > чт, 23 янв. 2020 г. в 13:15, Pavel Tupitsyn :
> > >
> > > - Cross-platform binary objects are totally possible, all those thin
> > > clients support them.
> > > - User attributes can be useful, no objections here
> > >
> > > However, I don't think we should allow arbitrary objects in user
> > attributes.
> > > Let's make them string only, much less to worry about.
> > >
> > > And using attributes for authentication still seems weird and dirty.
> > >
> > >
> > >
> > >
> > > On Thu, Jan 23, 2020 at 12:40 PM Dmitrii Ryabov  >
> > > wrote:
> > >
> > > > > Even if compact footer is disabled ?
> > > > Footer is checked in postWrite - much later class descriptor check.
> > > >
> > > > чт, 23 янв. 2020 г., 12:23 Alexei Scherbakov <
> > alexey.scherbak...@gmail.com
> > > > >:
> > > >
> > > > > чт, 23 янв. 2020 г. в 12:17, Dmitrii Ryabov  >:
> > > > >
> > > > > > > The protocol must be language-agnostic. If we add some features
> > > > there,
> > > > > > let's make sure they are usable from anywhere.
> > > > > >
> > > > > > That's why I want to allow primitives only. Any language can send
> > > > numbers
> > > > > > and strings.
> > > > > >
> > > > >
> > > > > In general it's possible to have cross-platform complex data
> > structures,
> > > > > for example see protobuf.
> > > > >
> > > > >
> > > > > >
> > > > > > Binary marshaller, before packing object to byte[], will try to
> use
> > > > > > discovery processor and send message containing class descriptor.
> > But
> > > > > thin
> > > > > > clients don't have discovery. Furthermore, if we write binary
> > > > marshaller
> > > > > > without class descriptor synchronization, we can get objects with
> > > > > different
> > > > > > class versions and corresponding exceptions.
> > > > > >
> > > > >
> > > > > Even if compact footer is disabled ?
> > > > >
> > > > >
> > > > > >
> > > > > > But we can say users to declare their classes in
> > > > > > META-INF/classnames.properties and current binary marshaller will
> > works
> > > > > > good.
> > > > > >
> > > > >
> > > > > This approach doesn't looks like cross-platform.
> > > > >
> > > > >
> > > > > >
> > > > > > чт, 23 янв. 2020 г., 12:13 Alex Plehanov <
> plehanov.a...@gmail.com
> > >:
> > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > > User attributes also (besides authentication) can be used to
> pass
> > > > some
> > > > > > info
> > > > > > > about an application that uses a client and then display this
> > > > > information
> > > > > > > in monitoring tools. Other vendors use such approach (Oracle
> DB,
> > for
> > > > > > > example, have DBMS_APPLICATION_INFO package, PostgreeSQL have
> > > > > > > application_name connection property and application
> information
> > > > > > available
> > > > > > > later in system views).
> > > > > > >
> > > > > > > About allowed data types: we should definitely limit attribute
> > types
> > > > to
> > > > > > > only primitive types. Thin client binary marshaller can't send
> > > > > > information
> > > > > > > about custom types before the handshake.
> > > > > > >
> > > > > > >
> > > > > > > ср, 22 янв. 2020 г. в 21:39, Pavel Tupitsyn <
> > ptupit...@apache.org>:
> > > > > > >
> > > > > > > > I've looked through the PR more closely, trying to understand
> > the
> > > > use
> > > > > > > case,
> > > > > > > > and there are some Java-specific things going on (left a
> > comment).
> > > > > > > > Please keep in mind that we have thin clients in Python,
> > Node.js,
> > > > > C++,
> > > > > > > C#.
> > > > > > > > The protocol must be language-agnostic. If we add some
> features
> > > > > there,
> > > > > > > > let's make sure they are usable from anywhere.
> > > > > > > >
> > > > > > > > On Wed, Jan 22, 2020 at 9:21 PM Pavel Tupitsyn <
> > > > ptupit...@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > The approach with UserAttributes map looks dirty to me and
> > raises
> > > > > > > > > questions:
> > > > > > > > >
> > > > > > > > > - Why is UserAttributes property related to authentication?
> > > > > > > > > - UserAttributes name implies that users can put there
> > anything
> > > > > they
> > > > > > > > want,
> 

Re: Add user attributes to thin clients

2020-01-23 Thread Alexei Scherbakov
> Footer is checked in postWrite - much later class descriptor check.

Well, my understanding was a binary object with compact footer = false is
totally standalone entity and can be read without any external metadata.
Dmitrii Ryabov can you double check ?

If we support only strings it will be necessary to encode binary values to
something like BASE64 which is not sounds good from usability side.



чт, 23 янв. 2020 г. в 13:26, Nikita Amelchev :

> +1 for the hardcoded String type only
>
> чт, 23 янв. 2020 г. в 13:15, Pavel Tupitsyn :
> >
> > - Cross-platform binary objects are totally possible, all those thin
> > clients support them.
> > - User attributes can be useful, no objections here
> >
> > However, I don't think we should allow arbitrary objects in user
> attributes.
> > Let's make them string only, much less to worry about.
> >
> > And using attributes for authentication still seems weird and dirty.
> >
> >
> >
> >
> > On Thu, Jan 23, 2020 at 12:40 PM Dmitrii Ryabov 
> > wrote:
> >
> > > > Even if compact footer is disabled ?
> > > Footer is checked in postWrite - much later class descriptor check.
> > >
> > > чт, 23 янв. 2020 г., 12:23 Alexei Scherbakov <
> alexey.scherbak...@gmail.com
> > > >:
> > >
> > > > чт, 23 янв. 2020 г. в 12:17, Dmitrii Ryabov :
> > > >
> > > > > > The protocol must be language-agnostic. If we add some features
> > > there,
> > > > > let's make sure they are usable from anywhere.
> > > > >
> > > > > That's why I want to allow primitives only. Any language can send
> > > numbers
> > > > > and strings.
> > > > >
> > > >
> > > > In general it's possible to have cross-platform complex data
> structures,
> > > > for example see protobuf.
> > > >
> > > >
> > > > >
> > > > > Binary marshaller, before packing object to byte[], will try to use
> > > > > discovery processor and send message containing class descriptor.
> But
> > > > thin
> > > > > clients don't have discovery. Furthermore, if we write binary
> > > marshaller
> > > > > without class descriptor synchronization, we can get objects with
> > > > different
> > > > > class versions and corresponding exceptions.
> > > > >
> > > >
> > > > Even if compact footer is disabled ?
> > > >
> > > >
> > > > >
> > > > > But we can say users to declare their classes in
> > > > > META-INF/classnames.properties and current binary marshaller will
> works
> > > > > good.
> > > > >
> > > >
> > > > This approach doesn't looks like cross-platform.
> > > >
> > > >
> > > > >
> > > > > чт, 23 янв. 2020 г., 12:13 Alex Plehanov  >:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > User attributes also (besides authentication) can be used to pass
> > > some
> > > > > info
> > > > > > about an application that uses a client and then display this
> > > > information
> > > > > > in monitoring tools. Other vendors use such approach (Oracle DB,
> for
> > > > > > example, have DBMS_APPLICATION_INFO package, PostgreeSQL have
> > > > > > application_name connection property and application information
> > > > > available
> > > > > > later in system views).
> > > > > >
> > > > > > About allowed data types: we should definitely limit attribute
> types
> > > to
> > > > > > only primitive types. Thin client binary marshaller can't send
> > > > > information
> > > > > > about custom types before the handshake.
> > > > > >
> > > > > >
> > > > > > ср, 22 янв. 2020 г. в 21:39, Pavel Tupitsyn <
> ptupit...@apache.org>:
> > > > > >
> > > > > > > I've looked through the PR more closely, trying to understand
> the
> > > use
> > > > > > case,
> > > > > > > and there are some Java-specific things going on (left a
> comment).
> > > > > > > Please keep in mind that we have thin clients in Python,
> Node.js,
> > > > C++,
> > > > > > C#.
> > > > > > > The protocol must be language-agnostic. If we add some features
> > > > there,
> > > > > > > let's make sure they are usable from anywhere.
> > > > > > >
> > > > > > > On Wed, Jan 22, 2020 at 9:21 PM Pavel Tupitsyn <
> > > ptupit...@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > The approach with UserAttributes map looks dirty to me and
> raises
> > > > > > > > questions:
> > > > > > > >
> > > > > > > > - Why is UserAttributes property related to authentication?
> > > > > > > > - UserAttributes name implies that users can put there
> anything
> > > > they
> > > > > > > want,
> > > > > > > > but what for? What are those additional use cases?
> > > > > > > >
> > > > > > > > I think we should focus on a specific problem at hand and
> avoid
> > > > > > > > unnecessary future-proofing.
> > > > > > > > What are current and potential future custom authenticators
> and
> > > > what
> > > > > > kind
> > > > > > > > of data do they need?
> > > > > > > >
> > > > > > > > On Wed, Jan 22, 2020 at 6:28 PM Nikita Amelchev <
> > > > > nsamelc...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> I think we should add this. It will provide an extra level
> of
> > > > > > security.
> > > > > > > >> 

Re: Add user attributes to thin clients

2020-01-23 Thread Nikita Amelchev
+1 for the hardcoded String type only

чт, 23 янв. 2020 г. в 13:15, Pavel Tupitsyn :
>
> - Cross-platform binary objects are totally possible, all those thin
> clients support them.
> - User attributes can be useful, no objections here
>
> However, I don't think we should allow arbitrary objects in user attributes.
> Let's make them string only, much less to worry about.
>
> And using attributes for authentication still seems weird and dirty.
>
>
>
>
> On Thu, Jan 23, 2020 at 12:40 PM Dmitrii Ryabov 
> wrote:
>
> > > Even if compact footer is disabled ?
> > Footer is checked in postWrite - much later class descriptor check.
> >
> > чт, 23 янв. 2020 г., 12:23 Alexei Scherbakov  > >:
> >
> > > чт, 23 янв. 2020 г. в 12:17, Dmitrii Ryabov :
> > >
> > > > > The protocol must be language-agnostic. If we add some features
> > there,
> > > > let's make sure they are usable from anywhere.
> > > >
> > > > That's why I want to allow primitives only. Any language can send
> > numbers
> > > > and strings.
> > > >
> > >
> > > In general it's possible to have cross-platform complex data structures,
> > > for example see protobuf.
> > >
> > >
> > > >
> > > > Binary marshaller, before packing object to byte[], will try to use
> > > > discovery processor and send message containing class descriptor. But
> > > thin
> > > > clients don't have discovery. Furthermore, if we write binary
> > marshaller
> > > > without class descriptor synchronization, we can get objects with
> > > different
> > > > class versions and corresponding exceptions.
> > > >
> > >
> > > Even if compact footer is disabled ?
> > >
> > >
> > > >
> > > > But we can say users to declare their classes in
> > > > META-INF/classnames.properties and current binary marshaller will works
> > > > good.
> > > >
> > >
> > > This approach doesn't looks like cross-platform.
> > >
> > >
> > > >
> > > > чт, 23 янв. 2020 г., 12:13 Alex Plehanov :
> > > >
> > > > > Hello,
> > > > >
> > > > > User attributes also (besides authentication) can be used to pass
> > some
> > > > info
> > > > > about an application that uses a client and then display this
> > > information
> > > > > in monitoring tools. Other vendors use such approach (Oracle DB, for
> > > > > example, have DBMS_APPLICATION_INFO package, PostgreeSQL have
> > > > > application_name connection property and application information
> > > > available
> > > > > later in system views).
> > > > >
> > > > > About allowed data types: we should definitely limit attribute types
> > to
> > > > > only primitive types. Thin client binary marshaller can't send
> > > > information
> > > > > about custom types before the handshake.
> > > > >
> > > > >
> > > > > ср, 22 янв. 2020 г. в 21:39, Pavel Tupitsyn :
> > > > >
> > > > > > I've looked through the PR more closely, trying to understand the
> > use
> > > > > case,
> > > > > > and there are some Java-specific things going on (left a comment).
> > > > > > Please keep in mind that we have thin clients in Python, Node.js,
> > > C++,
> > > > > C#.
> > > > > > The protocol must be language-agnostic. If we add some features
> > > there,
> > > > > > let's make sure they are usable from anywhere.
> > > > > >
> > > > > > On Wed, Jan 22, 2020 at 9:21 PM Pavel Tupitsyn <
> > ptupit...@apache.org
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > The approach with UserAttributes map looks dirty to me and raises
> > > > > > > questions:
> > > > > > >
> > > > > > > - Why is UserAttributes property related to authentication?
> > > > > > > - UserAttributes name implies that users can put there anything
> > > they
> > > > > > want,
> > > > > > > but what for? What are those additional use cases?
> > > > > > >
> > > > > > > I think we should focus on a specific problem at hand and avoid
> > > > > > > unnecessary future-proofing.
> > > > > > > What are current and potential future custom authenticators and
> > > what
> > > > > kind
> > > > > > > of data do they need?
> > > > > > >
> > > > > > > On Wed, Jan 22, 2020 at 6:28 PM Nikita Amelchev <
> > > > nsamelc...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> I think we should add this. It will provide an extra level of
> > > > > security.
> > > > > > >> This approach is used in many products, for example in AWS
> > (MFA).
> > > > [1]
> > > > > > >>
> > > > > > >> [1]
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> > https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#multi-factor-authentication
> > > > > > >>
> > > > > > >> ср, 22 янв. 2020 г. в 18:13, Andrey Kuznetsov <
> > stku...@gmail.com
> > > >:
> > > > > > >> >
> > > > > > >> > Hi, Pavel!
> > > > > > >> >
> > > > > > >> > Sometimes single authentication factor is not enough.
> > Attributes
> > > > > > >> proposed
> > > > > > >> > allow to add extra factors flexibly.
> > > > > > >> >
> > > > > > >> > ср, 22 янв. 2020 г., 17:39 Pavel Tupitsyn <
> > ptupit...@apache.org
> > > >:
> > > > > > >> >
> > > > > > >> > > Token can be sent instead of a password 

Re: Add user attributes to thin clients

2020-01-23 Thread Pavel Tupitsyn
- Cross-platform binary objects are totally possible, all those thin
clients support them.
- User attributes can be useful, no objections here

However, I don't think we should allow arbitrary objects in user attributes.
Let's make them string only, much less to worry about.

And using attributes for authentication still seems weird and dirty.




On Thu, Jan 23, 2020 at 12:40 PM Dmitrii Ryabov 
wrote:

> > Even if compact footer is disabled ?
> Footer is checked in postWrite - much later class descriptor check.
>
> чт, 23 янв. 2020 г., 12:23 Alexei Scherbakov  >:
>
> > чт, 23 янв. 2020 г. в 12:17, Dmitrii Ryabov :
> >
> > > > The protocol must be language-agnostic. If we add some features
> there,
> > > let's make sure they are usable from anywhere.
> > >
> > > That's why I want to allow primitives only. Any language can send
> numbers
> > > and strings.
> > >
> >
> > In general it's possible to have cross-platform complex data structures,
> > for example see protobuf.
> >
> >
> > >
> > > Binary marshaller, before packing object to byte[], will try to use
> > > discovery processor and send message containing class descriptor. But
> > thin
> > > clients don't have discovery. Furthermore, if we write binary
> marshaller
> > > without class descriptor synchronization, we can get objects with
> > different
> > > class versions and corresponding exceptions.
> > >
> >
> > Even if compact footer is disabled ?
> >
> >
> > >
> > > But we can say users to declare their classes in
> > > META-INF/classnames.properties and current binary marshaller will works
> > > good.
> > >
> >
> > This approach doesn't looks like cross-platform.
> >
> >
> > >
> > > чт, 23 янв. 2020 г., 12:13 Alex Plehanov :
> > >
> > > > Hello,
> > > >
> > > > User attributes also (besides authentication) can be used to pass
> some
> > > info
> > > > about an application that uses a client and then display this
> > information
> > > > in monitoring tools. Other vendors use such approach (Oracle DB, for
> > > > example, have DBMS_APPLICATION_INFO package, PostgreeSQL have
> > > > application_name connection property and application information
> > > available
> > > > later in system views).
> > > >
> > > > About allowed data types: we should definitely limit attribute types
> to
> > > > only primitive types. Thin client binary marshaller can't send
> > > information
> > > > about custom types before the handshake.
> > > >
> > > >
> > > > ср, 22 янв. 2020 г. в 21:39, Pavel Tupitsyn :
> > > >
> > > > > I've looked through the PR more closely, trying to understand the
> use
> > > > case,
> > > > > and there are some Java-specific things going on (left a comment).
> > > > > Please keep in mind that we have thin clients in Python, Node.js,
> > C++,
> > > > C#.
> > > > > The protocol must be language-agnostic. If we add some features
> > there,
> > > > > let's make sure they are usable from anywhere.
> > > > >
> > > > > On Wed, Jan 22, 2020 at 9:21 PM Pavel Tupitsyn <
> ptupit...@apache.org
> > >
> > > > > wrote:
> > > > >
> > > > > > The approach with UserAttributes map looks dirty to me and raises
> > > > > > questions:
> > > > > >
> > > > > > - Why is UserAttributes property related to authentication?
> > > > > > - UserAttributes name implies that users can put there anything
> > they
> > > > > want,
> > > > > > but what for? What are those additional use cases?
> > > > > >
> > > > > > I think we should focus on a specific problem at hand and avoid
> > > > > > unnecessary future-proofing.
> > > > > > What are current and potential future custom authenticators and
> > what
> > > > kind
> > > > > > of data do they need?
> > > > > >
> > > > > > On Wed, Jan 22, 2020 at 6:28 PM Nikita Amelchev <
> > > nsamelc...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> I think we should add this. It will provide an extra level of
> > > > security.
> > > > > >> This approach is used in many products, for example in AWS
> (MFA).
> > > [1]
> > > > > >>
> > > > > >> [1]
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#multi-factor-authentication
> > > > > >>
> > > > > >> ср, 22 янв. 2020 г. в 18:13, Andrey Kuznetsov <
> stku...@gmail.com
> > >:
> > > > > >> >
> > > > > >> > Hi, Pavel!
> > > > > >> >
> > > > > >> > Sometimes single authentication factor is not enough.
> Attributes
> > > > > >> proposed
> > > > > >> > allow to add extra factors flexibly.
> > > > > >> >
> > > > > >> > ср, 22 янв. 2020 г., 17:39 Pavel Tupitsyn <
> ptupit...@apache.org
> > >:
> > > > > >> >
> > > > > >> > > Token can be sent instead of a password (like git works with
> > > > GitHub
> > > > > >> > > tokens).
> > > > > >> > >
> > > > > >> > > For now I don't see a reason to include attributes into the
> > > > > handshake
> > > > > >> > > message.
> > > > > >> > >
> > > > > >> > > On Wed, Jan 22, 2020 at 5:32 PM Ilya Kasnacheev <
> > > > > >> ilya.kasnach...@gmail.com
> > > > > >> > > >
> > > > > >> > > wrote:
> > > > > >> 

Re: Add user attributes to thin clients

2020-01-23 Thread Dmitrii Ryabov
> Even if compact footer is disabled ?
Footer is checked in postWrite - much later class descriptor check.

чт, 23 янв. 2020 г., 12:23 Alexei Scherbakov :

> чт, 23 янв. 2020 г. в 12:17, Dmitrii Ryabov :
>
> > > The protocol must be language-agnostic. If we add some features there,
> > let's make sure they are usable from anywhere.
> >
> > That's why I want to allow primitives only. Any language can send numbers
> > and strings.
> >
>
> In general it's possible to have cross-platform complex data structures,
> for example see protobuf.
>
>
> >
> > Binary marshaller, before packing object to byte[], will try to use
> > discovery processor and send message containing class descriptor. But
> thin
> > clients don't have discovery. Furthermore, if we write binary marshaller
> > without class descriptor synchronization, we can get objects with
> different
> > class versions and corresponding exceptions.
> >
>
> Even if compact footer is disabled ?
>
>
> >
> > But we can say users to declare their classes in
> > META-INF/classnames.properties and current binary marshaller will works
> > good.
> >
>
> This approach doesn't looks like cross-platform.
>
>
> >
> > чт, 23 янв. 2020 г., 12:13 Alex Plehanov :
> >
> > > Hello,
> > >
> > > User attributes also (besides authentication) can be used to pass some
> > info
> > > about an application that uses a client and then display this
> information
> > > in monitoring tools. Other vendors use such approach (Oracle DB, for
> > > example, have DBMS_APPLICATION_INFO package, PostgreeSQL have
> > > application_name connection property and application information
> > available
> > > later in system views).
> > >
> > > About allowed data types: we should definitely limit attribute types to
> > > only primitive types. Thin client binary marshaller can't send
> > information
> > > about custom types before the handshake.
> > >
> > >
> > > ср, 22 янв. 2020 г. в 21:39, Pavel Tupitsyn :
> > >
> > > > I've looked through the PR more closely, trying to understand the use
> > > case,
> > > > and there are some Java-specific things going on (left a comment).
> > > > Please keep in mind that we have thin clients in Python, Node.js,
> C++,
> > > C#.
> > > > The protocol must be language-agnostic. If we add some features
> there,
> > > > let's make sure they are usable from anywhere.
> > > >
> > > > On Wed, Jan 22, 2020 at 9:21 PM Pavel Tupitsyn  >
> > > > wrote:
> > > >
> > > > > The approach with UserAttributes map looks dirty to me and raises
> > > > > questions:
> > > > >
> > > > > - Why is UserAttributes property related to authentication?
> > > > > - UserAttributes name implies that users can put there anything
> they
> > > > want,
> > > > > but what for? What are those additional use cases?
> > > > >
> > > > > I think we should focus on a specific problem at hand and avoid
> > > > > unnecessary future-proofing.
> > > > > What are current and potential future custom authenticators and
> what
> > > kind
> > > > > of data do they need?
> > > > >
> > > > > On Wed, Jan 22, 2020 at 6:28 PM Nikita Amelchev <
> > nsamelc...@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> I think we should add this. It will provide an extra level of
> > > security.
> > > > >> This approach is used in many products, for example in AWS (MFA).
> > [1]
> > > > >>
> > > > >> [1]
> > > > >>
> > > >
> > >
> >
> https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#multi-factor-authentication
> > > > >>
> > > > >> ср, 22 янв. 2020 г. в 18:13, Andrey Kuznetsov  >:
> > > > >> >
> > > > >> > Hi, Pavel!
> > > > >> >
> > > > >> > Sometimes single authentication factor is not enough. Attributes
> > > > >> proposed
> > > > >> > allow to add extra factors flexibly.
> > > > >> >
> > > > >> > ср, 22 янв. 2020 г., 17:39 Pavel Tupitsyn  >:
> > > > >> >
> > > > >> > > Token can be sent instead of a password (like git works with
> > > GitHub
> > > > >> > > tokens).
> > > > >> > >
> > > > >> > > For now I don't see a reason to include attributes into the
> > > > handshake
> > > > >> > > message.
> > > > >> > >
> > > > >> > > On Wed, Jan 22, 2020 at 5:32 PM Ilya Kasnacheev <
> > > > >> ilya.kasnach...@gmail.com
> > > > >> > > >
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > Hello!
> > > > >> > > >
> > > > >> > > > One does not send security certificate as attribute. The
> only
> > > way
> > > > to
> > > > >> > > obtain
> > > > >> > > > peer security certificate is to ask SSL engine to provide
> it.
> > > > >> > > >
> > > > >> > > > Nevertheless, I can see how it can be useful with e.g.
> > Kerberos,
> > > > >> which is
> > > > >> > > > token-based IIRC.
> > > > >> > > >
> > > > >> > > > Regards,
> > > > >> > > > --
> > > > >> > > > Ilya Kasnacheev
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > ср, 22 янв. 2020 г. в 17:20, Dmitrii Ryabov <
> > > > somefire...@gmail.com
> > > > >> >:
> > > > >> > > >
> > > > >> > > > > This map is something like user object from
> > > > `SecurityCredentials`.
> > > > >> > 

Re: Add user attributes to thin clients

2020-01-23 Thread Alexei Scherbakov
чт, 23 янв. 2020 г. в 12:17, Dmitrii Ryabov :

> > The protocol must be language-agnostic. If we add some features there,
> let's make sure they are usable from anywhere.
>
> That's why I want to allow primitives only. Any language can send numbers
> and strings.
>

In general it's possible to have cross-platform complex data structures,
for example see protobuf.


>
> Binary marshaller, before packing object to byte[], will try to use
> discovery processor and send message containing class descriptor. But thin
> clients don't have discovery. Furthermore, if we write binary marshaller
> without class descriptor synchronization, we can get objects with different
> class versions and corresponding exceptions.
>

Even if compact footer is disabled ?


>
> But we can say users to declare their classes in
> META-INF/classnames.properties and current binary marshaller will works
> good.
>

This approach doesn't looks like cross-platform.


>
> чт, 23 янв. 2020 г., 12:13 Alex Plehanov :
>
> > Hello,
> >
> > User attributes also (besides authentication) can be used to pass some
> info
> > about an application that uses a client and then display this information
> > in monitoring tools. Other vendors use such approach (Oracle DB, for
> > example, have DBMS_APPLICATION_INFO package, PostgreeSQL have
> > application_name connection property and application information
> available
> > later in system views).
> >
> > About allowed data types: we should definitely limit attribute types to
> > only primitive types. Thin client binary marshaller can't send
> information
> > about custom types before the handshake.
> >
> >
> > ср, 22 янв. 2020 г. в 21:39, Pavel Tupitsyn :
> >
> > > I've looked through the PR more closely, trying to understand the use
> > case,
> > > and there are some Java-specific things going on (left a comment).
> > > Please keep in mind that we have thin clients in Python, Node.js, C++,
> > C#.
> > > The protocol must be language-agnostic. If we add some features there,
> > > let's make sure they are usable from anywhere.
> > >
> > > On Wed, Jan 22, 2020 at 9:21 PM Pavel Tupitsyn 
> > > wrote:
> > >
> > > > The approach with UserAttributes map looks dirty to me and raises
> > > > questions:
> > > >
> > > > - Why is UserAttributes property related to authentication?
> > > > - UserAttributes name implies that users can put there anything they
> > > want,
> > > > but what for? What are those additional use cases?
> > > >
> > > > I think we should focus on a specific problem at hand and avoid
> > > > unnecessary future-proofing.
> > > > What are current and potential future custom authenticators and what
> > kind
> > > > of data do they need?
> > > >
> > > > On Wed, Jan 22, 2020 at 6:28 PM Nikita Amelchev <
> nsamelc...@gmail.com>
> > > > wrote:
> > > >
> > > >> I think we should add this. It will provide an extra level of
> > security.
> > > >> This approach is used in many products, for example in AWS (MFA).
> [1]
> > > >>
> > > >> [1]
> > > >>
> > >
> >
> https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#multi-factor-authentication
> > > >>
> > > >> ср, 22 янв. 2020 г. в 18:13, Andrey Kuznetsov :
> > > >> >
> > > >> > Hi, Pavel!
> > > >> >
> > > >> > Sometimes single authentication factor is not enough. Attributes
> > > >> proposed
> > > >> > allow to add extra factors flexibly.
> > > >> >
> > > >> > ср, 22 янв. 2020 г., 17:39 Pavel Tupitsyn :
> > > >> >
> > > >> > > Token can be sent instead of a password (like git works with
> > GitHub
> > > >> > > tokens).
> > > >> > >
> > > >> > > For now I don't see a reason to include attributes into the
> > > handshake
> > > >> > > message.
> > > >> > >
> > > >> > > On Wed, Jan 22, 2020 at 5:32 PM Ilya Kasnacheev <
> > > >> ilya.kasnach...@gmail.com
> > > >> > > >
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Hello!
> > > >> > > >
> > > >> > > > One does not send security certificate as attribute. The only
> > way
> > > to
> > > >> > > obtain
> > > >> > > > peer security certificate is to ask SSL engine to provide it.
> > > >> > > >
> > > >> > > > Nevertheless, I can see how it can be useful with e.g.
> Kerberos,
> > > >> which is
> > > >> > > > token-based IIRC.
> > > >> > > >
> > > >> > > > Regards,
> > > >> > > > --
> > > >> > > > Ilya Kasnacheev
> > > >> > > >
> > > >> > > >
> > > >> > > > ср, 22 янв. 2020 г. в 17:20, Dmitrii Ryabov <
> > > somefire...@gmail.com
> > > >> >:
> > > >> > > >
> > > >> > > > > This map is something like user object from
> > > `SecurityCredentials`.
> > > >> > > > > Sometimes login and password are not enough for security
> > checks.
> > > >> For
> > > >> > > > > example, we can send security certificate and validate it
> > inside
> > > >> > > > > authenticator.
> > > >> > > > >
> > > >> > > > > ср, 22 янв. 2020 г., 17:16 Igor Sapego  >:
> > > >> > > > >
> > > >> > > > > > Hi Dmitrii,
> > > >> > > > > >
> > > >> > > > > > Can you please explain your use case?
> > > >> > > > > > I'm not sure I'm getting what 

Re: Add user attributes to thin clients

2020-01-23 Thread Dmitrii Ryabov
> The protocol must be language-agnostic. If we add some features there,
let's make sure they are usable from anywhere.

That's why I want to allow primitives only. Any language can send numbers
and strings.

Binary marshaller, before packing object to byte[], will try to use
discovery processor and send message containing class descriptor. But thin
clients don't have discovery. Furthermore, if we write binary marshaller
without class descriptor synchronization, we can get objects with different
class versions and corresponding exceptions.

But we can say users to declare their classes in
META-INF/classnames.properties and current binary marshaller will works
good.

чт, 23 янв. 2020 г., 12:13 Alex Plehanov :

> Hello,
>
> User attributes also (besides authentication) can be used to pass some info
> about an application that uses a client and then display this information
> in monitoring tools. Other vendors use such approach (Oracle DB, for
> example, have DBMS_APPLICATION_INFO package, PostgreeSQL have
> application_name connection property and application information available
> later in system views).
>
> About allowed data types: we should definitely limit attribute types to
> only primitive types. Thin client binary marshaller can't send information
> about custom types before the handshake.
>
>
> ср, 22 янв. 2020 г. в 21:39, Pavel Tupitsyn :
>
> > I've looked through the PR more closely, trying to understand the use
> case,
> > and there are some Java-specific things going on (left a comment).
> > Please keep in mind that we have thin clients in Python, Node.js, C++,
> C#.
> > The protocol must be language-agnostic. If we add some features there,
> > let's make sure they are usable from anywhere.
> >
> > On Wed, Jan 22, 2020 at 9:21 PM Pavel Tupitsyn 
> > wrote:
> >
> > > The approach with UserAttributes map looks dirty to me and raises
> > > questions:
> > >
> > > - Why is UserAttributes property related to authentication?
> > > - UserAttributes name implies that users can put there anything they
> > want,
> > > but what for? What are those additional use cases?
> > >
> > > I think we should focus on a specific problem at hand and avoid
> > > unnecessary future-proofing.
> > > What are current and potential future custom authenticators and what
> kind
> > > of data do they need?
> > >
> > > On Wed, Jan 22, 2020 at 6:28 PM Nikita Amelchev 
> > > wrote:
> > >
> > >> I think we should add this. It will provide an extra level of
> security.
> > >> This approach is used in many products, for example in AWS (MFA). [1]
> > >>
> > >> [1]
> > >>
> >
> https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#multi-factor-authentication
> > >>
> > >> ср, 22 янв. 2020 г. в 18:13, Andrey Kuznetsov :
> > >> >
> > >> > Hi, Pavel!
> > >> >
> > >> > Sometimes single authentication factor is not enough. Attributes
> > >> proposed
> > >> > allow to add extra factors flexibly.
> > >> >
> > >> > ср, 22 янв. 2020 г., 17:39 Pavel Tupitsyn :
> > >> >
> > >> > > Token can be sent instead of a password (like git works with
> GitHub
> > >> > > tokens).
> > >> > >
> > >> > > For now I don't see a reason to include attributes into the
> > handshake
> > >> > > message.
> > >> > >
> > >> > > On Wed, Jan 22, 2020 at 5:32 PM Ilya Kasnacheev <
> > >> ilya.kasnach...@gmail.com
> > >> > > >
> > >> > > wrote:
> > >> > >
> > >> > > > Hello!
> > >> > > >
> > >> > > > One does not send security certificate as attribute. The only
> way
> > to
> > >> > > obtain
> > >> > > > peer security certificate is to ask SSL engine to provide it.
> > >> > > >
> > >> > > > Nevertheless, I can see how it can be useful with e.g. Kerberos,
> > >> which is
> > >> > > > token-based IIRC.
> > >> > > >
> > >> > > > Regards,
> > >> > > > --
> > >> > > > Ilya Kasnacheev
> > >> > > >
> > >> > > >
> > >> > > > ср, 22 янв. 2020 г. в 17:20, Dmitrii Ryabov <
> > somefire...@gmail.com
> > >> >:
> > >> > > >
> > >> > > > > This map is something like user object from
> > `SecurityCredentials`.
> > >> > > > > Sometimes login and password are not enough for security
> checks.
> > >> For
> > >> > > > > example, we can send security certificate and validate it
> inside
> > >> > > > > authenticator.
> > >> > > > >
> > >> > > > > ср, 22 янв. 2020 г., 17:16 Igor Sapego :
> > >> > > > >
> > >> > > > > > Hi Dmitrii,
> > >> > > > > >
> > >> > > > > > Can you please explain your use case?
> > >> > > > > > I'm not sure I'm getting what is the motivation of this
> > change.
> > >> > > > > >
> > >> > > > > > Best Regards,
> > >> > > > > > Igor
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On Wed, Jan 22, 2020 at 5:11 PM Pavel Tupitsyn <
> > >> ptupit...@apache.org
> > >> > > >
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Hi Dmitrii,
> > >> > > > > > >
> > >> > > > > > > Honestly, I could not grasp the problem, can you explain
> it
> > >> in more
> > >> > > > > > detail?
> > >> > > > > > > What do we solve by adding a map with arbitrary stuff to
> the

Re: Add user attributes to thin clients

2020-01-23 Thread Alex Plehanov
Hello,

User attributes also (besides authentication) can be used to pass some info
about an application that uses a client and then display this information
in monitoring tools. Other vendors use such approach (Oracle DB, for
example, have DBMS_APPLICATION_INFO package, PostgreeSQL have
application_name connection property and application information available
later in system views).

About allowed data types: we should definitely limit attribute types to
only primitive types. Thin client binary marshaller can't send information
about custom types before the handshake.


ср, 22 янв. 2020 г. в 21:39, Pavel Tupitsyn :

> I've looked through the PR more closely, trying to understand the use case,
> and there are some Java-specific things going on (left a comment).
> Please keep in mind that we have thin clients in Python, Node.js, C++, C#.
> The protocol must be language-agnostic. If we add some features there,
> let's make sure they are usable from anywhere.
>
> On Wed, Jan 22, 2020 at 9:21 PM Pavel Tupitsyn 
> wrote:
>
> > The approach with UserAttributes map looks dirty to me and raises
> > questions:
> >
> > - Why is UserAttributes property related to authentication?
> > - UserAttributes name implies that users can put there anything they
> want,
> > but what for? What are those additional use cases?
> >
> > I think we should focus on a specific problem at hand and avoid
> > unnecessary future-proofing.
> > What are current and potential future custom authenticators and what kind
> > of data do they need?
> >
> > On Wed, Jan 22, 2020 at 6:28 PM Nikita Amelchev 
> > wrote:
> >
> >> I think we should add this. It will provide an extra level of security.
> >> This approach is used in many products, for example in AWS (MFA). [1]
> >>
> >> [1]
> >>
> https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#multi-factor-authentication
> >>
> >> ср, 22 янв. 2020 г. в 18:13, Andrey Kuznetsov :
> >> >
> >> > Hi, Pavel!
> >> >
> >> > Sometimes single authentication factor is not enough. Attributes
> >> proposed
> >> > allow to add extra factors flexibly.
> >> >
> >> > ср, 22 янв. 2020 г., 17:39 Pavel Tupitsyn :
> >> >
> >> > > Token can be sent instead of a password (like git works with GitHub
> >> > > tokens).
> >> > >
> >> > > For now I don't see a reason to include attributes into the
> handshake
> >> > > message.
> >> > >
> >> > > On Wed, Jan 22, 2020 at 5:32 PM Ilya Kasnacheev <
> >> ilya.kasnach...@gmail.com
> >> > > >
> >> > > wrote:
> >> > >
> >> > > > Hello!
> >> > > >
> >> > > > One does not send security certificate as attribute. The only way
> to
> >> > > obtain
> >> > > > peer security certificate is to ask SSL engine to provide it.
> >> > > >
> >> > > > Nevertheless, I can see how it can be useful with e.g. Kerberos,
> >> which is
> >> > > > token-based IIRC.
> >> > > >
> >> > > > Regards,
> >> > > > --
> >> > > > Ilya Kasnacheev
> >> > > >
> >> > > >
> >> > > > ср, 22 янв. 2020 г. в 17:20, Dmitrii Ryabov <
> somefire...@gmail.com
> >> >:
> >> > > >
> >> > > > > This map is something like user object from
> `SecurityCredentials`.
> >> > > > > Sometimes login and password are not enough for security checks.
> >> For
> >> > > > > example, we can send security certificate and validate it inside
> >> > > > > authenticator.
> >> > > > >
> >> > > > > ср, 22 янв. 2020 г., 17:16 Igor Sapego :
> >> > > > >
> >> > > > > > Hi Dmitrii,
> >> > > > > >
> >> > > > > > Can you please explain your use case?
> >> > > > > > I'm not sure I'm getting what is the motivation of this
> change.
> >> > > > > >
> >> > > > > > Best Regards,
> >> > > > > > Igor
> >> > > > > >
> >> > > > > >
> >> > > > > > On Wed, Jan 22, 2020 at 5:11 PM Pavel Tupitsyn <
> >> ptupit...@apache.org
> >> > > >
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Hi Dmitrii,
> >> > > > > > >
> >> > > > > > > Honestly, I could not grasp the problem, can you explain it
> >> in more
> >> > > > > > detail?
> >> > > > > > > What do we solve by adding a map with arbitrary stuff to the
> >> client
> >> > > > > > > protocol handshake?
> >> > > > > > >
> >> > > > > > > On Wed, Jan 22, 2020 at 5:02 PM Dmitrii Ryabov <
> >> > > > somefire...@gmail.com>
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Hello, Igniters!
> >> > > > > > > >
> >> > > > > > > > I want to add the possibility of sending user defined
> >> attributes
> >> > > > from
> >> > > > > > > thin
> >> > > > > > > > clients. And check them inside custom authenticator during
> >> > > > handshake
> >> > > > > > [1].
> >> > > > > > > >
> >> > > > > > > > There is an issue in hardcoded binary writer for JDBC and
> >> > > > > > `IgniteClient`.
> >> > > > > > > > This writer searches for a classes in the JDK and
> >> > > > > > > > META-INF/classnames.properties, and tries to sync
> >> notdeclared
> >> > > > classes
> >> > > > > > > with
> >> > > > > > > > cluster. But fails because current classloading uses
> >> discovery.
> >> > > > > > > >
> >> > > > > > > > I'd like to keep 

Re: Add user attributes to thin clients

2020-01-23 Thread Alexei Scherbakov
Folks,

I did the initial review for a contribution.
The ultimate goal is to have the possibility to pass user defined
attribute-value pairs in all types of clients, same as user attributes in
java configuration [1].
Later they can be used on server side for various things, for example
in authentication.

The problem from my understanding is related to pass these attributes
reliably in multi-platform way.

As Pavel Tupitsyn suggested the protocol must be language-agnostic.

Probably we use binary marshaller with compactfooter=false (or language
specific implementation of binary marshaller) to pack attributes on client
side to byte[] and later unpack them on server side.
Thoughts ?

[1] org.apache.ignite.configuration.IgniteConfiguration#setUserAttributes


ср, 22 янв. 2020 г. в 21:39, Pavel Tupitsyn :

> I've looked through the PR more closely, trying to understand the use case,
> and there are some Java-specific things going on (left a comment).
> Please keep in mind that we have thin clients in Python, Node.js, C++, C#.
> The protocol must be language-agnostic. If we add some features there,
> let's make sure they are usable from anywhere.
>
> On Wed, Jan 22, 2020 at 9:21 PM Pavel Tupitsyn 
> wrote:
>
> > The approach with UserAttributes map looks dirty to me and raises
> > questions:
> >
> > - Why is UserAttributes property related to authentication?
> > - UserAttributes name implies that users can put there anything they
> want,
> > but what for? What are those additional use cases?
> >
> > I think we should focus on a specific problem at hand and avoid
> > unnecessary future-proofing.
> > What are current and potential future custom authenticators and what kind
> > of data do they need?
> >
> > On Wed, Jan 22, 2020 at 6:28 PM Nikita Amelchev 
> > wrote:
> >
> >> I think we should add this. It will provide an extra level of security.
> >> This approach is used in many products, for example in AWS (MFA). [1]
> >>
> >> [1]
> >>
> https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#multi-factor-authentication
> >>
> >> ср, 22 янв. 2020 г. в 18:13, Andrey Kuznetsov :
> >> >
> >> > Hi, Pavel!
> >> >
> >> > Sometimes single authentication factor is not enough. Attributes
> >> proposed
> >> > allow to add extra factors flexibly.
> >> >
> >> > ср, 22 янв. 2020 г., 17:39 Pavel Tupitsyn :
> >> >
> >> > > Token can be sent instead of a password (like git works with GitHub
> >> > > tokens).
> >> > >
> >> > > For now I don't see a reason to include attributes into the
> handshake
> >> > > message.
> >> > >
> >> > > On Wed, Jan 22, 2020 at 5:32 PM Ilya Kasnacheev <
> >> ilya.kasnach...@gmail.com
> >> > > >
> >> > > wrote:
> >> > >
> >> > > > Hello!
> >> > > >
> >> > > > One does not send security certificate as attribute. The only way
> to
> >> > > obtain
> >> > > > peer security certificate is to ask SSL engine to provide it.
> >> > > >
> >> > > > Nevertheless, I can see how it can be useful with e.g. Kerberos,
> >> which is
> >> > > > token-based IIRC.
> >> > > >
> >> > > > Regards,
> >> > > > --
> >> > > > Ilya Kasnacheev
> >> > > >
> >> > > >
> >> > > > ср, 22 янв. 2020 г. в 17:20, Dmitrii Ryabov <
> somefire...@gmail.com
> >> >:
> >> > > >
> >> > > > > This map is something like user object from
> `SecurityCredentials`.
> >> > > > > Sometimes login and password are not enough for security checks.
> >> For
> >> > > > > example, we can send security certificate and validate it inside
> >> > > > > authenticator.
> >> > > > >
> >> > > > > ср, 22 янв. 2020 г., 17:16 Igor Sapego :
> >> > > > >
> >> > > > > > Hi Dmitrii,
> >> > > > > >
> >> > > > > > Can you please explain your use case?
> >> > > > > > I'm not sure I'm getting what is the motivation of this
> change.
> >> > > > > >
> >> > > > > > Best Regards,
> >> > > > > > Igor
> >> > > > > >
> >> > > > > >
> >> > > > > > On Wed, Jan 22, 2020 at 5:11 PM Pavel Tupitsyn <
> >> ptupit...@apache.org
> >> > > >
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Hi Dmitrii,
> >> > > > > > >
> >> > > > > > > Honestly, I could not grasp the problem, can you explain it
> >> in more
> >> > > > > > detail?
> >> > > > > > > What do we solve by adding a map with arbitrary stuff to the
> >> client
> >> > > > > > > protocol handshake?
> >> > > > > > >
> >> > > > > > > On Wed, Jan 22, 2020 at 5:02 PM Dmitrii Ryabov <
> >> > > > somefire...@gmail.com>
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Hello, Igniters!
> >> > > > > > > >
> >> > > > > > > > I want to add the possibility of sending user defined
> >> attributes
> >> > > > from
> >> > > > > > > thin
> >> > > > > > > > clients. And check them inside custom authenticator during
> >> > > > handshake
> >> > > > > > [1].
> >> > > > > > > >
> >> > > > > > > > There is an issue in hardcoded binary writer for JDBC and
> >> > > > > > `IgniteClient`.
> >> > > > > > > > This writer searches for a classes in the JDK and
> >> > > > > > > > META-INF/classnames.properties, and tries to sync
> >> 

[jira] [Created] (IGNITE-12570) What is the root cause of IgniteCheckedException: Invalid handshake message

2020-01-23 Thread shubhendra sen (Jira)
shubhendra sen created IGNITE-12570:
---

 Summary: What is the root cause of IgniteCheckedException: Invalid 
handshake message
 Key: IGNITE-12570
 URL: https://issues.apache.org/jira/browse/IGNITE-12570
 Project: Ignite
  Issue Type: Bug
  Components: examples
Affects Versions: 2.7.5
Reporter: shubhendra sen


Two environments are there. On one environment ignite servers are working 
properly. but on the other environment i am getting an exception.

 

**

Closing NIO session because of unhandled exception.

org.apache.ignite.IgniteCheckedException: Invalid handshake message
 at 
org.apache.ignite.internal.processors.odbc.ClientListenerNioServerBuffer.read(ClientListenerNioServerBuffer.java:114)
 ~[ignite-core-8.7.5.jar!/:8.7.5]
 at 
org.apache.ignite.internal.processors.odbc.ClientListenerBufferedParser.decode(ClientListenerBufferedParser.java:59)
 ~[ignite-core-8.7.5.jar!/:8.7.5]
 at 
org.apache.ignite.internal.processors.odbc.ClientListenerBufferedParser.decode(ClientListenerBufferedParser.java:39)
 ~[ignite-core-8.7.5.jar!/:8.7.5]
 at 
org.apache.ignite.internal.util.nio.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:113)
 ~[ignite-core-8.7.5.jar!/:8.7.5]
 at 
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:108)
 ~[ignite-core-8.7.5.jar!/:8.7.5]
 at 
org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:3575)
 ~[ignite-core-8.7.5.jar!/:8.7.5]
 at 
org.apache.ignite.internal.util.nio.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:174)
 ~[ignite-core-8.7.5.jar!/:8.7.5]
 at 
org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1150)
 ~[ignite-core-8.7.5.jar!/:8.7.5]
 at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2411)
 ~[ignite-core-8.7.5.jar!/:8.7.5]
 at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2178)
 ~[ignite-core-8.7.5.jar!/:8.7.5]
 at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1819)
 ~[ignite-core-8.7.5.jar!/:8.7.5]
 at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119) 
~[ignite-core-8.7.5.jar!/:8.7.5]
 at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

 

So i have two Questions.

1) What could be the possible causes of this exception ?. and 

2)What re the solutions.

 

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]

2020-01-23 Thread Ivan Bessonov
Hi igniters,

there's a potential data corruption fix that I'd like you to include in the
next release:
https://issues.apache.org/jira/browse/IGNITE-12486https://issues.apache.org/jira/browse/IGNITE-12486


Can you please cherry-pick it? Thank you!

ср, 22 янв. 2020 г. в 17:45, Pavel Tupitsyn :

> Good idea about pre-release build of ignite-2.8 branch.
> However, I would not name it `rc`, since it is not really a release
> candidate. Make it `pre0` or something like that.
>
> For Ignite.NET I've uploaded pre-release NuGet packages built from current
> ignite-2.8 branch:
> https://www.nuget.org/packages/Apache.Ignite/2.8.0-alpha20200122
>
>
> On Wed, Jan 22, 2020 at 3:09 PM Ilya Kasnacheev  >
> wrote:
>
> > Hello!
> >
> > I have committed the bumping of essential dependencies' versions:
> > https://issues.apache.org/jira/browse/IGNITE-12540
> >
> > Would you mind including this change into the scope of 2.8? No point of
> > shipping known problematic JARs in our deliverable.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > ср, 22 янв. 2020 г. в 14:00, Maxim Muzafarov :
> >
> > > Alexey,
> > >
> > > Sure, I've just thought about it too a few days ago.
> > >
> > > On Wed, 22 Jan 2020 at 12:09, Anton Vinogradov  wrote:
> > > >
> > > > Good Idea, this will also check that the release process is alive.
> > > >
> > > > On Wed, Jan 22, 2020 at 12:04 PM Alexey Goncharuk <
> > > > alexey.goncha...@gmail.com> wrote:
> > > >
> > > > > Folks, Maxim,
> > > > >
> > > > > Do you mind if I build the current state of ignite-2.8 branch and
> > > upload a
> > > > > maven staging as rc0 (step 4.3.2 of the release process)? I want
> run
> > > some
> > > > > tests for the fixes that are already included to the branch.
> > > > >
> > > > > вт, 21 янв. 2020 г. в 14:28, Maxim Muzafarov :
> > > > >
> > > > > > Folks,
> > > > > >
> > > > > >
> > > > > > I think both of these issues [1] [2] are critical to 2.8 release
> > and
> > > > > > we must include them.
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12547
> > > > > > Excessive AtomicLong instantiations lead to GC pressure.
> > > > > >
> > > > > > [2] https://issues.apache.org/jira/browse/IGNITE-12530
> > > > > > Pages list caching can cause IgniteOOME when the checkpoint is
> > > > > > triggered by "too many dirty pages" reason.
> > > > > >
> > > > > >
> > > > > > On Mon, 20 Jan 2020 at 19:00, Alex Plehanov <
> > plehanov.a...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Guys,
> > > > > > >
> > > > > > > There is an issue [1] caused by page list caching [2], which
> also
> > > > > affects
> > > > > > > 2.8 release. IgniteOutOfMemoryException can be thrown in some
> > cases
> > > > > (data
> > > > > > > region is small, a checkpoint is triggered by "too many dirty
> > > pages"
> > > > > > reason
> > > > > > > and pages list cache is rather big).
> > > > > > > The fix is ready and merged to master, I suggest to include
> this
> > > fix to
> > > > > > 2.8
> > > > > > > release. What do you think?
> > > > > > >
> > > > > > > [1]: https://issues.apache.org/jira/browse/IGNITE-12530
> > > > > > > [2]: https://issues.apache.org/jira/browse/IGNITE-6930
> > > > > > >
> > > > > > > пн, 20 янв. 2020 г. в 12:57, Alexey Goncharuk <
> > > > > > alexey.goncha...@gmail.com>:
> > > > > > >
> > > > > > > > Maxim,
> > > > > > > >
> > > > > > > > I took a quick look at IGNITE-12456 and I am not sure it's
> > about
> > > data
> > > > > > > > corruption. In the attached logs blocked system threads are
> > > reported,
> > > > > > > > however, there is no enough information to investigate the
> > issue
> > > (the
> > > > > > full
> > > > > > > > thread dump was not attached). I asked the ticket creator to
> > > attach
> > > > > > missing
> > > > > > > > pieces.
> > > > > > > >
> > > > > > > > Should we consider moving this ticket to a next release?
> > > > > > > >
> > > > > > > > пн, 20 янв. 2020 г. в 08:54, Zhenya Stanilovsky
> > > > > >  > > > > > > > >:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Maxim, performance fix issue [1] already in master, if no
> > > > > > objections, can
> > > > > > > > > u merge it into 2.8 ? Thanks !
> > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12547
> > > > > > > > >
> > > > > > > > > >Igniters,
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >Here is the actual list of BLOCKER release issues:
> > > > > > > > > >
> > > > > > > > > >IGNITE-12456 Cluster Data Store grid gets Corrupted for
> Load
> > > test
> > > > > > > > > >*[Unassigned]* OPEN
> > > > > > > > > >IGNITE-12489 Error during purges by expiration: Unknown
> page
> > > type*
> > > > > > > > > >[Unassigned]* OPEN
> > > > > > > > > >IGNITE-8641 SpringDataExample should use
> example-ignite.xml
> > > config
> > > > > > > > > >*[Unassigned]* OPEN
> > > > > > > > > >
> > > > > > > > > >IGNITE-12398 Apache Ignite Cluster(Amazon S3 Based
> > Discovery)
> > > > > Nodes
> > > >