[GitHub] [pulsar] sijie commented on issue #3752: revise the schema default type not null

2019-03-04 Thread GitBox
sijie commented on issue #3752: revise the schema default type not null URL: https://github.com/apache/pulsar/pull/3752#issuecomment-469575212 @jerrypeng I am not sure we need to do it for JSON. because there is no *real* schema in JSON data. every field in a json struct is *optional*, no?

[GitHub] [pulsar] sijie edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
sijie edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469572165 > yup though I am not sure setting AllowNull is producing an incompatible schema. Incompatible to what? @jerrypeng `ReflectData.AllowNull`

[GitHub] [pulsar] sijie commented on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
sijie commented on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469572165 > yup though I am not sure setting AllowNull is producing an incompatible schema. Incompatible to what? @jerrypeng `ReflectData.AllowNull` and `R

[GitHub] [pulsar] rdhabalia commented on issue #3732: [pulsar-broker] Optimize message replay for large backlog consumer

2019-03-04 Thread GitBox
rdhabalia commented on issue #3732: [pulsar-broker] Optimize message replay for large backlog consumer URL: https://github.com/apache/pulsar/pull/3732#issuecomment-469570384 rerun java8 tests This is an automated message from

[GitHub] [pulsar] jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469565479 @sijie yup though I am not sure setting AllowNull is producing an incompatible schema. Incompatible to what? I guess if a user wrote some co

[GitHub] [pulsar] jerrypeng commented on issue #3752: revise the schema default type not null

2019-03-04 Thread GitBox
jerrypeng commented on issue #3752: revise the schema default type not null URL: https://github.com/apache/pulsar/pull/3752#issuecomment-469568495 @sijie @congbobo184 @merlimat if we are removing AllowNull or allowing the flag to be configurable, should we also do it for JSONSchema:

[GitHub] [pulsar] jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469565479 @sijie yup though I am not sure setting AllowNull is producing an incompatible schema. Incompatible to what? I guess if a user wrote some co

[GitHub] [pulsar] jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469565479 @sijie yup though I am not sure setting AllowNull is producing an incompatible schema. Incompatible to what? I guess if a user wrote some co

[GitHub] [pulsar] jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469565479 @sijie yup though I am not sure setting AllowNull is producing an incompatible schema. Incompatible to what? I guess if a user wrote some co

[GitHub] [pulsar] jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469565479 @sijie yup though I am not sure setting AllowNull is producing an incompatible schema. Incompatible to what? I guess if a user wrote some co

[GitHub] [pulsar] jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469565479 @sijie yup though I am not sure setting AllowNull is producing an incompatible schema. Incompatible to what? I guess if a user wrote some co

[GitHub] [pulsar] jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469565479 @sijie yup though I am not sure setting AllowNull is producing an incompatible schema. Incompatible to what? I guess if a user wrote some co

[GitHub] [pulsar] jerrypeng commented on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng commented on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469565479 @sijie yup though I am not sure setting AllowNull is producing an incompatible schema. Incompatible to what? I guess if a user wrote some code:

[GitHub] [pulsar] MarvinCai commented on issue #3753: Add support for Kafka's ProducerConfig.CONNECTIONS_MAX_IDLE_MS_CONFIG (#1090)

2019-03-04 Thread GitBox
MarvinCai commented on issue #3753: Add support for Kafka's ProducerConfig.CONNECTIONS_MAX_IDLE_MS_CONFIG (#1090) URL: https://github.com/apache/pulsar/pull/3753#issuecomment-469564914 rerun java8 tests rerun integration tests rerun cpp tests -

[GitHub] [pulsar] MarvinCai opened a new pull request #3753: Add support for Kafka's ProducerConfig.CONNECTIONS_MAX_IDLE_MS_CONFIG (#1090)

2019-03-04 Thread GitBox
MarvinCai opened a new pull request #3753: Add support for Kafka's ProducerConfig.CONNECTIONS_MAX_IDLE_MS_CONFIG (#1090) URL: https://github.com/apache/pulsar/pull/3753 Motivation: Support configure Kafka's ProducerConfig.CONNECTIONS_MAX_IDLE_MS_CONFIG in PulsarKafkaProducer by utilizin

[GitHub] [pulsar] sijie edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
sijie edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469563201 @jerrypeng correct. I added one more to complete the picture. ``` log.info("Pulsar AllowNull :{}", new String(

[GitHub] [pulsar] sijie commented on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
sijie commented on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469563201 @jerrypeng correct. I added one more to complete the picture. ``` log.info("Pulsar AllowNull :{}", new String(AvroSch

[GitHub] [pulsar] jerrypeng commented on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng commented on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469562848 Here is a Test project i have created: https://github.com/jerrypeng/TestAvro I verified the reason why the AvroSchema part of the shaded

[GitHub] [pulsar] jerrypeng commented on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng commented on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469560056 @sijie I think the reason why ```AvroSchema.of(User.class)``` is producing a different result is because we shade all the Avro dependencies and that

[GitHub] [pulsar] jiazhai commented on issue #3740: fix document description bugs

2019-03-04 Thread GitBox
jiazhai commented on issue #3740: fix document description bugs URL: https://github.com/apache/pulsar/pull/3740#issuecomment-469555199 run java8 tests This is an automated message from the Apache Git Service. To respond to the

[GitHub] [pulsar] jiazhai commented on issue #3677: PIP-30: interface and mutual change authentication

2019-03-04 Thread GitBox
jiazhai commented on issue #3677: PIP-30: interface and mutual change authentication URL: https://github.com/apache/pulsar/pull/3677#issuecomment-469555326 run java8 tests This is an automated message from the Apache Git Serv

[GitHub] [pulsar] sijie edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
sijie edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469554352 @jerrypeng I got a different result than yours ``` log.info("Pulsar AllowNull :{}", new String(AvroSchema.of(User.class).ge

[GitHub] [pulsar] sijie edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
sijie edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469554352 @jerrypeng I got a different result than yours ``` log.info("Pulsar AllowNull :{}", new String(AvroSchema.of(User.class).get

[GitHub] [pulsar] sijie commented on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
sijie commented on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469554352 @jerrypeng I got a different result than yours ``` Pulsar AllowNull :{"type":"record","name":"User","namespace":"example.avro","fields":[{"

[GitHub] [pulsar] srkukarni commented on issue #3717: Expose Secrets in Source/SinkContext

2019-03-04 Thread GitBox
srkukarni commented on issue #3717: Expose Secrets in Source/SinkContext URL: https://github.com/apache/pulsar/pull/3717#issuecomment-469553012 run java8 tests This is an automated message from the Apache Git Service. To respo

[GitHub] [pulsar] jerrypeng commented on issue #3751: fix function termination cleanup

2019-03-04 Thread GitBox
jerrypeng commented on issue #3751: fix function termination cleanup URL: https://github.com/apache/pulsar/pull/3751#issuecomment-469552993 rerun integration tests This is an automated message from the Apache Git Service. To r

[GitHub] [pulsar] srkukarni commented on issue #3722: Retry creation of assignment topic a few times before giving up

2019-03-04 Thread GitBox
srkukarni commented on issue #3722: Retry creation of assignment topic a few times before giving up URL: https://github.com/apache/pulsar/pull/3722#issuecomment-469553074 run java8 tests This is an automated message from the

[GitHub] [pulsar] srkukarni merged pull request #3704: On publish failures, log error and count them as sys exceptions

2019-03-04 Thread GitBox
srkukarni merged pull request #3704: On publish failures, log error and count them as sys exceptions URL: https://github.com/apache/pulsar/pull/3704 This is an automated message from the Apache Git Service. To respond to the

[pulsar] branch master updated: On publish failures, log error and count them as sys exceptions (#3704)

2019-03-04 Thread sanjeevrk
This is an automated email from the ASF dual-hosted git repository. sanjeevrk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new 07cebb1 On publish failures, log error and c

[GitHub] [pulsar] sijie commented on issue #3752: revise the schema default type not null

2019-03-04 Thread GitBox
sijie commented on issue #3752: revise the schema default type not null URL: https://github.com/apache/pulsar/pull/3752#issuecomment-469550028 @merlimat > The user defined will still be the same though, as the data serialized by it. the user defined will be still the same. th

[GitHub] [pulsar] jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469545652 @sijie That is the workflow you were worried about right? i.e. ``` schema A => generated pojo => ReflectData.parse => schema B ```

[GitHub] [pulsar] jerrypeng commented on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng commented on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469545652 @sijie That is the workflow you were worried about right? i.e. ``` schema A => generated pojo => ReflectData.parse => schema B ``` and sch

[GitHub] [pulsar] jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469545111 @sijie The User class is an Avro generated class based on the schema file. Here is the file: https://gist.github.com/jerrypeng/6bd350

[GitHub] [pulsar] merlimat commented on issue #3752: revise the schema default type not null

2019-03-04 Thread GitBox
merlimat commented on issue #3752: revise the schema default type not null URL: https://github.com/apache/pulsar/pull/3752#issuecomment-469545087 In this case the problem is that, if we change the default setting, the schema extracted from a given POJO will change. The user defined will sti

[GitHub] [pulsar] jerrypeng commented on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng commented on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469545111 @sijie The User class is an Avro generated class based on the schema file. ``` /** * Autogenerated by Avro * * DO NOT EDIT DIREC

[GitHub] [pulsar] sijie commented on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
sijie commented on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469543750 @jerrypeng I am surprised to see `{"name":"favorite_color","type":"string"}`is not a union type with `null` after using AllowNull. so my q

[GitHub] [pulsar] jerrypeng commented on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng commented on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469543016 Here is actually the line of code that actually just gets the "SCHEMA" field regardless of the java class: https://github.com/apache/avro/blob/re

[GitHub] [pulsar] sijie commented on issue #3752: revise the schema default type not null

2019-03-04 Thread GitBox
sijie commented on issue #3752: revise the schema default type not null URL: https://github.com/apache/pulsar/pull/3752#issuecomment-469542217 > Regarding the migration path, I'm not referring to migrate the schema definition, but rather to allow a one-time "incompatible" schema evolution

[GitHub] [pulsar] merlimat edited a comment on issue #3752: revise the schema default type not null

2019-03-04 Thread GitBox
merlimat edited a comment on issue #3752: revise the schema default type not null URL: https://github.com/apache/pulsar/pull/3752#issuecomment-469541275 Adding the flag and leave `allowNull=true` by default is good and easy. I think though that using `allowNull=false` would be a bet

[GitHub] [pulsar] merlimat commented on issue #3752: revise the schema default type not null

2019-03-04 Thread GitBox
merlimat commented on issue #3752: revise the schema default type not null URL: https://github.com/apache/pulsar/pull/3752#issuecomment-469541275 Adding the flag and leave `allowNull=false` by default is good and easy. I think though that using `allowNull=true` would be a better defa

[GitHub] [pulsar] jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469540392 @sijie @skyrocknroll I just experimented a little. Given the schema that @skyrocknroll posted on the pulsar chat as an example: `

[GitHub] [pulsar] jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469540392 @sijie @skyrocknroll I just experimented a little. Given the schema that @skyrocknroll posted on the pulsar chat as an example: `

[GitHub] [pulsar] jerrypeng commented on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng commented on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469540392 @sijie @skyrocknroll I just experimented a little. Given the schema that @skyrocknroll posted on the pulsar chat as an example: ``` {

[GitHub] [pulsar] skyrocknroll commented on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
skyrocknroll commented on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469539254 @merlimat Yes, that would be awesome! This is an automated message from the Apac

[GitHub] [pulsar] merlimat commented on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
merlimat commented on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469537088 @skyrocknroll I guess, for Python it might make sense to also allow to pass the JSON avro definition

[GitHub] [pulsar] merlimat commented on issue #3749: On shutdown, kill the function instance thread after interrupt attempt

2019-03-04 Thread GitBox
merlimat commented on issue #3749: On shutdown, kill the function instance thread after interrupt attempt URL: https://github.com/apache/pulsar/pull/3749#issuecomment-469536576 run java8 tests This is an automated mes

[GitHub] [pulsar] sijie commented on a change in pull request #3735: Implementing authentication for Pulsar Functions

2019-03-04 Thread GitBox
sijie commented on a change in pull request #3735: Implementing authentication for Pulsar Functions URL: https://github.com/apache/pulsar/pull/3735#discussion_r262338096 ## File path: pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/KubernetesRuntime.java

[GitHub] [pulsar] sijie commented on a change in pull request #3735: Implementing authentication for Pulsar Functions

2019-03-04 Thread GitBox
sijie commented on a change in pull request #3735: Implementing authentication for Pulsar Functions URL: https://github.com/apache/pulsar/pull/3735#discussion_r262337906 ## File path: pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/auth/NoOpFunctionAuthProvider.j

[GitHub] [pulsar] sijie commented on a change in pull request #3735: Implementing authentication for Pulsar Functions

2019-03-04 Thread GitBox
sijie commented on a change in pull request #3735: Implementing authentication for Pulsar Functions URL: https://github.com/apache/pulsar/pull/3735#discussion_r262337750 ## File path: pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/auth/FunctionAuthProvider.java

[GitHub] [pulsar] sijie commented on a change in pull request #3735: Implementing authentication for Pulsar Functions

2019-03-04 Thread GitBox
sijie commented on a change in pull request #3735: Implementing authentication for Pulsar Functions URL: https://github.com/apache/pulsar/pull/3735#discussion_r262337515 ## File path: pulsar-functions/proto/src/main/proto/Function.proto ## @@ -141,6 +141,11 @@ message Func

[GitHub] [pulsar] skyrocknroll edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
skyrocknroll edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469524620 We are using avro because avro being an standard and guaranteed to work across different part of the whole data pipeline. Any system we us

[GitHub] [pulsar] skyrocknroll commented on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
skyrocknroll commented on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469524620 We are using avro because avro being an standard and guaranteed to work across different part of the whole data pipeline. Any system we use in th

[GitHub] [pulsar] jiazhai commented on a change in pull request #3677: PIP-30: interface and mutual change authentication

2019-03-04 Thread GitBox
jiazhai commented on a change in pull request #3677: PIP-30: interface and mutual change authentication URL: https://github.com/apache/pulsar/pull/3677#discussion_r262337142 ## File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java ## @@ -4

[GitHub] [pulsar] sijie commented on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
sijie commented on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469522219 @jerrypeng > Using the reflect data parser in avro will produce the same schema. Glad we are on the same page now. > I agree we sho

[GitHub] [pulsar] sijie edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
sijie edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469522219 @jerrypeng > Using the reflect data parser in avro will produce the same schema. Glad that we are on the same page now. > I

[GitHub] [pulsar] sijie commented on issue #3752: revise the schema default type not null

2019-03-04 Thread GitBox
sijie commented on issue #3752: revise the schema default type not null URL: https://github.com/apache/pulsar/pull/3752#issuecomment-469521239 @congbobo184 @jerrypeng @merlimat : I don't think we should attempt to involve any data migration between schemas, that will be a nightmare.

[GitHub] [pulsar] jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469518971 @sijie actually this does work: schema A => generated pojo => ReflectData.parse => schema B Schema A == . Schema B Using t

[GitHub] [pulsar] jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469518971 @sijie actually this does work: schema A => generated pojo => ReflectData.parse => schema B Schema A == . Schema B Using t

[GitHub] [pulsar] jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469518971 @sijie actually this does work: schema A => generated pojo => ReflectData.parse => schema B Schema A == . Schema B Using t

[GitHub] [pulsar] jerrypeng commented on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng commented on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469518971 @sijie actually this does work: schema A => generated pojo => ReflectData.parse => schema B Schema A == . Schema B

[GitHub] [pulsar] sijie edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
sijie edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469518573 @jerrypeng > And from that generated class there is no way for reflect data avro library to generate the same schema unless we do the tric

[GitHub] [pulsar] sijie commented on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
sijie commented on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469518573 @jerrypeng > And from that generated class there is no way for reflect data avro library to generate the same schema unless we do the trick @merl

[GitHub] [pulsar] jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469514075 @sijie that not going to work. If you take look at the java code generated by avro tools with a given schema file its going to be not what yo

[GitHub] [pulsar] jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469514075 @sijie that not going to work. If you take look at the java code generated by avro tools with a given schema file its going to be not what yo

[GitHub] [pulsar] merlimat commented on issue #3752: revise the schema default type not null

2019-03-04 Thread GitBox
merlimat commented on issue #3752: revise the schema default type not null URL: https://github.com/apache/pulsar/pull/3752#issuecomment-469517939 Since the Python schema generation is following the current behavior of the Java client. We also need to change the defaults there. Again, with m

[GitHub] [pulsar] jerrypeng commented on issue #3751: fix function termination cleanup

2019-03-04 Thread GitBox
jerrypeng commented on issue #3751: fix function termination cleanup URL: https://github.com/apache/pulsar/pull/3751#issuecomment-469516026 rerun java8 tests rerun integration tests This is an automated message from the Apa

[GitHub] [pulsar] jerrypeng edited a comment on issue #3752: revise the schema default type not null

2019-03-04 Thread GitBox
jerrypeng edited a comment on issue #3752: revise the schema default type not null URL: https://github.com/apache/pulsar/pull/3752#issuecomment-469514362 @congbobo184 while I am ok if we decide to remove allow null from set by default, my only concern is this ix NOT a backwards compatible

[GitHub] [pulsar] jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469514075 @sijie that not going to work. If you take look at the java code generated by avro tools with a given schema file its going to be not what yo

[GitHub] [pulsar] jerrypeng edited a comment on issue #3752: revise the schema default type not null

2019-03-04 Thread GitBox
jerrypeng edited a comment on issue #3752: revise the schema default type not null URL: https://github.com/apache/pulsar/pull/3752#issuecomment-469514362 @congbobo184 while I am ok if we decide to remove allow null from set by default, but this is NOT a backwards compatible change. I thin

[GitHub] [pulsar] jerrypeng commented on issue #3752: revise the schema default type not null

2019-03-04 Thread GitBox
jerrypeng commented on issue #3752: revise the schema default type not null URL: https://github.com/apache/pulsar/pull/3752#issuecomment-469514362 @congbobo184 while I am ok if we decide to remove allow null from set by default, but this is NOT a backwards compatible change.

[GitHub] [pulsar] jerrypeng commented on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng commented on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469514075 @sijie that not going to work. If you take look at the java code generated by avro tools with a given schema file its going to be not what you expec

[GitHub] [pulsar] congbobo184 removed a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
congbobo184 removed a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469513849 #3752 This is an automated message from the Apache Git Service. To respon

[GitHub] [pulsar] congbobo184 commented on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
congbobo184 commented on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469513849 #3752 This is an automated message from the Apache Git Service. To respond to the

[GitHub] [pulsar] congbobo184 opened a new pull request #3752: revise the schema default type not null

2019-03-04 Thread GitBox
congbobo184 opened a new pull request #3752: revise the schema default type not null URL: https://github.com/apache/pulsar/pull/3752 Motivation make define define schema type unable null#3741 Modifications make the client define schema type unable null Verifying this cha

[GitHub] [pulsar] sijie commented on issue #3749: On shutdown, kill the function instance thread after interrupt attempt

2019-03-04 Thread GitBox
sijie commented on issue #3749: On shutdown, kill the function instance thread after interrupt attempt URL: https://github.com/apache/pulsar/pull/3749#issuecomment-469512606 run java8 tests This is an automated message from t

[GitHub] [pulsar] sijie commented on issue #3737: [python client] set protobuf version to be larger than 3.6.0

2019-03-04 Thread GitBox
sijie commented on issue #3737: [python client] set protobuf version to be larger than 3.6.0 URL: https://github.com/apache/pulsar/pull/3737#issuecomment-46951 run java8 tests This is an automated message from the Apache

[GitHub] [pulsar] sijie edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
sijie edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469506795 > Given a POJO generated by Avro, there is no way to determine whether this POJO was generated with a schema that allowed or not for null types.

[GitHub] [pulsar] sijie edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
sijie edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469506795 > Given a POJO generated by Avro, there is no way to determine whether this POJO was generated with a schema that allowed or not for null types.

[GitHub] [pulsar] sijie edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
sijie edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469506795 > Given a POJO generated by Avro, there is no way to determine whether this POJO was generated with a schema that allowed or not for null types.

[GitHub] [pulsar] sijie commented on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
sijie commented on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469506795 > Given a POJO generated by Avro, there is no way to determine whether this POJO was generated with a schema that allowed or not for null types. > but

[GitHub] [pulsar] srkukarni commented on issue #3717: Expose Secrets in Source/SinkContext

2019-03-04 Thread GitBox
srkukarni commented on issue #3717: Expose Secrets in Source/SinkContext URL: https://github.com/apache/pulsar/pull/3717#issuecomment-469503655 run java8 tests This is an automated message from the Apache Git Service. To respo

[GitHub] [pulsar] murong00 commented on issue #3745: [Issue #3726] Modify the usage of Connector CLI for better guidance

2019-03-04 Thread GitBox
murong00 commented on issue #3745: [Issue #3726] Modify the usage of Connector CLI for better guidance URL: https://github.com/apache/pulsar/pull/3745#issuecomment-469501212 @sijie Not found in version 2.3.0, just copy from the master and modify original_id, PTAL!

[GitHub] [pulsar] srkukarni commented on issue #3722: Retry creation of assignment topic a few times before giving up

2019-03-04 Thread GitBox
srkukarni commented on issue #3722: Retry creation of assignment topic a few times before giving up URL: https://github.com/apache/pulsar/pull/3722#issuecomment-469497850 run java8 tests This is an automated message from the

[GitHub] [pulsar] cckellogg commented on issue #3693: Fix topic name logic for partitioned topics

2019-03-04 Thread GitBox
cckellogg commented on issue #3693: Fix topic name logic for partitioned topics URL: https://github.com/apache/pulsar/pull/3693#issuecomment-469497781 rerun java8 tests This is an automated message from the Apache Git Service.

[GitHub] [pulsar] jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469488859 > The point here for a given POJO, Pulsar should maintain a basic contract: the schema generated by > Pulsar AVRO should be same/compatible wi

[GitHub] [pulsar] jerrypeng opened a new pull request #3751: fix function termination cleanup

2019-03-04 Thread GitBox
jerrypeng opened a new pull request #3751: fix function termination cleanup URL: https://github.com/apache/pulsar/pull/3751 ### Motivation Currently the termination routine runs once for every function instance scheduled on a particular worker. If there are multiple instances of a

[GitHub] [pulsar] jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469488859 > The point here for a given POJO, Pulsar should maintain a basic contract: the schema generated by > Pulsar AVRO should be same/compatible wi

[GitHub] [pulsar] jerrypeng commented on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng commented on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469488859 > The point here for a given POJO, Pulsar should maintain a basic contract: the schema generated by > Pulsar AVRO should be same/compatible with

[GitHub] [pulsar] jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
jerrypeng edited a comment on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469488859 > The point here for a given POJO, Pulsar should maintain a basic contract: the schema generated by > Pulsar AVRO should be same/compatible wi

[GitHub] [pulsar] merlimat commented on a change in pull request #3732: [pulsar-broker] Optimize message replay for large backlog consumer

2019-03-04 Thread GitBox
merlimat commented on a change in pull request #3732: [pulsar-broker] Optimize message replay for large backlog consumer URL: https://github.com/apache/pulsar/pull/3732#discussion_r262306086 ## File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/Persi

[GitHub] [pulsar] rdhabalia commented on issue #3732: [pulsar-broker] Optimize message replay for large backlog consumer

2019-03-04 Thread GitBox
rdhabalia commented on issue #3732: [pulsar-broker] Optimize message replay for large backlog consumer URL: https://github.com/apache/pulsar/pull/3732#issuecomment-469486832 rerun java8 tests This is an automated message from

[GitHub] [pulsar] merlimat commented on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
merlimat commented on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469486525 Though, I too think we should remove the AllowNull from the generator, but that's separate from the point that that alone won't guarantee we can gener

[GitHub] [pulsar] merlimat commented on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
merlimat commented on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469484274 > that's not the only case. People can use ReflectData to generate a schema and write events in a different system (let's call it source). when people

[GitHub] [pulsar] sijie commented on issue #3741: POJO AvroSchema always allowNull

2019-03-04 Thread GitBox
sijie commented on issue #3741: POJO AvroSchema always allowNull URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469483661 > I don't understand why you say we are not respecting the rules of AVRO > The only flag that could be settable is the allow null flag that gets set i

[GitHub] [pulsar] srkukarni commented on issue #3717: Expose Secrets in Source/SinkContext

2019-03-04 Thread GitBox
srkukarni commented on issue #3717: Expose Secrets in Source/SinkContext URL: https://github.com/apache/pulsar/pull/3717#issuecomment-469483460 run java8 tests This is an automated message from the Apache Git Service. To respo

[GitHub] [pulsar] srkukarni commented on issue #3704: On publish failures, log error and count them as sys exceptions

2019-03-04 Thread GitBox
srkukarni commented on issue #3704: On publish failures, log error and count them as sys exceptions URL: https://github.com/apache/pulsar/pull/3704#issuecomment-469483377 run java8 tests This is an automated message from the

[GitHub] [pulsar] jerrypeng commented on issue #3735: Implementing authentication for Pulsar Functions

2019-03-04 Thread GitBox
jerrypeng commented on issue #3735: Implementing authentication for Pulsar Functions URL: https://github.com/apache/pulsar/pull/3735#issuecomment-469481790 rerun java8 tests rerun cpp tests This is an automated message fro

[GitHub] [pulsar] merlimat opened a new pull request #3750: Negative acks in C++ client

2019-03-04 Thread GitBox
merlimat opened a new pull request #3750: Negative acks in C++ client URL: https://github.com/apache/pulsar/pull/3750 ### Motivation Similar to #3703, adding support for negative Acks in C++ client (and C wrapper) This

[GitHub] [pulsar] rdhabalia commented on issue #3749: On shutdown, kill the function instance thread after interrupt attempt

2019-03-04 Thread GitBox
rdhabalia commented on issue #3749: On shutdown, kill the function instance thread after interrupt attempt URL: https://github.com/apache/pulsar/pull/3749#issuecomment-469477124 rerun java8 tests This is an automated message

[pulsar] branch master updated: Adjust the serving threads to have a minimum of threads (#3698)

2019-03-04 Thread sanjeevrk
This is an automated email from the ASF dual-hosted git repository. sanjeevrk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new 9dc3df6 Adjust the serving threads to have a

  1   2   3   >