Re: [ANNOUNCE] New Committer: Asaf Mesika
Congratulations! Thanks, Haiting On Wed, Feb 21, 2024 at 9:35 PM Yubiao Feng wrote: > > Congratulations! > > Thanks > Yubiao Feng > > On Wed, Feb 21, 2024 at 12:50 AM Lari Hotari wrote: > > > The Apache Pulsar Project Management Committee (PMC) has invited > > Asaf Mesika https://github.com/asafm to become a committer and we > > are pleased to announce that he has accepted. > > > > Welcome and Congratulations, Asaf Mesika! > > > > Please join us in congratulating and welcoming Asaf onboard! > > > > Best Regards, > > > > Lari Hotari > > on behalf of the Pulsar PMC > >
[DISCUSS] Apache Pulsar 3.0.3 Release
Dear the Apache Pulsar Dev community, I would like to propose releasing Apache Pulsar 3.0.3. It's over two months since the release of 3.0.2, and there are 95 new commits in branch-3.0: https://github.com/apache/pulsar/compare/v3.0.2...branch-3.0 Please let me know if you have any essential fixes to include in Apache Pulsar 3.0.3. Regards, Heesung
Re: [DISCUSS] Migrate CLI parser from jcommander to picocli
+1 Please take into account Pulsar Shell extensions Enrico Il Mer 21 Feb 2024, 11:21 Girish Sharma ha scritto: > +1 (non-binding) > It has been a pain trying to figure out what's the exact sub-param name > given that some of them are 20+ characters long. > > > On Wed, Feb 21, 2024 at 3:47 PM Julien Jakubowski > wrote: > > > Thanks for this proposal! That should significantly improve the user > > experience with the CLI. > > > > > > > Le 21 févr. 2024 à 06:08, Zixuan Liu a écrit : > > > > > > Hi Pulsar Community, > > > > > > The pulsar CLI(pulsar-admin, pulsar-client, pulsar-shell, and so on) > uses > > > the jcommander [1] as CLI parser, which is an awesome project, but the > > > maintainer is not active and cannot keep up with modern CLI > > > features(auto-completion, sub-command, native-images, suggest commands, > > and > > > so on). Then I found the picocli [2] project to meet these needs, which > > is > > > active and powerful. For comparison, please see [3]. > > > > > > Error prompt: > > > ``` > > > bin/pulsar-admin clusters update cluster-a -b > > > > > > # jcommander > > > Need to provide just 1 parameter > > > > > > # picocli > > > Unknown option: '-b' > > > ``` > > > > > > Suggest commands: > > > ``` > > > bin/pulsar-admin cluste > > > > > > # jcommander > > > Expected a command, got cluste > > > > > > # picocli > > > Unmatched argument at index 0: 'cluste' > > > Did you mean: pulsar-admin clusters? > > > ``` > > > > > > What do you think about migrating CLI parse from jcommander to picocli? > > > > > > Thanks, > > > Zixuan > > > > > > [1] - https://github.com/cbeust/jcommander > > > [2] - https://picocli.info/ > > > [3] - https://github.com/remkop/picocli/wiki/picocli-vs-JCommander > > > > > -- > Girish Sharma >
Re: [DISCUSSION] Proposal to Replace Pulsar's Homegrown Configuration Framework with Gestalt Config
Good idea. It would be nice to just have the config library to read the environment variables directly, instead of having to write them down in the file. -- Matteo Merli On Wed, Feb 21, 2024 at 1:28 AM Lari Hotari wrote: > Hello everyone, > > I would like to bring up an issue with Pulsar's containers, specifically > regarding the method of overriding configurations. For instance, the > Apache Pulsar Helm chart employs "bin/apply-config-from-env.py > conf/broker.conf" and "bin/gen-yml-from-env.py > conf/functions_worker.yml" [1] to apply configurations passed in the > environment to the configuration files in the container's root file > system. > This approach fails when the container's root file system is read-only due > to > strict security policies (`readOnlyRootFilesystem` in > `securityContext`). This issue has been reported as #22088 [2]. > > A temporary fix could involve using a temporary file to modify the > configuration file when the filesystem is read-only. However, the Python > script solution is not ideal, and we should consider eliminating it. In > the long term, it would also be beneficial to remove the need for a > shell script to start Pulsar, but that's a separate issue. > > For configuration handling, we need a solution that can apply overrides > in memory, eliminating the need to modify on-disk files. Modern > configuration frameworks can do this out-of-the-box. Currently, Pulsar > uses a homegrown configuration framework. Instead of extending this > framework, I propose we discuss replacing it with the Gestalt Config > library [3]. This library, licensed under Apache-2.0, is a mature, > well-established solution for configuration handling. > > Switching to Gestalt Config would allow us to move towards a more > structured and modular configuration in Pulsar. Our current > configuration is not modular, as it relies on a "god object" for > configuration, which collects all possible configuration options. > Gestalt Config offers modular usage patterns similar to those of > Spring Boot's external configuration [4] and the MicroProfile Config [5] > in Quarkus. However, Gestalt Config does not pull in other dependencies, > giving it an advantage over Spring Boot and Quarkus configuration > solutions. > Other libraries in this category include the Typesafe config library [6] > from Lightbend with HOCON [7], commonly used in Scala and Akka-based > applications. > > Gestalt Config supports many configuration file formats, including flat > properties files, yaml, json, toml, and even hocon. It also offers > security features for reading secrets directly from Vault, AWS Secrets > Manager, and GCP Secret Manager, without the need to use the file system > or environment variables to inject secrets into the application > configuration. This could significantly improve Pulsar's security > posture. > > Pulsar's current "homegrown configuration framework" is quite simple, > implemented in a few classes with the main logic in > PulsarConfigurationLoader [8] and FieldParser [9] classes, called from > the PulsarBrokerStarter class [10]. > > The main question is: should we continue extending Pulsar's homegrown > configuration framework, or should we consider adopting a library like > Gestalt Config for future configuration use case improvements for > modularity, structured configuration, and security? > > Best regards, > > Lari > > References: > 1 - > https://github.com/apache/pulsar-helm-chart/blob/29ea17b3fceef65160620b9018d0dd0449a168c5/charts/pulsar/templates/broker-statefulset.yaml#L210-L221 > 2 - https://github.com/apache/pulsar/issues/22088 > 3 - https://github.com/gestalt-config/gestalt > 4 - > https://docs.spring.io/spring-boot/docs/current/reference/html/features.html#features.external-config > 5 - https://microprofile.io/specifications/microprofile-config/ > 6 - https://github.com/lightbend/config > 7 - https://github.com/lightbend/config/blob/main/HOCON.md > 8 - > https://github.com/apache/pulsar/blob/master/pulsar-broker-common/src/main/java/org/apache/pulsar/common/configuration/PulsarConfigurationLoader.java > 9 - > https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/java/org/apache/pulsar/common/util/FieldParser.java > 10 - > https://github.com/apache/pulsar/blob/db79096baaa3d7118aa026978a615ddc576f9183/pulsar-broker/src/main/java/org/apache/pulsar/PulsarBrokerStarter.java#L69-L76 >
Re: [ANNOUNCE] New Committer: Asaf Mesika
Congratulations! Thanks Yubiao Feng On Wed, Feb 21, 2024 at 12:50 AM Lari Hotari wrote: > The Apache Pulsar Project Management Committee (PMC) has invited > Asaf Mesika https://github.com/asafm to become a committer and we > are pleased to announce that he has accepted. > > Welcome and Congratulations, Asaf Mesika! > > Please join us in congratulating and welcoming Asaf onboard! > > Best Regards, > > Lari Hotari > on behalf of the Pulsar PMC >
Re: [ANNOUNCE] New Committer: Asaf Mesika
Congrats! Asaf Yike Xiao > On Feb 21, 2024, at 18:55, houxiaoyu wrote: > > Congratulations! > > Best, > houxiaoyu > > Max Xu 于2024年2月21日周三 17:32写道： >> >> Congratulations, Asaf! Well deserved. >> >> >> Best, >> Max Xu >> >> >>> On Wed, Feb 21, 2024 at 12:51 AM Lari Hotari wrote: >>> >>> The Apache Pulsar Project Management Committee (PMC) has invited >>> Asaf Mesika https://github.com/asafm to become a committer and we >>> are pleased to announce that he has accepted. >>> >>> Welcome and Congratulations, Asaf Mesika! >>> >>> Please join us in congratulating and welcoming Asaf onboard! >>> >>> Best Regards, >>> >>> Lari Hotari >>> on behalf of the Pulsar PMC >>>
Re: [ANNOUNCE] New Committer: Asaf Mesika
Congratulations! Best, houxiaoyu Max Xu 于2024年2月21日周三 17:32写道： > > Congratulations, Asaf! Well deserved. > > > Best, > Max Xu > > > On Wed, Feb 21, 2024 at 12:51 AM Lari Hotari wrote: > > > The Apache Pulsar Project Management Committee (PMC) has invited > > Asaf Mesika https://github.com/asafm to become a committer and we > > are pleased to announce that he has accepted. > > > > Welcome and Congratulations, Asaf Mesika! > > > > Please join us in congratulating and welcoming Asaf onboard! > > > > Best Regards, > > > > Lari Hotari > > on behalf of the Pulsar PMC > >
Re: [DISCUSS] Migrate CLI parser from jcommander to picocli
+1 (non-binding) It has been a pain trying to figure out what's the exact sub-param name given that some of them are 20+ characters long. On Wed, Feb 21, 2024 at 3:47 PM Julien Jakubowski wrote: > Thanks for this proposal! That should significantly improve the user > experience with the CLI. > > > > Le 21 févr. 2024 à 06:08, Zixuan Liu a écrit : > > > > Hi Pulsar Community, > > > > The pulsar CLI(pulsar-admin, pulsar-client, pulsar-shell, and so on) uses > > the jcommander [1] as CLI parser, which is an awesome project, but the > > maintainer is not active and cannot keep up with modern CLI > > features(auto-completion, sub-command, native-images, suggest commands, > and > > so on). Then I found the picocli [2] project to meet these needs, which > is > > active and powerful. For comparison, please see [3]. > > > > Error prompt: > > ``` > > bin/pulsar-admin clusters update cluster-a -b > > > > # jcommander > > Need to provide just 1 parameter > > > > # picocli > > Unknown option: '-b' > > ``` > > > > Suggest commands: > > ``` > > bin/pulsar-admin cluste > > > > # jcommander > > Expected a command, got cluste > > > > # picocli > > Unmatched argument at index 0: 'cluste' > > Did you mean: pulsar-admin clusters? > > ``` > > > > What do you think about migrating CLI parse from jcommander to picocli? > > > > Thanks, > > Zixuan > > > > [1] - https://github.com/cbeust/jcommander > > [2] - https://picocli.info/ > > [3] - https://github.com/remkop/picocli/wiki/picocli-vs-JCommander > -- Girish Sharma
Re: [DISCUSS] Migrate CLI parser from jcommander to picocli
Thanks for this proposal! That should significantly improve the user experience with the CLI. > Le 21 févr. 2024 à 06:08, Zixuan Liu a écrit : > > Hi Pulsar Community, > > The pulsar CLI(pulsar-admin, pulsar-client, pulsar-shell, and so on) uses > the jcommander [1] as CLI parser, which is an awesome project, but the > maintainer is not active and cannot keep up with modern CLI > features(auto-completion, sub-command, native-images, suggest commands, and > so on). Then I found the picocli [2] project to meet these needs, which is > active and powerful. For comparison, please see [3]. > > Error prompt: > ``` > bin/pulsar-admin clusters update cluster-a -b > > # jcommander > Need to provide just 1 parameter > > # picocli > Unknown option: '-b' > ``` > > Suggest commands: > ``` > bin/pulsar-admin cluste > > # jcommander > Expected a command, got cluste > > # picocli > Unmatched argument at index 0: 'cluste' > Did you mean: pulsar-admin clusters? > ``` > > What do you think about migrating CLI parse from jcommander to picocli? > > Thanks, > Zixuan > > [1] - https://github.com/cbeust/jcommander > [2] - https://picocli.info/ > [3] - https://github.com/remkop/picocli/wiki/picocli-vs-JCommander
Re: [DISCUSSION] Proposal to Replace Pulsar's Homegrown Configuration Framework with Gestalt Config
Lari, Il Mer 21 Feb 2024, 10:27 Lari Hotari ha scritto: > Hello everyone, > > I would like to bring up an issue with Pulsar's containers, specifically > regarding the method of overriding configurations. For instance, the > Apache Pulsar Helm chart employs "bin/apply-config-from-env.py > conf/broker.conf" and "bin/gen-yml-from-env.py > conf/functions_worker.yml" [1] to apply configurations passed in the > environment to the configuration files in the container's root file > system. > This approach fails when the container's root file system is read-only due > to > strict security policies (`readOnlyRootFilesystem` in > `securityContext`). This issue has been reported as #22088 [2]. > > A temporary fix could involve using a temporary file to modify the > configuration file when the filesystem is read-only. However, the Python > script solution is not ideal, and we should consider eliminating it. In > the long term, it would also be beneficial to remove the need for a > shell script to start Pulsar, but that's a separate issue. > > For configuration handling, we need a solution that can apply overrides > in memory, eliminating the need to modify on-disk files. Modern > configuration frameworks can do this out-of-the-box. Currently, Pulsar > uses a homegrown configuration framework. Instead of extending this > framework, I propose we discuss replacing it with the Gestalt Config > library [3]. This library, licensed under Apache-2.0, is a mature, > well-established solution for configuration handling. > > Switching to Gestalt Config would allow us to move towards a more > structured and modular configuration in Pulsar. Our current > configuration is not modular, as it relies on a "god object" for > configuration, which collects all possible configuration options. > Gestalt Config offers modular usage patterns similar to those of > Spring Boot's external configuration [4] and the MicroProfile Config [5] > in Quarkus. However, Gestalt Config does not pull in other dependencies, > giving it an advantage over Spring Boot and Quarkus configuration > solutions. > Other libraries in this category include the Typesafe config library [6] > from Lightbend with HOCON [7], commonly used in Scala and Akka-based > applications. > > Gestalt Config supports many configuration file formats, including flat > properties files, yaml, json, toml, and even hocon. It also offers > security features for reading secrets directly from Vault, AWS Secrets > Manager, and GCP Secret Manager, without the need to use the file system > or environment variables to inject secrets into the application > configuration. This could significantly improve Pulsar's security > posture. > > Pulsar's current "homegrown configuration framework" is quite simple, > implemented in a few classes with the main logic in > PulsarConfigurationLoader [8] and FieldParser [9] classes, called from > the PulsarBrokerStarter class [10]. > > The main question is: should we continue extending Pulsar's homegrown > configuration framework, or should we consider adopting a library like > Gestalt Config for future configuration use case improvements for > modularity, structured configuration, and security? > I like this proposal. One thing we have to take into account is that many third party modules, like ProtocolHandlers, Proxy Extensions and anything that is pluggable is using the configuration classes. We have to find a way to guarantee backward compatibility Enrico > Best regards, > > Lari > > References: > 1 - > https://github.com/apache/pulsar-helm-chart/blob/29ea17b3fceef65160620b9018d0dd0449a168c5/charts/pulsar/templates/broker-statefulset.yaml#L210-L221 > 2 - https://github.com/apache/pulsar/issues/22088 > 3 - https://github.com/gestalt-config/gestalt > 4 - > https://docs.spring.io/spring-boot/docs/current/reference/html/features.html#features.external-config > 5 - https://microprofile.io/specifications/microprofile-config/ > 6 - https://github.com/lightbend/config > 7 - https://github.com/lightbend/config/blob/main/HOCON.md > 8 - > https://github.com/apache/pulsar/blob/master/pulsar-broker-common/src/main/java/org/apache/pulsar/common/configuration/PulsarConfigurationLoader.java > 9 - > https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/java/org/apache/pulsar/common/util/FieldParser.java > 10 - > https://github.com/apache/pulsar/blob/db79096baaa3d7118aa026978a615ddc576f9183/pulsar-broker/src/main/java/org/apache/pulsar/PulsarBrokerStarter.java#L69-L76 >
Re: [ANNOUNCE] New Committer: Asaf Mesika
Congratulations, Asaf! Well deserved. Best, Max Xu On Wed, Feb 21, 2024 at 12:51 AM Lari Hotari wrote: > The Apache Pulsar Project Management Committee (PMC) has invited > Asaf Mesika https://github.com/asafm to become a committer and we > are pleased to announce that he has accepted. > > Welcome and Congratulations, Asaf Mesika! > > Please join us in congratulating and welcoming Asaf onboard! > > Best Regards, > > Lari Hotari > on behalf of the Pulsar PMC >
[DISCUSSION] Proposal to Replace Pulsar's Homegrown Configuration Framework with Gestalt Config
Hello everyone, I would like to bring up an issue with Pulsar's containers, specifically regarding the method of overriding configurations. For instance, the Apache Pulsar Helm chart employs "bin/apply-config-from-env.py conf/broker.conf" and "bin/gen-yml-from-env.py conf/functions_worker.yml" [1] to apply configurations passed in the environment to the configuration files in the container's root file system. This approach fails when the container's root file system is read-only due to strict security policies (`readOnlyRootFilesystem` in `securityContext`). This issue has been reported as #22088 [2]. A temporary fix could involve using a temporary file to modify the configuration file when the filesystem is read-only. However, the Python script solution is not ideal, and we should consider eliminating it. In the long term, it would also be beneficial to remove the need for a shell script to start Pulsar, but that's a separate issue. For configuration handling, we need a solution that can apply overrides in memory, eliminating the need to modify on-disk files. Modern configuration frameworks can do this out-of-the-box. Currently, Pulsar uses a homegrown configuration framework. Instead of extending this framework, I propose we discuss replacing it with the Gestalt Config library [3]. This library, licensed under Apache-2.0, is a mature, well-established solution for configuration handling. Switching to Gestalt Config would allow us to move towards a more structured and modular configuration in Pulsar. Our current configuration is not modular, as it relies on a "god object" for configuration, which collects all possible configuration options. Gestalt Config offers modular usage patterns similar to those of Spring Boot's external configuration [4] and the MicroProfile Config [5] in Quarkus. However, Gestalt Config does not pull in other dependencies, giving it an advantage over Spring Boot and Quarkus configuration solutions. Other libraries in this category include the Typesafe config library [6] from Lightbend with HOCON [7], commonly used in Scala and Akka-based applications. Gestalt Config supports many configuration file formats, including flat properties files, yaml, json, toml, and even hocon. It also offers security features for reading secrets directly from Vault, AWS Secrets Manager, and GCP Secret Manager, without the need to use the file system or environment variables to inject secrets into the application configuration. This could significantly improve Pulsar's security posture. Pulsar's current "homegrown configuration framework" is quite simple, implemented in a few classes with the main logic in PulsarConfigurationLoader [8] and FieldParser [9] classes, called from the PulsarBrokerStarter class [10]. The main question is: should we continue extending Pulsar's homegrown configuration framework, or should we consider adopting a library like Gestalt Config for future configuration use case improvements for modularity, structured configuration, and security? Best regards, Lari References: 1 - https://github.com/apache/pulsar-helm-chart/blob/29ea17b3fceef65160620b9018d0dd0449a168c5/charts/pulsar/templates/broker-statefulset.yaml#L210-L221 2 - https://github.com/apache/pulsar/issues/22088 3 - https://github.com/gestalt-config/gestalt 4 - https://docs.spring.io/spring-boot/docs/current/reference/html/features.html#features.external-config 5 - https://microprofile.io/specifications/microprofile-config/ 6 - https://github.com/lightbend/config 7 - https://github.com/lightbend/config/blob/main/HOCON.md 8 - https://github.com/apache/pulsar/blob/master/pulsar-broker-common/src/main/java/org/apache/pulsar/common/configuration/PulsarConfigurationLoader.java 9 - https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/java/org/apache/pulsar/common/util/FieldParser.java 10 - https://github.com/apache/pulsar/blob/db79096baaa3d7118aa026978a615ddc576f9183/pulsar-broker/src/main/java/org/apache/pulsar/PulsarBrokerStarter.java#L69-L76
