[DISCUSS] PIP-275: Introduce numWorkerThreadsForPersistentTopic to deprecate numWorkerThreadsForNonPersistentTopic in configuration

2023-06-05 Thread houxiaoyu
Hi Pulsar Community,

I am writing to start the discussion on PIP-275: Introduce
numWorkerThreadsForPersistentTopic to deprecate
numWorkerThreadsForNonPersistentTopic in configuration

PR with PIP contents: https://github.com/apache/pulsar/pull/20507

# Motivation

Introduce `numWorkerThreadsForPersistentTopic` to deprecate
`numWorkerThreadsForNonPersistentTopic`.

The `numWorkerThreadsForNonPersistentTopic` is used to specify for
PersistentTopic, not NonPersistentTopic. So I propose change the config
item from `numWorkerThreadsForNonPersistentTopic` to
`numWorkerThreadsForPersistentTopic`:

Thanks,
Xiaoyu Hou


Re: [DISCUSS] PIP-273: Enable hostname verification by default

2023-06-05 Thread Michael Marshall
> Should these changes be released in a minor version?

I think it is reasonable to make this change in 3.1.0. The change will
be covered in the release notes.

> For the Golang client,  I think we should introduce a
> `NewDefaultConfig` method to create the config instead of directly new
> config.

Makes sense, thanks.

- Michael

On Sat, Jun 3, 2023 at 9:37 AM Zixuan Liu  wrote:
>
> +1
>
> Default to enable the TLS hostname verification is important anywhere,
> which protected our application.
>
> Should these changes be released in a minor version?
>
> For the Golang client,  I think we should introduce a
> `NewDefaultConfig` method to create the config instead of directly new
> config.
>
> Thanks,
> Zixuan
>
> Michael Marshall  于2023年6月3日周六 05:39写道:
> >
> > I put together PRs for the python and the C++ clients:
> >
> > https://github.com/apache/pulsar-client-python/pull/128
> > https://github.com/apache/pulsar-client-cpp/pull/278
> >
> > I am not sure the right way to change the default for the go client
> > because go uses the zero value for structs, and we have the variable
> > named "TLSValidateHostname". Would it perhaps make sense to ignore
> > that variable and instead use "TLSHostnameVerificationDisabled"? Then,
> > false would be the secure default. However, it introduces a double
> > negative, which might make it harder for users to understand.
> >
> > I plan to start the vote next week.
> >
> > Thanks,
> > Michael
> >
> >
> > On Wed, May 31, 2023 at 11:08 PM Michael Marshall  
> > wrote:
> > >
> > > Hi Pulsar Community,
> > >
> > > I am writing to start the discussion on PIP 273 to enable hostname
> > > verification by default.
> > >
> > > PR with PIP contents: https://github.com/apache/pulsar/pull/20453
> > >
> > > I copy the content below (except for the associated svg of the pulsar
> > > network diagram).
> > >
> > > I look forward to your feedback.
> > >
> > > Thanks,
> > > Michael
> > >
> > > 
> > >
> > > # PIP 273: Enable hostname verification by default
> > >
> > > # Background knowledge
> > >
> > > When using TLS to secure a network connection from client to server,
> > > hostname verification is the step where the client verifies that the
> > > server's certificate contains the expected Subject Alternative Name or
> > > the Common Name. This step is essential for preventing man in the
> > > middle attacks where a malicious server presents a certificate that is
> > > cryptographically valid but was intended for a different hostname than
> > > the one the client is trying to connect to.
> > >
> > > [RFC 2818](https://datatracker.ietf.org/doc/html/rfc2818#section-3.1)
> > > provides additional details on the hostname verification process as a
> > > part of the TLS handshake and why a client must verify the identity of
> > > the server.
> > >
> > > It is very helpful to understand the network topology of a Pulsar
> > > Cluster when looking to enable hostname verification. Here is a
> > > diagram I created. It is possibly incomplete:
> > >
> > > (see PIP PR to see diagram)
> > >
> > > # Motivation
> > >
> > > The primary motivation is to improve the security of Pulsar by making
> > > it secure by default for hostname verification.
> > >
> > > Pulsar currently disables hostname verification by default, which led
> > > to some security issues in the past. For example,
> > > [CVE-2022-33682](https://github.com/apache/pulsar/wiki/CVE-2022-33682)
> > > happened because there was no way to enable hostname verification
> > > within the broker or the proxy. If hostname verification had been
> > > enabled by default, the lack of configurability would not have been a
> > > security concern but a usability concern.
> > >
> > > # Goals
> > >
> > > ## In Scope
> > >
> > > * Enable hostname verification by default for all clients in the
> > > Pulsar ecosystem.
> > > * Make it possible to configure hostname verification per
> > > geo-replicated cluster.
> > > * Consolidate the name of the hostname verification configuration
> > > option across all server components.
> > >
> > > ## Out of Scope
> > >
> > > * This PIP does not seek to enable TLS by default, which is relevant
> > > because hostname verification only takes affect when TLS is enabled.
> > > * Client configuration names will not be renamed because that would
> > > introduce binary incompatibility.
> > > * Tiered storage clients
> > > * ETCd client configuration -- it enables hostname verification by
> > > default already
> > > * Zookeeper client configuration -- it enables hostname verification
> > > by default already
> > > * Function worker's gRPC client to function instances -- TLS is not
> > > configurable for this connection
> > > * Function state storage to bookkeeper -- TLS is not configurable for
> > > this connection
> > > * Pulsar SQL worker's hostname verification configuration name, which
> > > is `pulsar.tls-hostname-verification-enable`
> > >
> > > # High Level Design
> > >
> > > * Enable hostname verification by default 

Re: [VOTE] PIP-272

2023-06-05 Thread Rui Fu
+1

Best,

Rui Fu
On Jun 6, 2023 at 09:50 +0800, Pengcheng Jiang 
, wrote:
> Hello, community:
>
> This thread is to start a vote for PIP-272: Add stateStorageConfig to
> WorkerConfig.
>
> Discussion thread:
> https://lists.apache.org/thread/pwfv7nj64frfnbw7jfydzx8my15b3lj6
> PR: https://github.com/apache/pulsar/pull/20455
>
> Sincerely
> Pengcheng Jiang


[VOTE] PIP-272

2023-06-05 Thread Pengcheng Jiang
Hello, community:

This thread is to start a vote for PIP-272: Add stateStorageConfig to
WorkerConfig.

Discussion thread:
https://lists.apache.org/thread/pwfv7nj64frfnbw7jfydzx8my15b3lj6
PR: https://github.com/apache/pulsar/pull/20455

Sincerely
Pengcheng Jiang


Call for Presentations, Community Over Code Asia 2023

2023-06-05 Thread Rich Bowen
You are receiving this message because you are subscribed to one more
more developer mailing lists at the Apache Software Foundation.

The call for presentations is now open at
"https://apachecon.com/acasia2023/cfp.html;, and will be closed by
Sunday, Jun 18th, 2023 11:59 PM GMT.

The event will be held in Beijing, China, August 18-20, 2023.

We are looking for presentations about anything relating to Apache
Software Foundation projects, open-source governance, community, and
software development.
In particular, this year we are building content tracks around the
following specific topics/projects:

AI / Machine learning
API / Microservice
Community
CloudNative
Data Storage & Computing
DataOps
Data Lake & Data Warehouse
OLAP & Data Analysis
Performance Engineering
Incubator
IoT/IIoT
Messaging
RPC
Streaming
Workflow / Data Processing
Web Server / Tomcat

If your proposed presentation falls into one of these categories,
please select that topic in the CFP entry form. Or select Others if
it’s related to another topic or project area.

Looking forward to hearing from you!

Willem Jiang, and the Community Over Code planners



[DISCUSS] PIP-274: Support pluggable topic compactor

2023-06-05 Thread Cong Zhao
Hi Pulsar Community,

I am writing to start the discussion on PIP 274 to support pluggable topic 
compactor

PR with PIP contents: https://github.com/apache/pulsar/pull/20493

Thanks,
Cong Zhao