Re: [DISCUSS] [PIP-165] Auto release client useless connections

[Yubiao Feng]

We can make sure there is at least one connection in the pool for lookup,
but we can't reduce the time it takes to reconnect to ownership. Do you
have a good idea.

On Fri, May 27, 2022 at 10:15 AM Haiting Jiang 
wrote:

> Hi Yubiao,
>
> It seems one possible downside of this pip is increasing produce latency
> when topic ownership changed, any way to avoid that?
>
> Thanks,
> Haiting
>
>
> On 2022/05/26 06:31:37 Yubiao Feng wrote:
> > I open a pip to discuss Auto release client useless connections, could
> you
> > help me review
> >
> >
> > ## Motivation
> > Currently, the Pulsar client keeps the connection even if no producers or
> > consumers use this connection.
> > If a client produces messages to topic A and we have 3 brokers 1, 2, 3.
> Due
> > to the bundle unloading(load manager)
> > topic ownership will change from A to B and finally to C. For now, the
> > client-side will keep 3 connections to all 3 brokers.
> > We can optimize this part to reduce the broker side connections, the
> client
> > should close the unused connections.
> >
> > So a mechanism needs to be added to release unwanted connections.
> >
> > ### Why are there idle connections?
> >
> > 1.When configuration `maxConnectionsPerHosts ` is not set to 0, the
> > connection is not closed at all.
> > The design is to hold a fixed number of connections per Host, avoiding
> > frequent closing and creation.
> >
> >
> https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConnectionPool.java#L325-L335
> >
> > 2-1. When clients receive `command-close`, will reconnect immediately.
> > It's designed to make it possible to reconnect, rebalance, and unload.
> >
> >
> https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConnectionHandler.java#L122-L141
> >
> > 2-2. The broker will close client connections before writing ownership
> info
> > to the ZK. Then clients will get deprecated broker address when it tries
> > lookup.
> >
> >
> https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java#L1282-L1293
> >
> > ## Goal
> > Automatically release connections that are no longer used.
> >
> > - Scope
> >   - **Pulsar client**
> > Contains connections used by consumers, Producers, and Transactions.
> >
> >   - **Pulsar proxy**
> > Contains only the connection between Proxy and broker
> >
> > ## Approach
> > Periodically check for idle connections and close them.
> >
> > ## Changes
> >
> > ### API changes
> > **ClientCnx** added an idle check method to mark idle time.
> >
> > ```java
> > /** Create time. **/
> > private final long createTime;
> > /** The time when marks the connection is idle. **/
> > private long IdleMarkTime;
> > /** The time when the last valid data was transmitted. **/
> > private long lastWorkTime;
> > /** Stat. enumerated values: using, idle_marked, before_release,
> released**/
> > private int stat;
> > /**
> >   * Check client connection is now free. This method may change the state
> > to idle.
> >   * This method will not change the state to idle.
> >   */
> > public boolen doIdleCheck();
> > /** Get stat **/
> > public int getStat();
> > /** Change stat **/
> > public int setStat(int originalStat, int newStat);
> > ```
> >
> > ### Configuration changes
> > We can set the check frequency and release rule for idle connections at
> > `ClientConfigurationData`.
> >
> > ```java
> > @ApiModelProperty(
> > name = "autoReleaseIdleConnectionsEnabled",
> > value = "Do you want to automatically clean up unused
> connections"
> > )
> > private boolean autoReleaseIdleConnectionsEnabled = true;
> >
> > @ApiModelProperty(
> > name = "connectionMaxIdleSeconds",
> > value = "Release the connection if it is not used for more than
> > [connectionMaxIdleSeconds] seconds"
> > )
> > private int connectionMaxIdleSeconds = 180;
> >
> > @ApiModelProperty(
> > name = "connectionIdleDetectionIntervalSeconds",
> > value = "How often check idle connections"
> > )
> > private int connectionIdleDetectionIntervalSeconds = 60;
> > ```
> >
> > ## Implementation
> >
> > - **Pulsar client**
> > If no consumer, producer, or transaction uses the current connection,
> > release it.
> >
> > - **Pulsar proxy**
> > If the connection has not transmitted valid data for a long time, release
> > it.
> >
> >
> > Yubiao Feng
> > Thanks
> >
>

Re: [DISCUSS] [PIP-165] Auto release client useless connections

[Haiting Jiang]

Hi Yubiao,

It seems one possible downside of this pip is increasing produce latency when 
topic ownership changed, any way to avoid that? 

Thanks,
Haiting


On 2022/05/26 06:31:37 Yubiao Feng wrote:
> I open a pip to discuss Auto release client useless connections, could you
> help me review
> 
> 
> ## Motivation
> Currently, the Pulsar client keeps the connection even if no producers or
> consumers use this connection.
> If a client produces messages to topic A and we have 3 brokers 1, 2, 3. Due
> to the bundle unloading(load manager)
> topic ownership will change from A to B and finally to C. For now, the
> client-side will keep 3 connections to all 3 brokers.
> We can optimize this part to reduce the broker side connections, the client
> should close the unused connections.
> 
> So a mechanism needs to be added to release unwanted connections.
> 
> ### Why are there idle connections?
> 
> 1.When configuration `maxConnectionsPerHosts ` is not set to 0, the
> connection is not closed at all.
> The design is to hold a fixed number of connections per Host, avoiding
> frequent closing and creation.
> 
> https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConnectionPool.java#L325-L335
> 
> 2-1. When clients receive `command-close`, will reconnect immediately.
> It's designed to make it possible to reconnect, rebalance, and unload.
> 
> https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConnectionHandler.java#L122-L141
> 
> 2-2. The broker will close client connections before writing ownership info
> to the ZK. Then clients will get deprecated broker address when it tries
> lookup.
> 
> https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java#L1282-L1293
> 
> ## Goal
> Automatically release connections that are no longer used.
> 
> - Scope
>   - **Pulsar client**
> Contains connections used by consumers, Producers, and Transactions.
> 
>   - **Pulsar proxy**
> Contains only the connection between Proxy and broker
> 
> ## Approach
> Periodically check for idle connections and close them.
> 
> ## Changes
> 
> ### API changes
> **ClientCnx** added an idle check method to mark idle time.
> 
> ```java
> /** Create time. **/
> private final long createTime;
> /** The time when marks the connection is idle. **/
> private long IdleMarkTime;
> /** The time when the last valid data was transmitted. **/
> private long lastWorkTime;
> /** Stat. enumerated values: using, idle_marked, before_release, released**/
> private int stat;
> /**
>   * Check client connection is now free. This method may change the state
> to idle.
>   * This method will not change the state to idle.
>   */
> public boolen doIdleCheck();
> /** Get stat **/
> public int getStat();
> /** Change stat **/
> public int setStat(int originalStat, int newStat);
> ```
> 
> ### Configuration changes
> We can set the check frequency and release rule for idle connections at
> `ClientConfigurationData`.
> 
> ```java
> @ApiModelProperty(
> name = "autoReleaseIdleConnectionsEnabled",
> value = "Do you want to automatically clean up unused connections"
> )
> private boolean autoReleaseIdleConnectionsEnabled = true;
> 
> @ApiModelProperty(
> name = "connectionMaxIdleSeconds",
> value = "Release the connection if it is not used for more than
> [connectionMaxIdleSeconds] seconds"
> )
> private int connectionMaxIdleSeconds = 180;
> 
> @ApiModelProperty(
> name = "connectionIdleDetectionIntervalSeconds",
> value = "How often check idle connections"
> )
> private int connectionIdleDetectionIntervalSeconds = 60;
> ```
> 
> ## Implementation
> 
> - **Pulsar client**
> If no consumer, producer, or transaction uses the current connection,
> release it.
> 
> - **Pulsar proxy**
> If the connection has not transmitted valid data for a long time, release
> it.
> 
> 
> Yubiao Feng
> Thanks
> 

[GitHub] [pulsar-site] Anonymitaet commented on pull request #97: Sync language tab choices

[GitBox]

Anonymitaet commented on PR #97:
URL: https://github.com/apache/pulsar-site/pull/97#issuecomment-1139196816

   @SignorMercurio for this improvement, can you open an issue in the Pulsar 
repo and label it w/ `doc-required`? Thanks
   
   > Note that some pages use options like pulsar-admin, REST API and Java, 
while others use Admin CLI, REST API and Java Admin API. Not sure if we should 
make these options consistent, but I've ensured that the options on the same 
page is consistent so that the tab choices are synced on the same page.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] Anonymitaet commented on pull request #97: Sync language tab choices

[GitBox]

Anonymitaet commented on PR #97:
URL: https://github.com/apache/pulsar-site/pull/97#issuecomment-1139196817

   @SignorMercurio for this improvement, can you open an issue in the Pulsar 
repo and label it w/ `doc-required`? Thanks
   
   > Note that some pages use options like pulsar-admin, REST API and Java, 
while others use Admin CLI, REST API and Java Admin API. Not sure if we should 
make these options consistent, but I've ensured that the options on the same 
page is consistent so that the tab choices are synced on the same page.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] Anonymitaet commented on pull request #97: Sync language tab choices

[GitBox]

Anonymitaet commented on PR #97:
URL: https://github.com/apache/pulsar-site/pull/97#issuecomment-1139195830

   @SignorMercurio yes, source doc files live in 
https://github.com/apache/pulsar/tree/master/site2, the website build process 
is done in https://github.com/apache/pulsar-site


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] Anonymitaet commented on pull request #97: Sync language tab choices

[GitBox]

Anonymitaet commented on PR #97:
URL: https://github.com/apache/pulsar-site/pull/97#issuecomment-1139195829

   @SignorMercurio yes, source doc files live in 
https://github.com/apache/pulsar/tree/master/site2, the website build process 
is done in https://github.com/apache/pulsar-site


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] urfreespace commented on pull request #97: Sync language tab choices

[GitBox]

urfreespace commented on PR #97:
URL: https://github.com/apache/pulsar-site/pull/97#issuecomment-1139194263

   > @SignorMercurio yes we need to make the texts for options consistent. Is 
inconsistency a blocker for your PR? I guess not?
   > 
   > @urfreespace can you help review the PR?
   
   @Anonymitaet @SignorMercurio good job, but you summit these docs changes 
from repo `apache/pulsar` 
https://github.com/apache/pulsar/tree/master/site2/docs, In fact, 
`apache/pulsar-site` will auto-sync docs changes from `apache/pulsar` every 6 
hours


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] urfreespace merged pull request #98: Add committer 'zike'

[GitBox]

urfreespace merged PR #98:
URL: https://github.com/apache/pulsar-site/pull/98


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] Anonymitaet commented on pull request #97: Sync language tab choices

[GitBox]

Anonymitaet commented on PR #97:
URL: https://github.com/apache/pulsar-site/pull/97#issuecomment-1139147882

   @SignorMercurio yes we need to make the texts for options consistent. Is 
inconsistency a blocker for your PR? I guess not?
   
   @urfreespace can you help review the PR? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] [PIP-165] Auto release client useless connections

[Neng Lu]

This is a good idea.

Also one thing i realize is people are replying with +1, is this considered
as a vote?

On Thu, May 26, 2022 at 6:26 AM mattison chao 
wrote:

> +1
>
> Best,
> Mattison
>
> On Thu, 26 May 2022 at 15:25, Gavin Gao  wrote:
>
> > +1
> >
> >
> > On 2022/05/26 06:31:37 Yubiao Feng wrote:
> > > I open a pip to discuss Auto release client useless connections, could
> > you
> > > help me review
> > >
> > >
> > > ## Motivation
> > > Currently, the Pulsar client keeps the connection even if no producers
> or
> > > consumers use this connection.
> > > If a client produces messages to topic A and we have 3 brokers 1, 2, 3.
> > Due
> > > to the bundle unloading(load manager)
> > > topic ownership will change from A to B and finally to C. For now, the
> > > client-side will keep 3 connections to all 3 brokers.
> > > We can optimize this part to reduce the broker side connections, the
> > client
> > > should close the unused connections.
> > >
> > > So a mechanism needs to be added to release unwanted connections.
> > >
> > > ### Why are there idle connections?
> > >
> > > 1.When configuration `maxConnectionsPerHosts ` is not set to 0, the
> > > connection is not closed at all.
> > > The design is to hold a fixed number of connections per Host, avoiding
> > > frequent closing and creation.
> > >
> > >
> >
> https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConnectionPool.java#L325-L335
> > >
> > > 2-1. When clients receive `command-close`, will reconnect immediately.
> > > It's designed to make it possible to reconnect, rebalance, and unload.
> > >
> > >
> >
> https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConnectionHandler.java#L122-L141
> > >
> > > 2-2. The broker will close client connections before writing ownership
> > info
> > > to the ZK. Then clients will get deprecated broker address when it
> tries
> > > lookup.
> > >
> > >
> >
> https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java#L1282-L1293
> > >
> > > ## Goal
> > > Automatically release connections that are no longer used.
> > >
> > > - Scope
> > >   - **Pulsar client**
> > > Contains connections used by consumers, Producers, and Transactions.
> > >
> > >   - **Pulsar proxy**
> > > Contains only the connection between Proxy and broker
> > >
> > > ## Approach
> > > Periodically check for idle connections and close them.
> > >
> > > ## Changes
> > >
> > > ### API changes
> > > **ClientCnx** added an idle check method to mark idle time.
> > >
> > > ```java
> > > /** Create time. **/
> > > private final long createTime;
> > > /** The time when marks the connection is idle. **/
> > > private long IdleMarkTime;
> > > /** The time when the last valid data was transmitted. **/
> > > private long lastWorkTime;
> > > /** Stat. enumerated values: using, idle_marked, before_release,
> > released**/
> > > private int stat;
> > > /**
> > >   * Check client connection is now free. This method may change the
> state
> > > to idle.
> > >   * This method will not change the state to idle.
> > >   */
> > > public boolen doIdleCheck();
> > > /** Get stat **/
> > > public int getStat();
> > > /** Change stat **/
> > > public int setStat(int originalStat, int newStat);
> > > ```
> > >
> > > ### Configuration changes
> > > We can set the check frequency and release rule for idle connections at
> > > `ClientConfigurationData`.
> > >
> > > ```java
> > > @ApiModelProperty(
> > > name = "autoReleaseIdleConnectionsEnabled",
> > > value = "Do you want to automatically clean up unused
> > connections"
> > > )
> > > private boolean autoReleaseIdleConnectionsEnabled = true;
> > >
> > > @ApiModelProperty(
> > > name = "connectionMaxIdleSeconds",
> > > value = "Release the connection if it is not used for more than
> > > [connectionMaxIdleSeconds] seconds"
> > > )
> > > private int connectionMaxIdleSeconds = 180;
> > >
> > > @ApiModelProperty(
> > > name = "connectionIdleDetectionIntervalSeconds",
> > > value = "How often check idle connections"
> > > )
> > > private int connectionIdleDetectionIntervalSeconds = 60;
> > > ```
> > >
> > > ## Implementation
> > >
> > > - **Pulsar client**
> > > If no consumer, producer, or transaction uses the current connection,
> > > release it.
> > >
> > > - **Pulsar proxy**
> > > If the connection has not transmitted valid data for a long time,
> release
> > > it.
> > >
> > >
> > > Yubiao Feng
> > > Thanks
> > >
> >
>


-- 
Best Regards,
Neng

Re: [Discuss] Update Helm Chart to Support 2.10 Docker Image

[Neng Lu]

Hi All,

I'm curious to learn once the image is run as non-root, how can we debug or
investigate production issues inside a running cluster?

On Thu, May 19, 2022 at 12:14 PM Michael Marshall 
wrote:

> Hello Pulsar Community,
>
> With the 2.10.0 release, our Pulsar Docker images default to run as a
> non-root user. In order to use the 2.10.0 Docker image with the Apache
> Pulsar Helm Chart, we need to merge this PR [0]. If you're able,
> please review it. Once merged, I propose that we follow up with a
> release so that users wanting to upgrade to 2.10.0 have an upgrade
> path.
>
> Thanks,
> Michael
>
> [0] https://github.com/apache/pulsar-helm-chart/pull/266
>


-- 
Best Regards,
Neng

Re: [DISCUSS] PIP-157 Exclusive Producer: new mode ExclusiveWithFencing

[Neng Lu]

Hi All,

Would "Preemptive" Mode make sense?

On Wed, May 11, 2022 at 8:56 AM Matteo Merli  wrote:

> +1
>
>
> On Tue, May 10, 2022 at 5:56 AM Enrico Olivelli 
> wrote:
>
> > Hello,
> > I created a new PIP about a new AccessMode for the Producer.
> > https://github.com/apache/pulsar/issues/15528
> >
> > This is the PR: https://github.com/apache/pulsar/pull/15488
> >
> > Honestly I don't like the name "ExclusiveWithFencing", any suggestion
> > is really appreciated.
> >
> > Enrico
> >
> > Motivation
> >
> > In PIP-68 we introduced two access modes for the Producer:
> >
> > Exclusive: The producer is the only one who can publish to the topic.
> > Fail if there is another Exclusive Producer connected to the topic
> > while creating the new Producer.
> > WaitForExclusive: Like Exclusive, but instead of Failing we are going
> > to wait for the current Exclusive Producer to disconnect.
> >
> > Those two modes are very powerful and allow you to perform some kind
> > of Locking on a topic.
> >
> > We are missing a third more, in which the Producer always succeeds to
> > acquire the Exclusive lock on the topic by fencing out any other
> > Producer that is connected, even the current Exclusive Producer and
> > the other Producers waiting in WaitForExclusive mode.
> >
> > Goal
> >
> > The modes that are available with PIP-68 require a writer to acquire
> > the lock and release it as soon as possible in order to allow other
> > clients to write to the topic.
> >
> > With the new mode it will be possible to implement locking in another
> > way: the Producer holds the lock until someone else steals it. This
> > way when you have very low contention you can achieve better latency
> > for writes because you don't have to acquire the lock every time you
> > want to write,.
> >
> > API and Wire protocol Changes
> >
> > Changes:
> >
> > a new constant on the Wire Protocol for AccessMode
> > a new constant in the Java Client API AccessMode#ExclusiveWithFencing
> >
> > Implementation
> >
> > The new mode will behave mostly like AccessMode#Exclusive but instead
> > of failing in case of the presence of other Producers it will force
> > all of the current connected Producers to be removed and invalidated
> > (they will see ProducerFencedException).
> >
> > Reject Alternatives
> >
> > None
> >
> --
> --
> Matteo Merli
> 
>


-- 
Best Regards,
Neng

Re: [DISCUSS] PIP-166: Function add NONE delivery semantics

[Neng Lu]

Some suggestions:

1. Instead of deleting the `autoAck`, keep it but not use it in the code.
And documented clearly it's deprecated for the following 2~3 release. And
then delete it.
2. For `PulsarSinkAtLeastOnceProcessor` and
`PulsarSinkEffectivelyOnceProcessor`, if `NONE` is configured, it defaults
to ATLEAST_ONCE.
3. Need to let users know the behavior when they call `record.ack()` inside
the function implementation.

On Thu, May 12, 2022 at 1:52 AM Baozi 
wrote:

> Hi Pulsar community,
>
> I open a https://github.com/apache/pulsar/issues/15560 for Function add
> NONE delivery semantics
>
> Let me know what you think.
>
>
> Thanks,
> Baodi Shi
>
>
> ## Motivation
>
> Currently Function supports three delivery semantics, and also provides
> autoAck to control whether to automatically ack.
> Because autoAck affects the delivery semantics of Function, it can be
> confusing for users to understand the relationship between these two
> parameters.
>
> For example, when the user configures `Guarantees == ATMOST_ONCE` and
> `autoAck == false`, then the framework will not help the user to ack
> messages, and the processing semantics may become `ATLEAST_ONCE`.
>
> The delivery semantics provided by Function should be clear. When the user
> sets the guarantees, the framework should ensure point-to-point semantic
> processing and cannot be affected by other parameters.
>
> ## Goal
>
> Added `NONE` delivery semantics and delete `autoAck` config.
>
> The original intention of `autoAck` semantics is that users want to
> control the timing of ack by themselves. When autoAck == false, the
> processing semantics provided by the framework should be invalid. Then we
> can add `NONE` processing semantics to replace the autoAck == false
> scenario.
>
> When the user configuration `ProcessingGuarantees == NONE`, the framework
> does not help the user to do any ack operations, and the ack is left to the
> user to handle. In other cases, the framework guarantees processing
> semantics.
>
> ## API Changes
> 1. Add `NONE` type to ProcessingGuarantees
> ``` java
> public enum ProcessingGuarantees {
>   ATLEAST_ONCE,
>   ATMOST_ONCE,
>   EFFECTIVELY_ONCE,
>   NONE
> }
> ```
>
> 2. Delete autoAck config in FunctionConfig
> ``` java
> public class FunctionConfig {
> -private Boolean autoAck;
> }
> ```
>
> ## Implementation
>
> 1. In `PulsarSinkAtLeastOnceProcessor` and
> `PulsarSinkEffectivelyOnceProcessor`, when `ProcessingGuarantees != NONE`
> can be ack.
>
> <
> https://github.com/apache/pulsar/blob/c49a977de4b0b525ec80e5070bc90eddcc7cddad/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/PulsarSink.java#L274-L276
> >
>
> 2. When the delivery semantic is `ATMOST_ONCE`, the message will be acked
> immediately after receiving the message, no longer affected by the autoAck
> configuration.
>
>
> https://github.com/apache/pulsar/blob/c49a977de4b0b525ec80e5070bc90eddcc7cddad/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java#L271-L276
>
> 3. When user call `record.ack()` in function, just  `ProcessingGuarantees
> == NONE` can be work.
>
> ## Plan test
> The main test and assert is that when ProcessingGuarantees == NONE, the
> function framework will not do any ack operations for the user.
>
> ## Compatibility
> 1. This change will invalidate the user's setting of autoAck, which should
> be explained in the documentation and provide parameter verification to
> remind the user.
> 2. Runtimes of other languages ​​need to maintain consistent processing
> logic (python, go).
>
>
>

-- 
Best Regards,
Neng

Re: [VOTE] [PIP-153] Optimize metadataPositions in MLPendingAckStore

[Neng Lu]

Hi All,

+1 (non-binding)

On Mon, May 23, 2022 at 5:09 AM Enrico Olivelli  wrote:

> +1 (binding)
> it looks like the patch has already been committed
>
>
> https://github.com/apache/pulsar/commit/ebca19b522fd9f4496689ca7d32ede345d28511a
>
> Enrico
>
> Il giorno lun 16 mag 2022 alle ore 07:18 Hang Chen
>  ha scritto:
> >
> > +1 (binding)
> >
> > Thanks,
> > Hang
> >
> > mattison chao  于2022年5月12日周四 21:17写道：
> > >
> > > +1 (non-binding)
> > >
> > > Best,
> > > Mattison
> > >
> > > On Thu, 12 May 2022 at 12:15, Ran Gao  wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > Best,
> > > > Ran
> > > >
> > > > On 2022/04/25 06:54:58 一苇以航 wrote:
> > > > > Hi Pulsar Community.
> > > > >
> > > > > This is the voting thread for PIP-153. It will stay open for at
> least 48
> > > > hours.
> > > > >
> > > > > The proposal can be found:
> https://github.com/apache/pulsar/issues/15073
> > > > >
> > > > > Discuss thread:
> > > > > https://lists.apache.org/thread/svmbp8ybn6l8o0o8dmvsysnb86qj77r3
> > > > >
> > > > > Thanks,
> > > > > Xiangying
> > > > >
> > > > > 
> > > >
>

[GitHub] [pulsar-helm-chart] oguzhantortop commented on issue #132: kop

[GitBox]

oguzhantortop commented on issue #132:
URL: 
https://github.com/apache/pulsar-helm-chart/issues/132#issuecomment-1138883377

   > Here is a chart which contains kop and mop 
https://github.com/streamnative/charts/blob/master/charts/pulsar/values.yaml#L94
   
   hi, isn't this chart based on streamnative platform instead of opensource 
pulsar. So just changing images would work or not? Or should we get the pieces 
which looks for component.kop boolean and merge them into opensource pulsar 
charts. Also can someone tell why do we need istio and envoy proxy for kop 
installations?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: Discuss about closing Pulsar resource

[Nicolò Boschi]

Hi,

the initial intention of https://github.com/apache/pulsar/pull/15638 was
about fixing thread leaks in tests which lead to Mockito weird behaviours.
Now a few executors do not wait for thread termination and it makes the
tests unstable, mainly with mocked objects. The AsyncCloseable addresses a
different problem so I will create another pull for that.

In order to respect the brokerShutdownTimeoutMs we definitely need that
every component handled by the BrokerService support the async closing.
Other than adding the interface and making all those classes implement it,
we need to ensure the order of components being closed will not lead to
deadlocks (which is a current problem) and it's not trivial.
Another important aspect is that now brokerShutdownTimeoutMs has a default
value which respects all the components handled in the async executor but
there are other components not included in this shutdown time. So it is
possible that we'd need to set a higher default value.

A good plan would be to create an ad-hoc pull request for the
"AsyncCloneable" interface where all the work is addressed in one shot.

Nicolò Boschi


Il giorno gio 26 mag 2022 alle ore 03:57 guo jiwei 
ha scritto:

> Hi community:
> In the past few days, when investigating the problem of pulsar close
> being hung, I saw that many resources are called synchronously and have no
> timeout when they are closed. To solve this problem,
> patch-15638 , add the async
> close method and timeout.
> patch-15680 , uniformly
> increase the timeout in the pulsar close method.
> patch-15777 , add timeout in
> the close method.
> However, neither increasing the async interface nor the timeout is
> guaranteed to shut down the pulsar service within
> `brokerShutdownTimeoutMs`.
>
> In order to ensure the uniformity of resource closure, I hope to discuss
> with you a relatively reasonable solution.
>
>
> Regards
> Jiwei Guo (Tboy)
>

[VOTE] PIP-168: Support zero-copy of NIC to NIC on Proxy

[zhaocong]

Hi Pulsar Community,


I would like to start a VOTE on "Support zero-copy of NIC to NIC on Proxy"
(PIP-168).


The proposal can be read at https://github.com/apache/pulsar/issues/15631

and the discussion thead is available at

https://lists.apache.org/thread/gjys9tvbd5hy28mbkbcq7wkqfldycn7v


Voting will stay open for at least 48h.


Thanks,

Cong Zhao

Re: [DISCUSS] [PIP-165] Auto release client useless connections

[mattison chao]

+1

Best,
Mattison

On Thu, 26 May 2022 at 15:25, Gavin Gao  wrote:

> +1
>
>
> On 2022/05/26 06:31:37 Yubiao Feng wrote:
> > I open a pip to discuss Auto release client useless connections, could
> you
> > help me review
> >
> >
> > ## Motivation
> > Currently, the Pulsar client keeps the connection even if no producers or
> > consumers use this connection.
> > If a client produces messages to topic A and we have 3 brokers 1, 2, 3.
> Due
> > to the bundle unloading(load manager)
> > topic ownership will change from A to B and finally to C. For now, the
> > client-side will keep 3 connections to all 3 brokers.
> > We can optimize this part to reduce the broker side connections, the
> client
> > should close the unused connections.
> >
> > So a mechanism needs to be added to release unwanted connections.
> >
> > ### Why are there idle connections?
> >
> > 1.When configuration `maxConnectionsPerHosts ` is not set to 0, the
> > connection is not closed at all.
> > The design is to hold a fixed number of connections per Host, avoiding
> > frequent closing and creation.
> >
> >
> https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConnectionPool.java#L325-L335
> >
> > 2-1. When clients receive `command-close`, will reconnect immediately.
> > It's designed to make it possible to reconnect, rebalance, and unload.
> >
> >
> https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConnectionHandler.java#L122-L141
> >
> > 2-2. The broker will close client connections before writing ownership
> info
> > to the ZK. Then clients will get deprecated broker address when it tries
> > lookup.
> >
> >
> https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java#L1282-L1293
> >
> > ## Goal
> > Automatically release connections that are no longer used.
> >
> > - Scope
> >   - **Pulsar client**
> > Contains connections used by consumers, Producers, and Transactions.
> >
> >   - **Pulsar proxy**
> > Contains only the connection between Proxy and broker
> >
> > ## Approach
> > Periodically check for idle connections and close them.
> >
> > ## Changes
> >
> > ### API changes
> > **ClientCnx** added an idle check method to mark idle time.
> >
> > ```java
> > /** Create time. **/
> > private final long createTime;
> > /** The time when marks the connection is idle. **/
> > private long IdleMarkTime;
> > /** The time when the last valid data was transmitted. **/
> > private long lastWorkTime;
> > /** Stat. enumerated values: using, idle_marked, before_release,
> released**/
> > private int stat;
> > /**
> >   * Check client connection is now free. This method may change the state
> > to idle.
> >   * This method will not change the state to idle.
> >   */
> > public boolen doIdleCheck();
> > /** Get stat **/
> > public int getStat();
> > /** Change stat **/
> > public int setStat(int originalStat, int newStat);
> > ```
> >
> > ### Configuration changes
> > We can set the check frequency and release rule for idle connections at
> > `ClientConfigurationData`.
> >
> > ```java
> > @ApiModelProperty(
> > name = "autoReleaseIdleConnectionsEnabled",
> > value = "Do you want to automatically clean up unused
> connections"
> > )
> > private boolean autoReleaseIdleConnectionsEnabled = true;
> >
> > @ApiModelProperty(
> > name = "connectionMaxIdleSeconds",
> > value = "Release the connection if it is not used for more than
> > [connectionMaxIdleSeconds] seconds"
> > )
> > private int connectionMaxIdleSeconds = 180;
> >
> > @ApiModelProperty(
> > name = "connectionIdleDetectionIntervalSeconds",
> > value = "How often check idle connections"
> > )
> > private int connectionIdleDetectionIntervalSeconds = 60;
> > ```
> >
> > ## Implementation
> >
> > - **Pulsar client**
> > If no consumer, producer, or transaction uses the current connection,
> > release it.
> >
> > - **Pulsar proxy**
> > If the connection has not transmitted valid data for a long time, release
> > it.
> >
> >
> > Yubiao Feng
> > Thanks
> >
>

[GitHub] [pulsar-site] urfreespace merged pull request #99: backup old website next version docs to website directory

[GitBox]

urfreespace merged PR #99:
URL: https://github.com/apache/pulsar-site/pull/99


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] [PIP-165] Auto release client useless connections

[Gavin Gao]

+1


On 2022/05/26 06:31:37 Yubiao Feng wrote:
> I open a pip to discuss Auto release client useless connections, could you
> help me review
> 
> 
> ## Motivation
> Currently, the Pulsar client keeps the connection even if no producers or
> consumers use this connection.
> If a client produces messages to topic A and we have 3 brokers 1, 2, 3. Due
> to the bundle unloading(load manager)
> topic ownership will change from A to B and finally to C. For now, the
> client-side will keep 3 connections to all 3 brokers.
> We can optimize this part to reduce the broker side connections, the client
> should close the unused connections.
> 
> So a mechanism needs to be added to release unwanted connections.
> 
> ### Why are there idle connections?
> 
> 1.When configuration `maxConnectionsPerHosts ` is not set to 0, the
> connection is not closed at all.
> The design is to hold a fixed number of connections per Host, avoiding
> frequent closing and creation.
> 
> https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConnectionPool.java#L325-L335
> 
> 2-1. When clients receive `command-close`, will reconnect immediately.
> It's designed to make it possible to reconnect, rebalance, and unload.
> 
> https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConnectionHandler.java#L122-L141
> 
> 2-2. The broker will close client connections before writing ownership info
> to the ZK. Then clients will get deprecated broker address when it tries
> lookup.
> 
> https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java#L1282-L1293
> 
> ## Goal
> Automatically release connections that are no longer used.
> 
> - Scope
>   - **Pulsar client**
> Contains connections used by consumers, Producers, and Transactions.
> 
>   - **Pulsar proxy**
> Contains only the connection between Proxy and broker
> 
> ## Approach
> Periodically check for idle connections and close them.
> 
> ## Changes
> 
> ### API changes
> **ClientCnx** added an idle check method to mark idle time.
> 
> ```java
> /** Create time. **/
> private final long createTime;
> /** The time when marks the connection is idle. **/
> private long IdleMarkTime;
> /** The time when the last valid data was transmitted. **/
> private long lastWorkTime;
> /** Stat. enumerated values: using, idle_marked, before_release, released**/
> private int stat;
> /**
>   * Check client connection is now free. This method may change the state
> to idle.
>   * This method will not change the state to idle.
>   */
> public boolen doIdleCheck();
> /** Get stat **/
> public int getStat();
> /** Change stat **/
> public int setStat(int originalStat, int newStat);
> ```
> 
> ### Configuration changes
> We can set the check frequency and release rule for idle connections at
> `ClientConfigurationData`.
> 
> ```java
> @ApiModelProperty(
> name = "autoReleaseIdleConnectionsEnabled",
> value = "Do you want to automatically clean up unused connections"
> )
> private boolean autoReleaseIdleConnectionsEnabled = true;
> 
> @ApiModelProperty(
> name = "connectionMaxIdleSeconds",
> value = "Release the connection if it is not used for more than
> [connectionMaxIdleSeconds] seconds"
> )
> private int connectionMaxIdleSeconds = 180;
> 
> @ApiModelProperty(
> name = "connectionIdleDetectionIntervalSeconds",
> value = "How often check idle connections"
> )
> private int connectionIdleDetectionIntervalSeconds = 60;
> ```
> 
> ## Implementation
> 
> - **Pulsar client**
> If no consumer, producer, or transaction uses the current connection,
> release it.
> 
> - **Pulsar proxy**
> If the connection has not transmitted valid data for a long time, release
> it.
> 
> 
> Yubiao Feng
> Thanks
> 

Re: [DISCUSS] [PIP-165] Auto release client useless connections

[Enrico Olivelli]

Make sense to me

Enrico

Il giorno gio 26 mag 2022 alle ore 08:31 Yubiao Feng
 ha scritto:
>
> I open a pip to discuss Auto release client useless connections, could you
> help me review
>
>
> ## Motivation
> Currently, the Pulsar client keeps the connection even if no producers or
> consumers use this connection.
> If a client produces messages to topic A and we have 3 brokers 1, 2, 3. Due
> to the bundle unloading(load manager)
> topic ownership will change from A to B and finally to C. For now, the
> client-side will keep 3 connections to all 3 brokers.
> We can optimize this part to reduce the broker side connections, the client
> should close the unused connections.
>
> So a mechanism needs to be added to release unwanted connections.
>
> ### Why are there idle connections?
>
> 1.When configuration `maxConnectionsPerHosts ` is not set to 0, the
> connection is not closed at all.
> The design is to hold a fixed number of connections per Host, avoiding
> frequent closing and creation.
>
> https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConnectionPool.java#L325-L335
>
> 2-1. When clients receive `command-close`, will reconnect immediately.
> It's designed to make it possible to reconnect, rebalance, and unload.
>
> https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConnectionHandler.java#L122-L141
>
> 2-2. The broker will close client connections before writing ownership info
> to the ZK. Then clients will get deprecated broker address when it tries
> lookup.
>
> https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java#L1282-L1293
>
> ## Goal
> Automatically release connections that are no longer used.
>
> - Scope
>   - **Pulsar client**
> Contains connections used by consumers, Producers, and Transactions.
>
>   - **Pulsar proxy**
> Contains only the connection between Proxy and broker
>
> ## Approach
> Periodically check for idle connections and close them.
>
> ## Changes
>
> ### API changes
> **ClientCnx** added an idle check method to mark idle time.
>
> ```java
> /** Create time. **/
> private final long createTime;
> /** The time when marks the connection is idle. **/
> private long IdleMarkTime;
> /** The time when the last valid data was transmitted. **/
> private long lastWorkTime;
> /** Stat. enumerated values: using, idle_marked, before_release, released**/
> private int stat;
> /**
>   * Check client connection is now free. This method may change the state
> to idle.
>   * This method will not change the state to idle.
>   */
> public boolen doIdleCheck();
> /** Get stat **/
> public int getStat();
> /** Change stat **/
> public int setStat(int originalStat, int newStat);
> ```
>
> ### Configuration changes
> We can set the check frequency and release rule for idle connections at
> `ClientConfigurationData`.
>
> ```java
> @ApiModelProperty(
> name = "autoReleaseIdleConnectionsEnabled",
> value = "Do you want to automatically clean up unused connections"
> )
> private boolean autoReleaseIdleConnectionsEnabled = true;
>
> @ApiModelProperty(
> name = "connectionMaxIdleSeconds",
> value = "Release the connection if it is not used for more than
> [connectionMaxIdleSeconds] seconds"
> )
> private int connectionMaxIdleSeconds = 180;
>
> @ApiModelProperty(
> name = "connectionIdleDetectionIntervalSeconds",
> value = "How often check idle connections"
> )
> private int connectionIdleDetectionIntervalSeconds = 60;
> ```
>
> ## Implementation
>
> - **Pulsar client**
> If no consumer, producer, or transaction uses the current connection,
> release it.
>
> - **Pulsar proxy**
> If the connection has not transmitted valid data for a long time, release
> it.
>
>
> Yubiao Feng
> Thanks

[DISCUSS] [PIP-165] Auto release client useless connections

[Yubiao Feng]

I open a pip to discuss Auto release client useless connections, could you
help me review


## Motivation
Currently, the Pulsar client keeps the connection even if no producers or
consumers use this connection.
If a client produces messages to topic A and we have 3 brokers 1, 2, 3. Due
to the bundle unloading(load manager)
topic ownership will change from A to B and finally to C. For now, the
client-side will keep 3 connections to all 3 brokers.
We can optimize this part to reduce the broker side connections, the client
should close the unused connections.

So a mechanism needs to be added to release unwanted connections.

### Why are there idle connections?

1.When configuration `maxConnectionsPerHosts ` is not set to 0, the
connection is not closed at all.
The design is to hold a fixed number of connections per Host, avoiding
frequent closing and creation.

https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConnectionPool.java#L325-L335

2-1. When clients receive `command-close`, will reconnect immediately.
It's designed to make it possible to reconnect, rebalance, and unload.

https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConnectionHandler.java#L122-L141

2-2. The broker will close client connections before writing ownership info
to the ZK. Then clients will get deprecated broker address when it tries
lookup.

https://github.com/apache/pulsar/blob/72349117c4fd9825adaaf16d3588a695e8a9dd27/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java#L1282-L1293

## Goal
Automatically release connections that are no longer used.

- Scope
  - **Pulsar client**
Contains connections used by consumers, Producers, and Transactions.

  - **Pulsar proxy**
Contains only the connection between Proxy and broker

## Approach
Periodically check for idle connections and close them.

## Changes

### API changes
**ClientCnx** added an idle check method to mark idle time.

```java
/** Create time. **/
private final long createTime;
/** The time when marks the connection is idle. **/
private long IdleMarkTime;
/** The time when the last valid data was transmitted. **/
private long lastWorkTime;
/** Stat. enumerated values: using, idle_marked, before_release, released**/
private int stat;
/**
  * Check client connection is now free. This method may change the state
to idle.
  * This method will not change the state to idle.
  */
public boolen doIdleCheck();
/** Get stat **/
public int getStat();
/** Change stat **/
public int setStat(int originalStat, int newStat);
```

### Configuration changes
We can set the check frequency and release rule for idle connections at
`ClientConfigurationData`.

```java
@ApiModelProperty(
name = "autoReleaseIdleConnectionsEnabled",
value = "Do you want to automatically clean up unused connections"
)
private boolean autoReleaseIdleConnectionsEnabled = true;

@ApiModelProperty(
name = "connectionMaxIdleSeconds",
value = "Release the connection if it is not used for more than
[connectionMaxIdleSeconds] seconds"
)
private int connectionMaxIdleSeconds = 180;

@ApiModelProperty(
name = "connectionIdleDetectionIntervalSeconds",
value = "How often check idle connections"
)
private int connectionIdleDetectionIntervalSeconds = 60;
```

## Implementation

- **Pulsar client**
If no consumer, producer, or transaction uses the current connection,
release it.

- **Pulsar proxy**
If the connection has not transmitted valid data for a long time, release
it.


Yubiao Feng
Thanks