[GitHub] [pulsar-manager] mattisonchao commented on issue #441: Support HerdDB in the standard docker image.

[GitBox]

mattisonchao commented on issue #441:
URL: https://github.com/apache/pulsar-manager/issues/441#issuecomment-1012854673


   I'm current work on it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-manager] mattisonchao opened a new issue #441: Support HerdDB in the standard docker image.

[GitBox]

mattisonchao opened a new issue #441:
URL: https://github.com/apache/pulsar-manager/issues/441


   ## Motivation
   
   Now, The docker image starts PostGRE, and it is a waste of resources. So, we 
can to use HearDB instead.
   
   Some code link: 
https://github.com/apache/pulsar-manager/blob/73e4ae37cdab1d8f1f6c08fd5aaca91c612687b6/docker/startup.sh


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

?????? [VOTE] PIP-129: Introduce intermediate state for ledger deletion

[zhangao]

+1 (non-binding)




----
??: 
   "dev"

https://github.com/apache/pulsar/issues/13526
 
  
 
  ## Motivation
 
  Under the current ledger-trimming design in
 
 `org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl#internalTrimLedgers`,
  we need to collect those ledgers that need to be deleted first, and 
then
  perform the asynchronous deletion of the ledger concurrently, but we 
do
 not
  continue to pay attention to whether the deletion operation is 
completed.
  If the meta-information update has been successfully completed but an
 error
  occurs during the asynchronous deletion, the ledger may not be 
deleted,
 but
  at the logical level we think that the deletion has been completed, 
which
  will make this part of the data remain in the storage layer forever 
(such
  as bk). As the usage time of the cluster becomes longer, the residual
 data
  that cannot be deleted will gradually increase.
 
  In order to achieve this goal, we can separate the logic of
  meta-information update and ledger deletion. In the trimming process, 
we
  can first mark which ledgers are deletable, and update the results to 
the
  metadatastore. We can perform the deletion of marked ledgers
 asynchronously
  in the callback of updating the meta information, so that the original
  logic can be retained seamlessly. Therefore, when we are rolling 
upgrade
 or
  rollback, the only difference is whether the deleted ledger is marked 
for
  deletion.
 
  To be more specific:
  1. for upgrade, only the marker information of ledger has been added, 
and
  the logical sequence of deletion has not changed.
  2. for rollback, some ledgers that have been marked for deletion may 
not
 be
  deleted due to the restart of the broker. This behavior is consistent
 with
  the original version.
 
  In addition, if the ledger that has been marked is not deleted
  successfully, the marker will not be removed. So for this part of
 ledgers,
  every time trimming is triggered, it will be deleted again, which is
  equivalent to a check and retry mechanism.
 
  ## Goal
 
  We need to modify some logic in
 
 `org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl#internalTrimLedgers`
  so that the ledger deletion logic in ledger-trimming is split into two
  stages, marking and deleting. Once the marker information is updated 
to
 the
  metadatastore, every trimming will try to trigger the ledger deletion
 until
  all the deleteable ledgers are successfully deleted.
 
  ## Implementation
 
  This proposal aims to separate the deletion logic in ledger-trimming, 
so
  that `ManagedLedgerImpl#internalTrimLedgers` is responsible for 
marking
 the
  deletable ledgers and then perform actual ledger deletion according to
 the
  metadatastore.
 
  Therefore, the entire trimming process is broken down into the 
following
  steps:
 
  1. mark deletable ledgers and update ledger metadata.
  2. do acutual ledger deletion after metadata is updated.
 
  For step 1, we can store the marker of deletable information in
  `org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl#propertiesMap`.
 When
  retrieving the deleted ledger information, we can directly query by
  iterating `propertiesMap`. If this solution is not accepted, maybe we 
can
  create a new znode to store these information, but this approach will 
not
  be able to reuse the current design.
 
  For step 2, we can perform the deletion of marked ledgers 
asynchronously
 in
  the callback of updating the meta information. And every trimming will
  trigger the check and delete for those deleteable ledgers.
 
  Related PR: https://github.com/apache/pulsar/pull/13575

Re: [VOTE] PIP-121: Pulsar cluster level auto failover on client side

[Hang Chen]

Thanks for your participation.

Close this vote with 3 (+1) bindings and 5 (+1) non-bindings, 0 (-1).

Regards,
Hang

r...@apache.org  于2022年1月12日周三 11:52写道：
>
> +1 (non-binding)
>
> --
> Thanks
> Xiaolong Ran
>
> Zike Yang  于2022年1月12日周三 09:58写道：
>
> > +1 (non-binding)
> >
> > On Wed, Jan 12, 2022 at 9:52 AM Haiting Jiang 
> > wrote:
> > >
> > > +1
> > >
> > > On 2022/01/12 00:09:26 Matteo Merli wrote:
> > > > +1
> > > > --
> > > > Matteo Merli
> > > > 
> > > >
> > > > On Tue, Jan 11, 2022 at 12:07 PM Neng Lu  wrote:
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > On Mon, Jan 10, 2022 at 12:40 AM PengHui Li 
> > wrote:
> > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > > Penghui
> > > > > >
> > > > > > On Mon, Jan 10, 2022 at 4:38 PM Enrico Olivelli <
> > eolive...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > +1 (binding)
> > > > > > >
> > > > > > > Enrico
> > > > > > >
> > > > > > > Il giorno lun 10 gen 2022 alle ore 07:45 Hang Chen
> > > > > > >  ha scritto:
> > > > > > > >
> > > > > > > > This is the voting thread for PIP-121. It will stay open for
> > at least
> > > > > > 48
> > > > > > > > hours.
> > > > > > > >
> > > > > > > > https://github.com/apache/pulsar/issues/13315
> > > > > > > >
> > > > > > > > Pasted below for quoting convenience.
> > > > > > > >
> > > > > > > > -
> > > > > > > > ### Motivation
> > > > > > > > We have geo-replication to support Pulsar cluster level
> > failover. We
> > > > > > > > can set up Pulsar cluster A as a primary cluster in data
> > center A, and
> > > > > > > > setup Pulsar cluster B as backup cluster in data center B.
> > Then we
> > > > > > > > configure geo-replication between cluster A and cluster B. All
> > the
> > > > > > > > clients are connected to the Pulsar cluster by DNS. If cluster
> > A is
> > > > > > > > down, we should switch the DNS to point the target Pulsar
> > cluster from
> > > > > > > > cluster A to cluster B. After the clients are resolved to
> > cluster B,
> > > > > > > > they can produce and consume messages normally. After cluster A
> > > > > > > > recovers, the administrator should switch the DNS back to
> > cluster A.
> > > > > > > >
> > > > > > > > However, the current method has two shortcomings.
> > > > > > > > 1. The administrator should monitor the status of all Pulsar
> > clusters,
> > > > > > > > and switch the DNS as soon as possible when cluster A is down.
> > The
> > > > > > > > switch and recovery is not automatic and recovery time is
> > controlled
> > > > > > > > by the administrator, which will put the administrator under
> > heavy
> > > > > > > > load.
> > > > > > > > 2. The Pulsar client and DNS system have a cache. When the
> > > > > > > > administrator switches the DNS from cluster A to Cluster B, it
> > will
> > > > > > > > take some time for cache trigger timeout, which will delay
> > client
> > > > > > > > recovery time and lead to the product/consumer message failing.
> > > > > > > >
> > > > > > > > ### Goal
> > > > > > > > It's better to provide an automatic cluster level failure
> > recovery
> > > > > > > > mechanism to make pulsar cluster failover more effective. We
> > should
> > > > > > > > support pulsar clients auto switching from cluster A to
> > cluster B when
> > > > > > > > it detects cluster A has been down according to the configured
> > > > > > > > detecting policy and switch back to cluster A when it has
> > recovered.
> > > > > > > > The reason why we should switch back to cluster A is that most
> > > > > > > > applications may be deployed in data center A and they have low
> > > > > > > > network cost for communicating with pulsar cluster A. If they
> > keep
> > > > > > > > visiting pulsar cluster B, they have high network cost, and
> > cause high
> > > > > > > > produce/consume latency.
> > > > > > > >
> > > > > > > > In order to improve the DNS cache problem, we should provide an
> > > > > > > > administrator controlled switch provider for administrators to
> > update
> > > > > > > > service URLs.
> > > > > > > >
> > > > > > > > In the end, we should provide an auto service URL switch
> > provider and
> > > > > > > > administrator controlled switch provider.
> > > > > > > >
> > > > > > > > ### Design
> > > > > > > > We have already provided the `ServiceUrlProvider` interface to
> > support
> > > > > > > > different service URLs. In order to support automatic cluster
> > level
> > > > > > > > failure auto recovery, we can provide different
> > ServiceUrlProvider
> > > > > > > > implementations. For current requirements, we can provide
> > > > > > > > `AutoClusterFailover` and `ControlledClusterFailover`.
> > > > > > > >
> > > > > > > >  AutoClusterFailover
> > > > > > > > In order to support auto switching from the primary cluster to
> > the
> > > > > > > > secondary, we can provide a probe task, which will probe the
> > activity
> > > > > > > > of the primary cluster and the secondary one. When it finds the
> > > > > > > > primary cluster failed more than 

[DISCUSSION] Support custom and pluggable consumer selector

[zhangao]

Hi Pulsar Community, 

When we try to Introduce pulsar to our existing system, compatibility is our 
first consideration.

Firstly, our production system uses jump consistent hash algorithm to select 
consumers, but pulsar uses range consistent hash algorithm natively. It's 
impossible for us to guarantee that the same node can consume data with the 
same routing key from pulsar and our existing system simultaneously.

It's better pulsar supports custom consumer selector when using key shared 
subscription. 



Thanks
Zhangao




[1] https://github.com/apache/pulsar/issues/13473 [2] 
https://github.com/apache/pulsar/pull/13470

Re: [VOTE] PIP-129: Introduce intermediate state for ledger deletion

[mattison chao]

+1 (non-binding)

Best,
Mattison

On Fri, 14 Jan 2022 at 11:19, Hang Chen  wrote:

> +1 (binding)
>
> Best,
> Hang
>
> Zhanpeng Wu  于2022年1月14日周五 10:37写道：
> >
> > This is the voting thread for PIP-129. It will stay open for at least 48
> > hours.  Pasted below for quoting convenience.
> >
> > 
> >
> > https://github.com/apache/pulsar/issues/13526
> >
> > 
> >
> > ## Motivation
> >
> > Under the current ledger-trimming design in
> >
> `org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl#internalTrimLedgers`,
> > we need to collect those ledgers that need to be deleted first, and then
> > perform the asynchronous deletion of the ledger concurrently, but we do
> not
> > continue to pay attention to whether the deletion operation is completed.
> > If the meta-information update has been successfully completed but an
> error
> > occurs during the asynchronous deletion, the ledger may not be deleted,
> but
> > at the logical level we think that the deletion has been completed, which
> > will make this part of the data remain in the storage layer forever (such
> > as bk). As the usage time of the cluster becomes longer, the residual
> data
> > that cannot be deleted will gradually increase.
> >
> > In order to achieve this goal, we can separate the logic of
> > meta-information update and ledger deletion. In the trimming process, we
> > can first mark which ledgers are deletable, and update the results to the
> > metadatastore. We can perform the deletion of marked ledgers
> asynchronously
> > in the callback of updating the meta information, so that the original
> > logic can be retained seamlessly. Therefore, when we are rolling upgrade
> or
> > rollback, the only difference is whether the deleted ledger is marked for
> > deletion.
> >
> > To be more specific:
> > 1. for upgrade, only the marker information of ledger has been added, and
> > the logical sequence of deletion has not changed.
> > 2. for rollback, some ledgers that have been marked for deletion may not
> be
> > deleted due to the restart of the broker. This behavior is consistent
> with
> > the original version.
> >
> > In addition, if the ledger that has been marked is not deleted
> > successfully, the marker will not be removed. So for this part of
> ledgers,
> > every time trimming is triggered, it will be deleted again, which is
> > equivalent to a check and retry mechanism.
> >
> > ## Goal
> >
> > We need to modify some logic in
> >
> `org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl#internalTrimLedgers`
> > so that the ledger deletion logic in ledger-trimming is split into two
> > stages, marking and deleting. Once the marker information is updated to
> the
> > metadatastore, every trimming will try to trigger the ledger deletion
> until
> > all the deleteable ledgers are successfully deleted.
> >
> > ## Implementation
> >
> > This proposal aims to separate the deletion logic in ledger-trimming, so
> > that `ManagedLedgerImpl#internalTrimLedgers` is responsible for marking
> the
> > deletable ledgers and then perform actual ledger deletion according to
> the
> > metadatastore.
> >
> > Therefore, the entire trimming process is broken down into the following
> > steps:
> >
> > 1. mark deletable ledgers and update ledger metadata.
> > 2. do acutual ledger deletion after metadata is updated.
> >
> > For step 1, we can store the marker of deletable information in
> > `org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl#propertiesMap`.
> When
> > retrieving the deleted ledger information, we can directly query by
> > iterating `propertiesMap`. If this solution is not accepted, maybe we can
> > create a new znode to store these information, but this approach will not
> > be able to reuse the current design.
> >
> > For step 2, we can perform the deletion of marked ledgers asynchronously
> in
> > the callback of updating the meta information. And every trimming will
> > trigger the check and delete for those deleteable ledgers.
> >
> > Related PR: https://github.com/apache/pulsar/pull/13575
>

Re: [VOTE] PIP-122: Change loadBalancer default loadSheddingStrategy to ThresholdShedder

[Hang Chen]

Thanks for your participation.

Close this vote with 3 (+1) bindings and 5 (+1) non-bindings, 0 (-1).

Regards,
Hang

PengHui Li  于2022年1月13日周四 11:03写道：
>
> +1
>
> Penghui
>
> On Thu, Jan 13, 2022 at 7:54 AM Aloys Zhang  wrote:
>
> > +1
> >
> > Michael Marshall  于2022年1月12日周三 13:37写道：
> >
> > > +1 - assuming we ensure that the `ThresholdShedder` has unit test
> > coverage.
> > >
> > > Thanks,
> > > Michael
> > >
> > >
> > > On Tue, Jan 11, 2022 at 9:53 PM r...@apache.org  > >
> > > wrote:
> > > >
> > > > +1 (non-binding)
> > > >
> > > > --
> > > >
> > > > Thanks
> > > > Xiaolong Ran
> > > >
> > > > Haiting Jiang  于2022年1月12日周三 09:52写道：
> > > >
> > > > > +1
> > > > >
> > > > > On 2022/01/10 06:47:44 Hang Chen wrote:
> > > > > > This is the voting thread for PIP-122. It will stay open for at
> > > least 48
> > > > > > hours.
> > > > > >
> > > > > > https://github.com/apache/pulsar/issues/13340
> > > > > >
> > > > > > Pasted below for quoting convenience.
> > > > > >
> > > > > > 
> > > > > >
> > > > > > ### Motivation
> > > > > > The ThresholdShedder load balance policy since Pulsar 2.6.0 by
> > > > > > https://github.com/apache/pulsar/pull/6772. It can resolve many
> > load
> > > > > > balance issues of `OverloadShedder` and works well in many Pulsar
> > > > > > production clusters.
> > > > > >
> > > > > > In Pulsar 2.6.0, 2.7.0, 2.8.0 and 2.9.0, Pulsar's default load
> > > balance
> > > > > > policy is `OverloadShedder`.
> > > > > >
> > > > > > I think it's a good time for 2.10 to change default load balance
> > > > > > policy to `ThresholdShedder`, it will make throughput more balance
> > > > > > between brokers.
> > > > > >
> > > > > > ### Proposed Changes
> > > > > > In 2.10 release，for `broker.conf`, change
> > > > > > `loadBalancerLoadSheddingStrategy` from
> > > > > > `org.apache.pulsar.broker.loadbalance.impl.OverloadShedder` to
> > > > > > `org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder`
> > > > > >
> > > > >
> > >
> >

Re: [VOTE] PIP-129: Introduce intermediate state for ledger deletion

[Hang Chen]

+1 (binding)

Best,
Hang

Zhanpeng Wu  于2022年1月14日周五 10:37写道：
>
> This is the voting thread for PIP-129. It will stay open for at least 48
> hours.  Pasted below for quoting convenience.
>
> 
>
> https://github.com/apache/pulsar/issues/13526
>
> 
>
> ## Motivation
>
> Under the current ledger-trimming design in
> `org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl#internalTrimLedgers`,
> we need to collect those ledgers that need to be deleted first, and then
> perform the asynchronous deletion of the ledger concurrently, but we do not
> continue to pay attention to whether the deletion operation is completed.
> If the meta-information update has been successfully completed but an error
> occurs during the asynchronous deletion, the ledger may not be deleted, but
> at the logical level we think that the deletion has been completed, which
> will make this part of the data remain in the storage layer forever (such
> as bk). As the usage time of the cluster becomes longer, the residual data
> that cannot be deleted will gradually increase.
>
> In order to achieve this goal, we can separate the logic of
> meta-information update and ledger deletion. In the trimming process, we
> can first mark which ledgers are deletable, and update the results to the
> metadatastore. We can perform the deletion of marked ledgers asynchronously
> in the callback of updating the meta information, so that the original
> logic can be retained seamlessly. Therefore, when we are rolling upgrade or
> rollback, the only difference is whether the deleted ledger is marked for
> deletion.
>
> To be more specific:
> 1. for upgrade, only the marker information of ledger has been added, and
> the logical sequence of deletion has not changed.
> 2. for rollback, some ledgers that have been marked for deletion may not be
> deleted due to the restart of the broker. This behavior is consistent with
> the original version.
>
> In addition, if the ledger that has been marked is not deleted
> successfully, the marker will not be removed. So for this part of ledgers,
> every time trimming is triggered, it will be deleted again, which is
> equivalent to a check and retry mechanism.
>
> ## Goal
>
> We need to modify some logic in
> `org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl#internalTrimLedgers`
> so that the ledger deletion logic in ledger-trimming is split into two
> stages, marking and deleting. Once the marker information is updated to the
> metadatastore, every trimming will try to trigger the ledger deletion until
> all the deleteable ledgers are successfully deleted.
>
> ## Implementation
>
> This proposal aims to separate the deletion logic in ledger-trimming, so
> that `ManagedLedgerImpl#internalTrimLedgers` is responsible for marking the
> deletable ledgers and then perform actual ledger deletion according to the
> metadatastore.
>
> Therefore, the entire trimming process is broken down into the following
> steps:
>
> 1. mark deletable ledgers and update ledger metadata.
> 2. do acutual ledger deletion after metadata is updated.
>
> For step 1, we can store the marker of deletable information in
> `org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl#propertiesMap`. When
> retrieving the deleted ledger information, we can directly query by
> iterating `propertiesMap`. If this solution is not accepted, maybe we can
> create a new znode to store these information, but this approach will not
> be able to reuse the current design.
>
> For step 2, we can perform the deletion of marked ledgers asynchronously in
> the callback of updating the meta information. And every trimming will
> trigger the check and delete for those deleteable ledgers.
>
> Related PR: https://github.com/apache/pulsar/pull/13575

RE: [DISCUSSION] PIP-129: Introduce intermediate state for ledger deletion

[Zhanpeng Wu]

Looks there is no objection, I will start the official vote for PIP-129

On 2022/01/12 01:58:27 Zhanpeng Wu wrote:
> https://github.com/apache/pulsar/issues/13526
>
> 
>
> ## Motivation
>
> Under the current ledger-trimming design in
>
`org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl#internalTrimLedgers`,
> we need to collect those ledgers that need to be deleted first, and then
> perform the asynchronous deletion of the ledger concurrently, but we do
not
> continue to pay attention to whether the deletion operation is completed.
> If the meta-information update has been successfully completed but an
error
> occurs during the asynchronous deletion, the ledger may not be deleted,
but
> at the logical level we think that the deletion has been completed, which
> will make this part of the data remain in the storage layer forever (such
> as bk). As the usage time of the cluster becomes longer, the residual data
> that cannot be deleted will gradually increase.
>
> In order to achieve this goal, we can separate the logic of
> meta-information update and ledger deletion. In the trimming process, we
> can first mark which ledgers are deletable, and update the results to the
> metadatastore. We can perform the deletion of marked ledgers
asynchronously
> in the callback of updating the meta information, so that the original
> logic can be retained seamlessly. Therefore, when we are rolling upgrade
or
> rollback, the only difference is whether the deleted ledger is marked for
> deletion.
>
> To be more specific:
> 1. for upgrade, only the marker information of ledger has been added, and
> the logical sequence of deletion has not changed.
> 2. for rollback, some ledgers that have been marked for deletion may not
be
> deleted due to the restart of the broker. This behavior is consistent with
> the original version.
>
> In addition, if the ledger that has been marked is not deleted
> successfully, the marker will not be removed. So for this part of ledgers,
> every time trimming is triggered, it will be deleted again, which is
> equivalent to a check and retry mechanism.
>
> ## Goal
>
> We need to modify some logic in
> `org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl#internalTrimLedgers`
> so that the ledger deletion logic in ledger-trimming is split into two
> stages, marking and deleting. Once the marker information is updated to
the
> metadatastore, every trimming will try to trigger the ledger deletion
until
> all the deleteable ledgers are successfully deleted.
>
> ## Implementation
>
> This proposal aims to separate the deletion logic in ledger-trimming, so
> that `ManagedLedgerImpl#internalTrimLedgers` is responsible for marking
the
> deletable ledgers and then perform actual ledger deletion according to the
> metadatastore.
>
> Therefore, the entire trimming process is broken down into the following
> steps:
>
> 1. mark deletable ledgers and update ledger metadata.
> 2. do acutual ledger deletion after metadata is updated.
>
> For step 1, we can store the marker of deletable information in
> `org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl#propertiesMap`. When
> retrieving the deleted ledger information, we can directly query by
> iterating `propertiesMap`. If this solution is not accepted, maybe we can
> create a new znode to store these information, but this approach will not
> be able to reuse the current design.
>
> For step 2, we can perform the deletion of marked ledgers asynchronously
in
> the callback of updating the meta information. And every trimming will
> trigger the check and delete for those deleteable ledgers.
>
> Related PR: https://github.com/apache/pulsar/pull/13575
>

Re: [Discuss] Create new issues to SDKs in different languages

[r...@apache.org]

Good ideas, we need such a mechanism to complement the current process to
ensure that other language SDKs know what's going on with the Java SDK,
which would be a good start for other language SDKs. We can clearly see
where the gap between the current SDK and the Java SDK is, which new
functions we need to support, and which bugs we need to modify.

We can perform this process manually first, although it is troublesome, but
in this action, we can find out what we need to do to automate such a
process, I believe this will be a good start.

--
Thanks
Xiaolong Ran

PengHui Li  于2022年1月13日周四 11:16写道：

> I'm not sure if the bots can detect if the change is a Java client change,
> maybe based on the changes introduced in which directory.
>
> The main headache here is missing it. If there are some mechanisms that can
> remind us.
> It will be great. Looks like
>
> "hey, new changes introduced in Java client, you might need to create an
> issue to other clients repos"
>
> Use a label to allow the bots to sync to other repos LGTM here, we can run
> it manually first.
> So that we can know more clearly if we need automatic synchronization.
>
> Thanks,
> Penghui
>
> On Wed, Jan 12, 2022 at 9:06 PM Zike Yang 
> wrote:
>
> > +1
> >
> > > I wonder if we can create issue in client repo automatically with bots
> > for PRs labelled"component/client" in pulsar repo.
> > > This would save the extra effort for the reviewer.
> >
> > But there are many PRs with "component/client" label that are specific
> > to java client changes. I think these should not be added to other
> > clients' repos.
> >
> >
> >
> > On Wed, Jan 12, 2022 at 4:18 PM Haiting Jiang 
> > wrote:
> > >
> > > +1. Great idea.
> > >
> > > I wonder if we can create issue in client repo automatically with bots
> > for PRs labelled"component/client" in pulsar repo.
> > > This would save the extra effort for the reviewer.
> > >
> > > Thanks,
> > > Haiting Jiang
> > >
> > > On 2022/01/12 03:45:18 "r...@apache.org" wrote:
> > > > Hello everyone:
> > > >
> > > > At present, all our PIP and related function changes are mainly in
> the
> > Java
> > > > language, and all new functions will be merged into the Java SDK
> > first, but
> > > > for SDKs in other languages, this is completely a black box, they
> don't
> > > > know what changes or optimizations have been made on the Java SDK
> side.
> > > >
> > > > The most typical problem is that when users of other languages
> > encounter
> > > > various problems during use, when the maintainers of other languages
> > want
> > > > to fix these problems, we do not know that the Java SDK side has made
> > these
> > > > changes. Therefore, every current solution is to constantly check
> > where the
> > > > gap of the current Java SDK is, which brings great challenges to the
> > > > maintainers themselves.
> > > >
> > > > So here is an idea, when the committters/PMC responsible for
> reviewing
> > the
> > > > Java SDK can do more to help evaluate whether these PIPs or new
> changes
> > > > need to support this function in other languages, and then the
> > > > corresponding issue is created in the corresponding SDK, so that it
> is
> > > > convenient for the maintainers of other language SDKs to further
> > evaluate
> > > > the priority of this function, and it can also attract more
> > contributors
> > > > who are good at certain languages to claim the corresponding issue
> and
> > > > contribute the corresponding function.
> > > >
> > > > --
> > > > Thanks
> > > > Xiaolong Ran
> > > >
> >
> >
> >
> > --
> > Zike Yang
> >
>

[VOTE] PIP-129: Introduce intermediate state for ledger deletion

[Zhanpeng Wu]

This is the voting thread for PIP-129. It will stay open for at least 48
hours.  Pasted below for quoting convenience.



https://github.com/apache/pulsar/issues/13526



## Motivation

Under the current ledger-trimming design in
`org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl#internalTrimLedgers`,
we need to collect those ledgers that need to be deleted first, and then
perform the asynchronous deletion of the ledger concurrently, but we do not
continue to pay attention to whether the deletion operation is completed.
If the meta-information update has been successfully completed but an error
occurs during the asynchronous deletion, the ledger may not be deleted, but
at the logical level we think that the deletion has been completed, which
will make this part of the data remain in the storage layer forever (such
as bk). As the usage time of the cluster becomes longer, the residual data
that cannot be deleted will gradually increase.

In order to achieve this goal, we can separate the logic of
meta-information update and ledger deletion. In the trimming process, we
can first mark which ledgers are deletable, and update the results to the
metadatastore. We can perform the deletion of marked ledgers asynchronously
in the callback of updating the meta information, so that the original
logic can be retained seamlessly. Therefore, when we are rolling upgrade or
rollback, the only difference is whether the deleted ledger is marked for
deletion.

To be more specific:
1. for upgrade, only the marker information of ledger has been added, and
the logical sequence of deletion has not changed.
2. for rollback, some ledgers that have been marked for deletion may not be
deleted due to the restart of the broker. This behavior is consistent with
the original version.

In addition, if the ledger that has been marked is not deleted
successfully, the marker will not be removed. So for this part of ledgers,
every time trimming is triggered, it will be deleted again, which is
equivalent to a check and retry mechanism.

## Goal

We need to modify some logic in
`org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl#internalTrimLedgers`
so that the ledger deletion logic in ledger-trimming is split into two
stages, marking and deleting. Once the marker information is updated to the
metadatastore, every trimming will try to trigger the ledger deletion until
all the deleteable ledgers are successfully deleted.

## Implementation

This proposal aims to separate the deletion logic in ledger-trimming, so
that `ManagedLedgerImpl#internalTrimLedgers` is responsible for marking the
deletable ledgers and then perform actual ledger deletion according to the
metadatastore.

Therefore, the entire trimming process is broken down into the following
steps:

1. mark deletable ledgers and update ledger metadata.
2. do acutual ledger deletion after metadata is updated.

For step 1, we can store the marker of deletable information in
`org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl#propertiesMap`. When
retrieving the deleted ledger information, we can directly query by
iterating `propertiesMap`. If this solution is not accepted, maybe we can
create a new znode to store these information, but this approach will not
be able to reuse the current design.

For step 2, we can perform the deletion of marked ledgers asynchronously in
the callback of updating the meta information. And every trimming will
trigger the check and delete for those deleteable ledgers.

Related PR: https://github.com/apache/pulsar/pull/13575

[GitHub] [pulsar-adapters] nlu90 commented on a change in pull request #31: [Issue #29] [pulsar-spark] Adding SparkPulsarReliableReceiver

[GitBox]

nlu90 commented on a change in pull request #31:
URL: https://github.com/apache/pulsar-adapters/pull/31#discussion_r784417599



##
File path: 
pulsar-spark/src/main/scala/org/apache/spark/streaming/pulsar/SparkStreamingReliablePulsarReceiver.scala
##
@@ -0,0 +1,181 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.spark.streaming.pulsar
+
+import com.google.common.util.concurrent.RateLimiter
+import org.apache.pulsar.client.api._
+import org.apache.pulsar.client.impl.PulsarClientImpl
+import org.apache.spark.SparkConf
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming.receiver.Receiver
+import org.slf4j.LoggerFactory
+
+import scala.collection.JavaConversions._
+import scala.collection.JavaConverters._
+import scala.collection.mutable.ListBuffer
+
+/**
+  * Custom spark receiver to pull messages from Pubsub topic and push into 
reliable store.
+  * If backpressure is enabled,the message ingestion rate for this receiver 
will be managed by Spark.
+  *
+  * Following spark configurations can be used to control rates and block size
+  * spark.streaming.backpressure.enabled
+  * spark.streaming.backpressure.initialRate
+  * spark.streaming.receiver.maxRate
+  * spark.streaming.blockQueueSize: Controlling block size
+  * spark.streaming.backpressure.pid.minRate
+  *
+  * See Spark streaming configurations doc
+  * https://spark.apache.org/docs/latest/configuration.html#spark-streaming
+  *
+  * @param storageLevel  Storage level to be used
+  * @param serviceUrlService URL to the Pubsub Cluster
+  * @param pulsarConfMap of PulsarConf containing keys from 
`org.apache.pulsar.client.impl.conf.ConsumerConfigurationData`
+  * @param authenticationAuthentication object for authenticating 
consumer to pulsar. Use `AuthenticationDisabled` is authentication is disabled.
+  * @param sparkConf Spark configs
+  * @param maxPollSize   Max number of records to read by consumer 
in one batch
+  * @param consumerName  Consumer name to be used by receiver
+  * @param autoAcknowledge   Acknowledge pubsub message or not
+  * @param rateMultiplierFactor  Rate multiplier factor in case you want 
to have more rate than what PIDRateEstimator suggests. > 1.0  is useful in case
+  *  of spark dynamic allocation to utilise 
extra resources. Keep it 1.0 if spark dynamic allocation is disabled.
+  */
+
+class SparkStreamingReliablePulsarReceiver(override val storageLevel: 
StorageLevel,
+   val serviceUrl: String,
+   val pulsarConf: Map[String, Any],
+   val authentication: Authentication,
+   val sparkConf: SparkConf,
+   val maxPollSize: Int,
+   val consumerName: String,
+   val autoAcknowledge: Boolean = true,
+   val rateMultiplierFactor: Double  = 
1.0) extends Receiver[SparkPulsarMessage](storageLevel) {
+
+  private val maxRateLimit: Long = 
sparkConf.getLong("spark.streaming.receiver.maxRate", Long.MaxValue)
+  private val blockSize: Int = 
sparkConf.getInt("spark.streaming.blockQueueSize", maxPollSize)
+  private val blockIntervalMs: Long = 
sparkConf.getTimeAsMs("spark.streaming.blockInterval", "200ms")
+
+  lazy val rateLimiter: RateLimiter = RateLimiter.create(math.min(
+sparkConf.getLong("spark.streaming.backpressure.initialRate", 
maxRateLimit),
+maxRateLimit
+  ).toDouble)
+
+  val buffer: ListBuffer[SparkPulsarMessage] = new 
ListBuffer[SparkPulsarMessage]()
+
+  private var pulsarClient: PulsarClient = null
+  private var consumer: Consumer[Array[Byte]] = null
+  private var consumerThread: Thread = null
+
+  var latestStorePushTime: Long = System.currentTimeMillis()
+
+  override def onStart(): Unit = {
+try {
+  pulsarClient = 
PulsarClient.builder.serviceUrl(serviceUrl).authentication(authentication).build
+  

Re: [VOTE] PIP-132: Include message header size when check maxMessageSize of non-batch message on the client side.

[Neng Lu]

+1 (non-binding)

On Wed, Jan 12, 2022 at 7:07 PM PengHui Li  wrote:

> +1 (binding)
>
> This is a behavior change, which we should highlight in the release note.
>
> Penghui
>
> On Thu, Jan 13, 2022 at 12:44 AM Chris Herzog 
> wrote:
>
> > +1 (non)
> >
> >
> > On Tue, Jan 11, 2022 at 9:46 PM r...@apache.org  >
> > wrote:
> >
> > > +1 (non-binding)
> > >
> > > --
> > > Thanks
> > > Xiaolong Ran
> > >
> > > mattison chao  于2022年1月12日周三 10:15写道：
> > >
> > > > +1  (non-binding)
> > > >
> > > > On Wed, 12 Jan 2022 at 09:59, Zike Yang  > .invalid>
> > > > wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > On Wed, Jan 12, 2022 at 9:58 AM Haiting Jiang <
> > jianghait...@apache.org
> > > >
> > > > > wrote:
> > > > > >
> > > > > > This is the voting thread for PIP-132. It will stay open for at
> > least
> > > > 48
> > > > > hours.
> > > > > >
> > > > > > https://github.com/apache/pulsar/issues/13591
> > > > > >
> > > > > > Pasted below for quoting convenience.
> > > > > >
> > > > > > 
> > > > > >
> > > > > > ## Motivation
> > > > > >
> > > > > > Currently, Pulsar client (Java) only checks payload size for max
> > > > message
> > > > > size validation.
> > > > > >
> > > > > > Client throws TimeoutException if we produce a message with too
> > many
> > > > > properties, see [1].
> > > > > > But the root cause is that is trigged TooLongFrameException in
> > broker
> > > > > server.
> > > > > >
> > > > > > In this PIP, I propose to include message header size when check
> > > > > maxMessageSize of non-batch
> > > > > > messages, this brings the following benefits.
> > > > > > 1. Clients can throw InvalidMessageException immediately if
> > > properties
> > > > > takes too much storage space.
> > > > > > 2. This will make the behaviour consistent with topic level max
> > > message
> > > > > size check in broker.
> > > > > > 3. Strictly limit the entry size less than maxMessageSize, avoid
> > > > sending
> > > > > message to bookkeeper failed.
> > > > > >
> > > > > > ## Goal
> > > > > >
> > > > > > Include message header size when check maxMessageSize for
> non-batch
> > > > > message on the client side.
> > > > > >
> > > > > > ## Implementation
> > > > > >
> > > > > > ```
> > > > > > // Add a size check in
> > > > > org.apache.pulsar.client.impl.ProducerImpl#processOpSendMsg
> > > > > > if (op.msg != null // for non-batch messages only
> > > > > > && op.getMessageHeaderAndPayloadSize() >
> > > > ClientCnx.getMaxMessageSize()) {
> > > > > > // finish send op with InvalidMessageException
> > > > > > releaseSemaphoreForSendOp(op);
> > > > > > op.sendComplete(new PulsarClientException(new
> > > InvalidMessageException,
> > > > > op.sequenceId));
> > > > > > }
> > > > > >
> > > > > >
> > > > > > //
> > > > >
> > > >
> > >
> >
> org.apache.pulsar.client.impl.ProducerImpl.OpSendMsg#getMessageHeaderAndPayloadSize
> > > > > >
> > > > > > public int getMessageHeaderAndPayloadSize() {
> > > > > > ByteBuf cmdHeader = cmd.getFirst();
> > > > > > cmdHeader.markReaderIndex();
> > > > > > int totalSize = cmdHeader.readInt();
> > > > > > int cmdSize = cmdHeader.readInt();
> > > > > > int msgHeadersAndPayloadSize = totalSize - cmdSize - 4;
> > > > > > cmdHeader.resetReaderIndex();
> > > > > > return msgHeadersAndPayloadSize;
> > > > > > }
> > > > > > ```
> > > > > >
> > > > > > ## Reject Alternatives
> > > > > > Add a new property like "maxPropertiesSize" or "maxHeaderSize" in
> > > > > broker.conf and pass it to
> > > > > > client like maxMessageSize. But the implementation is much more
> > > > complex,
> > > > > and don't have the
> > > > > > benefit 2 and 3 mentioned in Motivation.
> > > > > >
> > > > > > ## Compatibility Issue
> > > > > > As a matter of fact, this PIP narrows down the sendable range.
> > > > > Previously, when maxMessageSize
> > > > > > is 1KB, it's ok to send message with 1KB properties and 1KB
> > payload.
> > > > But
> > > > > with this PIP, the
> > > > > > sending will fail with InvalidMessageException.
> > > > > >
> > > > > > One conservative way is to add a boolean config
> > > > > "includeHeaderInSizeCheck" to enable this
> > > > > > feature. But I think it's OK to enable this directly as it's more
> > > > > reasonable, and I don't see good
> > > > > > migration plan if we add a config for this.
> > > > > >
> > > > > > [1] https://github.com/apache/pulsar/issues/13560
> > > > > >
> > > > > > Thanks,
> > > > > > Haiting Jiang
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Zike Yang
> > > > >
> > > >
> > >
> >
> >
> > --
> >
> >
> > Chris Herzog Messaging Team | O 630 300 7718 | M 815 263 3764 |
> > www.tibco.com
> >
> > 
> >
>


-- 
Best Regards,
Neng

[GitHub] [pulsar-client-node] csauvage opened a new issue #189: Fail to build `pulsar-client` on M1

[GitBox]

csauvage opened a new issue #189:
URL: https://github.com/apache/pulsar-client-node/issues/189


   Hi there. 
   
   I'm facing the same issues as #46 and #87 trying to install (and build) 
`pulsar-client` on M1 Pro. 
   
   - [x] OS : MacOS 12.0.1 (arch. M1 Pro - 2021)
   - [x] Node :  v16.13.2
   - [x] NPM : 8.1.2
   - [x] Node-gyp : node-gyp@7.1.0
   - [x] Python Version : 2.7.18
   - [x] libpulsar: 2.9.1_1
   
   
   ## Error 
   
   ```bash
   ...
   gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
 CC(target) 
Release/obj.target/nothing/node_modules/node-addon-api/src/nothing.o
 LIBTOOL-STATIC Release/nothing.a
   warning: 
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool:
 archive library: Release/nothing.a the table of contents is empty (no object 
file members in the library define global symbols)
 CXX(target) Release/obj.target/Pulsar/src/addon.o
   In file included from ../src/addon.cc:20:
   ../src/Message.h:24:10: fatal error: 'pulsar/c/message.h' file not found
   #include 
^~~~
   1 error generated.
   make: *** [Release/obj.target/Pulsar/src/addon.o] Error 1
   gyp ERR! build error
   gyp ERR! stack Error: `make` failed with exit code: 2
   gyp ERR! stack at ChildProcess.onExit 
(/Users/clementsauvage/Github/***/backend/node_modules/node-gyp/lib/build.js:194:23)
   gyp ERR! stack at ChildProcess.emit (node:events:390:28)
   gyp ERR! stack at Process.ChildProcess._handle.onexit 
(node:internal/child_process:290:12)
   gyp ERR! System Darwin 21.1.0
   ...
   ```
   
   ## What did I try 
   
   - [x] Building libpulsar myself : Python Boost not found and impossible to 
install it from homebrew
   - [x] Trying other version : Tried 1.2 / 1.3 & 1.4 same issues
   - [x] Trying to change npm / node / python version 
   - [x] Switching between NPM & Yarn
   
   
   
   ## Full trace
   
   ```
   backend git:(master) ✗ yarn add pulsar-client@1.2.0
   yarn add v1.22.17
   [1/4]   Resolving packages...
   warning pulsar-client > node-pre-gyp@0.12.0: Please upgrade to 
@mapbox/node-pre-gyp: the non-scoped node-pre-gyp package is deprecated and 
only the @mapbox scoped package will recieve updates in the future
   warning pulsar-client > node-gyp > tar@2.2.2: This version of tar is no 
longer supported, and will not receive security updates. Please upgrade asap.
   warning pulsar-client > node-gyp > request@2.88.2: request has been 
deprecated, see https://github.com/request/request/issues/3142
   [2/4]   Fetching packages...
   [3/4]   Linking dependencies...
   warning " > @sentry/react@6.16.1" has unmet peer dependency "react@15.x || 
16.x || 17.x".
   warning "swagger-jsdoc > swagger-parser > 
@apidevtools/swagger-parser@10.0.2" has unmet peer dependency 
"openapi-types@>=7".
   [4/4]   Building fresh packages...
   [1/6] ⠂ bcrypt
   [2/6] ⠂ iltorb
   [3/6] ⠂ node-zopfli-es
   [4/6] ⠂ puppeteer
   error /Users/clementsauvage/Github/***/backend/node_modules/pulsar-client: 
Command failed.
   Exit code: 1
   Command: node-pre-gyp install --fallback-to-build
   Arguments:
   Directory: 
/Users/clementsauvage/Github/***/backend/node_modules/pulsar-client
   Output:
   node-pre-gyp info it worked if it ends with ok
   node-pre-gyp info using node-pre-gyp@0.12.0
   node-pre-gyp info using node@16.13.2 | darwin | arm64
   node-pre-gyp WARN Using request for node-pre-gyp https download
   node-pre-gyp info check checked for 
"/Users/clementsauvage/Github/***/backend/node_modules/pulsar-client/build/Release/libpulsar.node"
 (not found)
   node-pre-gyp http GET 
https://pulsar.apache.org/docs/en/client-libraries-cpp/libpulsar-v1.2.0-node-v93-darwin-arm64.tar.gz
   node-pre-gyp http 404 
https://pulsar.apache.org/docs/en/client-libraries-cpp/libpulsar-v1.2.0-node-v93-darwin-arm64.tar.gz
   node-pre-gyp WARN Tried to download(404): 
https://pulsar.apache.org/docs/en/client-libraries-cpp/libpulsar-v1.2.0-node-v93-darwin-arm64.tar.gz
   node-pre-gyp WARN Pre-built binaries not found for pulsar-client@1.2.0 and 
node@16.13.2 (node-v93 ABI, unknown) (falling back to source compile with 
node-gyp)
   node-pre-gyp http 404 status code downloading tarball 
https://pulsar.apache.org/docs/en/client-libraries-cpp/libpulsar-v1.2.0-node-v93-darwin-arm64.tar.gz
   gyp info it worked if it ends with ok
   gyp info using node-gyp@7.1.0
   gyp info using node@16.13.2 | darwin | arm64
   gyp info ok
   gyp info it worked if it ends with ok
   gyp info using node-gyp@7.1.0
   gyp info using node@16.13.2 | darwin | arm64
   gyp info find Python using Python version 2.7.18 found at 
"/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python"
   (node:15144) [DEP0150] DeprecationWarning: Setting process.config is 
deprecated. In the future the property will be read-only.
   (Use `node --trace-deprecation ...` to show where the warning was created)
   gyp info spawn 

Re: [DISCUSS] [Transaction] Clear namespace backlog can't clear system topic and system sub backlog.

[PengHui Li]

I agree with the point, we should avoid the `clearNamespaceBacklog(String
namespace)` to apply to the internal topic or internal cursor.
It will introduce abnormal behaviors, e.g. clear a replication cursor
backlog, clear a dedup cursor backlog, clear a compaction cursor backlog.

I would suggest letting the `clearNamespaceBacklog(String namespace)`
method does not apply to the Pulsar internal topics and cursors.

Thanks,
Penghui

On Thu, Jan 13, 2022 at 4:32 PM 丛搏  wrote:

> Hi :
>
> issue link : https://github.com/streamnative/support-tickets/issues/408 <
> https://github.com/streamnative/support-tickets/issues/408>
>
> # Background
> Now we have many clear namespace backlog interfaces etc.
> ```
> clearNamespaceBacklog(String namespace)
>
> clearNamespaceBacklogAsync(String namespace)
>
> clearNamespaceBacklogForSubscription(String namespace, String subscription)
>
> clearNamespaceBacklogForSubscriptionAsync(String namespace, String
> subscription)
>
> clearNamespaceBundleBacklog(String namespace, String bundle)
>
> clearNamespaceBundleBacklogAsync(String namespace, String bundle)
>
> clearNamespaceBundleBacklogForSubscription(String namespace, String
> bundle, String subscription)
>
> clearNamespaceBundleBacklogForSubscriptionAsync(String namespace, String
> bundle, String subscription)
> ```
> There are two types of interfaces:
> 1. clear namespace backlog with subscriptionName
> 2. clear namespace backlog no subscriptionName
>
> But there have a problem, users do not know that there are many system
> topics in the namespace.
>
> ```
>
> /**
>  * Local topic name for the namespace events.
>  */
> public static final String NAMESPACE_EVENTS_LOCAL_NAME =
> "__change_events";
>
> /**
>  * Local topic name for the transaction buffer snapshot.
>  */
> public static final String TRANSACTION_BUFFER_SNAPSHOT =
> "__transaction_buffer_snapshot";
>
> public static final String SUBSCRIPTION_NAME =
> "transaction-buffer-sub";
> ```
> User only want to clear the backlog witch they own created and sub. But
> now we have clear the system topic sub backlog when user don't specify
> subName and clear system sub with a topic witch user created. It will bring
> many problems. etc.
> 1. clear `NAMESPACE_EVENTS_LOCAL_NAME` will clear any topic policy in this
> namespace, but user don't know and they don't want to be cleared. If the
> user does not understand the problems caused by this interface, it will
> delete many topic policy configurations and is difficult to recover, which
> will lead to unpredictable and disastrous problems.
> 2. clear `TRANSACTION_BUFFER_SNAPSHOT` topic backlog will cause
> transaction buffer recover incompletely.
> 3. clear `transaction-buffer-sub` sub also will cause transaction buffer
> recover incompletely.
> 4. if add new system topic and system sub, we should think about this
> problem.
>
> # Motivation
> We want to solve the above problems, but we can't make users unable to
> delete the backlog of system topic and system sub.
> So, we should solve types of interface
> There are two types of interfaces:
> 1. clear namespace backlog with subscriptionName
> 2. clear namespace backlog no subscriptionName
>
> when clear namespace backlog no subscriptionName:
> It represents user know the sub name is system subName and really want to
> delete it, We should not block the user clear system sub backlog and do not
> allow the user to create system sub. If the user uses it, it means that the
> user has considered the result of the clear system sub backlog
>
> when clear namespace backlog with subscriptionName:
> Because the user may not consider the existence of system topic and system
> sub, and the wrong clear backlog may lead to serious results. Therefore,
> this interface cannot delete the backlog of system topic and the backlog of
> system sub. If the user really wants to clear backlog, please bring
> subscriptionName and use another type of interface with subscriptionName.
>
> **Clear namespace and clear namespace bundle all need to handle is
> logical.**
>
> # implement
>
> ```
> class SystemTopicName {
> /**
>  * Local topic name for the namespace events.
>  */
> public static final String NAMESPACE_EVENTS_LOCAL_NAME =
> "__change_events";
>
> /**
>  * Local topic name for the transaction buffer snapshot.
>  */
> public static final String TRANSACTION_BUFFER_SNAPSHOT =
> "__transaction_buffer_snapshot";
> }
> ```
> ```
> class SystemSubName {
>
> public static final String SUBSCRIPTION_NAME =
> "transaction-buffer-sub";
>
> }
> ```
>
> 1. In order to make the code easier to maintain and reduce redundancy, we
> need to move systemTopic and systemSub to a specific classes to manage.
> 2. If the newly added systemTopic and systemSub are created from this
> class, unpredictable problems can be prevented
>
>
>
> Let me know if you have different ideas or suggestions！！
>
> Thanks!
> Bo Cong

[GitHub] [pulsar-manager] tuteng merged pull request #435: Fix for Sidebar items not visible to admin and superadmin

[GitBox]

tuteng merged pull request #435:
URL: https://github.com/apache/pulsar-manager/pull/435


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-manager] tuteng merged pull request #436: Allow user to assign tenant as resource to role

[GitBox]

tuteng merged pull request #436:
URL: https://github.com/apache/pulsar-manager/pull/436


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-manager] dragonls opened a new pull request #440: Add filterable to el-select enables filtering for tenant/namespace/topic

[GitBox]

dragonls opened a new pull request #440:
URL: https://github.com/apache/pulsar-manager/pull/440


   Master Issue: #439 
   
   ### Motivation
   
   Add filterable to el-select enables filtering for tenant/namespace/topic, 
which can be quite useful especially when there are many options.
   
   ### Modifications
   
   Add filterable to el-select enables filtering.
   
   ### Verifying this change
   
   - [ ] Make sure that the change passes the `./gradlew build` checks.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-manager] dragonls opened a new issue #439: Support text filtering in Select component for tenant/namespace/topic

[GitBox]

dragonls opened a new issue #439:
URL: https://github.com/apache/pulsar-manager/issues/439


   See https://element.eleme.cn/#/en-US/component/select 
   
   It can be much useful to enable filtering.
   
   
![image](https://user-images.githubusercontent.com/2565118/149328354-738d31fb-40b9-4855-a3a4-84d8ba72e750.png)
   
![image](https://user-images.githubusercontent.com/2565118/149328396-0c3ddb6a-d788-4203-a9e0-2be3ceba9425.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-adapters] sijie commented on pull request #31: [Issue #29] [pulsar-spark] Adding SparkPulsarReliableReceiver

[GitBox]

sijie commented on pull request #31:
URL: https://github.com/apache/pulsar-adapters/pull/31#issuecomment-1011938940


   @nlu90 can you take a look at this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSSION] PIP-135: Include MetadataStore backend for Etcd

[Aloys Zhang]

+1

Hang Chen  于2022年1月13日周四 14:39写道：

> +1
>
> Best,
> Hang
>
> Enrico Olivelli  于2022年1月13日周四 14:20写道：
> >
> > +1
> >
> > Enrico
> >
> > Il Gio 13 Gen 2022, 06:01 Joe F  ha scritto:
> >
> > > +1
> > >
> > > On Wed, Jan 12, 2022 at 3:52 PM Aloys Zhang 
> wrote:
> > >
> > > > +1
> > > >
> > > > 陳智弘  于2022年1月12日周三 10:19写道：
> > > >
> > > > > +1
> > > > >
> > > > > Haiting Jiang  於 2022年1月12日 週三 09:50 寫道：
> > > > >
> > > > > > +1
> > > > > >
> > > > > > On 2022/01/12 01:44:21 PengHui Li wrote:
> > > > > > > +1
> > > > > > >
> > > > > > > Penghui
> > > > > > >
> > > > > > > On Wed, Jan 12, 2022 at 8:39 AM mattison chao <
> > > > mattisonc...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1
> > > > > > > >
> > > > > > > > On Wed, 12 Jan 2022 at 08:09, Matteo Merli <
> mme...@apache.org>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > https://github.com/apache/pulsar/issues/13717
> > > > > > > > >
> > > > > > > > > -
> > > > > > > > >
> > > > > > > > > ## Motivation
> > > > > > > > >
> > > > > > > > > Since all the pieces that composed the proposal in PIP-45
> were
> > > > > > finally
> > > > > > > > > merged
> > > > > > > > > and are currently ready for 2.10 release, it is now
> possible to
> > > > add
> > > > > > other
> > > > > > > > > metadata backends that can be used to support a BookKeeper
> +
> > > > Pulsar
> > > > > > > > > cluster.
> > > > > > > > >
> > > > > > > > > One of the popular systems that is most commonly used as an
> > > > > > alternative
> > > > > > > > of
> > > > > > > > > ZooKeeper is Etcd, thus it makes sense to have this as the
> > > first
> > > > > > > > > non-zookeeper
> > > > > > > > > implementation.
> > > > > > > > >
> > > > > > > > > ## Goal
> > > > > > > > >
> > > > > > > > > Provide an Etcd implementation for the `MetadataStore` API.
> > > This
> > > > > will
> > > > > > > > allow
> > > > > > > > > users to deploy Pulsar clusters using Etcd service for the
> > > > metadata
> > > > > > and
> > > > > > > > it
> > > > > > > > > will
> > > > > > > > > not require the presence of ZooKeeper.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > ## Implementation
> > > > > > > > >
> > > > > > > > >  * Use the existing JEtcd Java client library for Etcd
> > > > > > > > >  * Extends the `AbstractBatchedMetadataStore` class, in
> order
> > > to
> > > > > > reuse
> > > > > > > > the
> > > > > > > > >transparent batching logic that will be shared with the
> > > > > ZooKeeper
> > > > > > > > >implementation.
> > > > > > > > >
> > > > > > > > > Work in progress:
> https://github.com/apache/pulsar/pull/13225
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Matteo Merli
> > > > > > > > > 
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>

[DISCUSS] [Transaction] Clear namespace backlog can't clear system topic and system sub backlog.

[丛搏]

Hi :

issue link : https://github.com/streamnative/support-tickets/issues/408 


# Background
Now we have many clear namespace backlog interfaces etc.
```
clearNamespaceBacklog(String namespace)

clearNamespaceBacklogAsync(String namespace)

clearNamespaceBacklogForSubscription(String namespace, String subscription)

clearNamespaceBacklogForSubscriptionAsync(String namespace, String subscription)

clearNamespaceBundleBacklog(String namespace, String bundle)

clearNamespaceBundleBacklogAsync(String namespace, String bundle)

clearNamespaceBundleBacklogForSubscription(String namespace, String bundle, 
String subscription)

clearNamespaceBundleBacklogForSubscriptionAsync(String namespace, String 
bundle, String subscription)
```
There are two types of interfaces:
1. clear namespace backlog with subscriptionName
2. clear namespace backlog no subscriptionName

But there have a problem, users do not know that there are many system topics 
in the namespace. 

```

/**
 * Local topic name for the namespace events.
 */
public static final String NAMESPACE_EVENTS_LOCAL_NAME = "__change_events";

/**
 * Local topic name for the transaction buffer snapshot.
 */
public static final String TRANSACTION_BUFFER_SNAPSHOT = 
"__transaction_buffer_snapshot";

public static final String SUBSCRIPTION_NAME = "transaction-buffer-sub";
```
User only want to clear the backlog witch they own created and sub. But now we 
have clear the system topic sub backlog when user don't specify subName and 
clear system sub with a topic witch user created. It will bring many problems. 
etc.
1. clear `NAMESPACE_EVENTS_LOCAL_NAME` will clear any topic policy in this 
namespace, but user don't know and they don't want to be cleared. If the user 
does not understand the problems caused by this interface, it will delete many 
topic policy configurations and is difficult to recover, which will lead to 
unpredictable and disastrous problems.
2. clear `TRANSACTION_BUFFER_SNAPSHOT` topic backlog will cause transaction 
buffer recover incompletely.
3. clear `transaction-buffer-sub` sub also will cause transaction buffer 
recover incompletely.
4. if add new system topic and system sub, we should think about this problem.

# Motivation
We want to solve the above problems, but we can't make users unable to delete 
the backlog of system topic and system sub.
So, we should solve types of interface
There are two types of interfaces:
1. clear namespace backlog with subscriptionName
2. clear namespace backlog no subscriptionName

when clear namespace backlog no subscriptionName: 
It represents user know the sub name is system subName and really want to 
delete it, We should not block the user clear system sub backlog and do not 
allow the user to create system sub. If the user uses it, it means that the 
user has considered the result of the clear system sub backlog

when clear namespace backlog with subscriptionName:
Because the user may not consider the existence of system topic and system sub, 
and the wrong clear backlog may lead to serious results. Therefore, this 
interface cannot delete the backlog of system topic and the backlog of system 
sub. If the user really wants to clear backlog, please bring subscriptionName 
and use another type of interface with subscriptionName.

**Clear namespace and clear namespace bundle all need to handle is logical.**

# implement

```
class SystemTopicName {
/**
 * Local topic name for the namespace events.
 */
public static final String NAMESPACE_EVENTS_LOCAL_NAME = "__change_events";

/**
 * Local topic name for the transaction buffer snapshot.
 */
public static final String TRANSACTION_BUFFER_SNAPSHOT = 
"__transaction_buffer_snapshot";
}
```
```
class SystemSubName {

public static final String SUBSCRIPTION_NAME = "transaction-buffer-sub";

}
```

1. In order to make the code easier to maintain and reduce redundancy, we need 
to move systemTopic and systemSub to a specific classes to manage.
2. If the newly added systemTopic and systemSub are created from this class, 
unpredictable problems can be prevented



Let me know if you have different ideas or suggestions！！

Thanks!
Bo Cong