[GitHub] [pulsar] startjava added a comment to the discussion: what is "滚动策略循环"?

GitHub user startjava added a comment to the discussion: what is "滚动策略循环"? https://pulsar.apache.org/docs/next/concepts-architecture-overview/#managed-ledgers A ledger can be deleted when all cursors have consumed the messages it contains. This allows for periodic rollover of ledgers.

Re: [DISCUSS] PIP-221: Make TableView support read the non-persistent topic

Hi Joe, > I am not sure about the semantics of TableView on a non-persistent topic. > What happens if the client crashes? What is the base state for the table? If users use a non-persistent topic as the TableView topic, when the client crashes, the TableViews data will be lose. The current

[Discuss] Deprecate Index-based Publisher Stat Aggregation in Topics Partitioned-Stats

Dear Pulsar Community, We recently found a bug in `pulsar-admin topics partitioned-stats api` that could incur a memory burst, high GC time, or OOM. For this issue, I proposed a fix by deprecating the aggregatePublisherStatsByProducerName config and

Re: [DISCUSS] PIP-221: Make TableView support read the non-persistent topic

Hi Michael, > What time is used to compute expiration? Is it the publish time or the > receive time? This TTL will be based on the message publish time. We can also make it configurable if users have this demand. > Also, are there cases that will reset a key's timer? If some keys need to reset

Re: [Vote] PIP-215: Configurable TopicCompactionStrategy for StrategicTwoPhaseCompactor and TableView

+1 Thanks, Penghui On Wed, Nov 9, 2022 at 12:52 AM Heesung Sohn wrote: > Dear Pulsar Community, > > Please review and vote on this PIP. > > PIP link: https://github.com/apache/pulsar/issues/18099 > > Thank you, > -Heesung >

Re: [DISCUSS] PIP-221: Make TableView support read the non-persistent topic

I am not sure about the semantics of TableView on a non-persistent topic. Exactly how does this work? What happens if the client crashes? What is the base state for the table? What exactly can I expect as a user from this? Joe On Sun, Nov 13, 2022 at 8:57 PM Kai Wang wrote: > Hi,

Re: [DISCUSS] PIP-221: Make TableView support read the non-persistent topic

> And this PIP title has been changed to `Make TableView support TTL`. What time is used to compute expiration? Is it the publish time or the receive time? Also, are there cases that will reset a key's timer? Thanks, Michael On Mon, Nov 14, 2022 at 2:40 AM Enrico Olivelli wrote: > > Il giorno

[GitHub] [pulsar] github-actions[bot] added a comment to the discussion: deb packages

GitHub user github-actions[bot] added a comment to the discussion: deb packages The issue had no activity for 30 days, mark with Stale label. GitHub link: https://github.com/apache/pulsar/discussions/18464#discussioncomment-4137548 This is an automatically sent email for

[GitHub] [pulsar] hpvd added a comment to the discussion: deb packages

GitHub user hpvd added a comment to the discussion: deb packages +1 for this! GitHub link: https://github.com/apache/pulsar/discussions/18464#discussioncomment-4137549 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to:

[GitHub] [pulsar] github-actions[bot] added a comment to the discussion: deb packages

GitHub user github-actions[bot] added a comment to the discussion: deb packages The issue had no activity for 30 days, mark with Stale label. GitHub link: https://github.com/apache/pulsar/discussions/18464#discussioncomment-4137550 This is an automatically sent email for

[GitHub] [pulsar] pablopla added a comment to the discussion: deb packages

GitHub user pablopla added a comment to the discussion: deb packages @hwittenborn makedeb could be great. Why is this issue stale? GitHub link: https://github.com/apache/pulsar/discussions/18464#discussioncomment-4137547 This is an automatically sent email for dev@pulsar.apache.org. To

[GitHub] [pulsar] github-actions[bot] added a comment to the discussion: deb packages

GitHub user github-actions[bot] added a comment to the discussion: deb packages The issue had no activity for 30 days, mark with Stale label. GitHub link: https://github.com/apache/pulsar/discussions/18464#discussioncomment-4137546 This is an automatically sent email for

[GitHub] [pulsar] hwittenborn added a comment to the discussion: deb packages

GitHub user hwittenborn added a comment to the discussion: deb packages Would [makedeb](https://makedeb.org) be of any interested for this? It's a project I personally lead, but I think it could greatly reduce the maintenance cost of having a Debian package. GitHub link:

Re: Request a site ID and tracking code for Apache Pulsar

Hi Tison, We currently don't offer account creation outside of the Privacy group. However, feel free to send me your requirements for the segment and I'll create it for you :) Thanks, Martijn On Thu, Nov 10, 2022 at 7:07 AM tison wrote: > Hi Martijn, > > Thanks for your help! We successfully

[GitHub] [pulsar] oversearch created a discussion: Question: copy or move topics between Pulsar clusters

GitHub user oversearch created a discussion: Question: copy or move topics between Pulsar clusters **Is your feature request related to a problem? Please describe.** I'm wondering if it's currently possible to copy or move a topic that exists in one Pulsar cluster to another independent

[GitHub] [pulsar] codelipenghui added a comment to the discussion: Question: copy or move topics between Pulsar clusters

GitHub user codelipenghui added a comment to the discussion: Question: copy or move topics between Pulsar clusters The issue had no activity for 30 days, mark with Stale label. GitHub link: https://github.com/apache/pulsar/discussions/18461#discussioncomment-4136189 This is an

Re: Data quality problem

While we can get caught up in the specifics of exactly how JSON Schema is supported in the Kafka ecosystem, it is ultimately possible if desired, and is common, even if not part of open-source Apache Kafka. Devin's assertion is that JSON Schema compliant payload validation and schema evolution

[GitHub] [pulsar] bigbang489 added a comment to the discussion: [Pulsar Client] Is there a way to specify a schema version instead of uploading SchemaInfo to broker?

GitHub user bigbang489 added a comment to the discussion: [Pulsar Client] Is there a way to specify a schema version instead of uploading SchemaInfo to broker? We are building an adapter to integrate Pulsar to SAP Integraion system. Sender adapter will consume message from a topic, Receiver

[GitHub] [pulsar] sijie added a comment to the discussion: [Pulsar Client] Is there a way to specify a schema version instead of uploading SchemaInfo to broker?

GitHub user sijie added a comment to the discussion: [Pulsar Client] Is there a way to specify a schema version instead of uploading SchemaInfo to broker? > Is it better if we just specify the schema version instead of whole > SchemaInfo? We send the schema definition when a client connect

[GitHub] [pulsar] bigbang489 created a discussion: [Pulsar Client] Is there a way to specify a schema version instead of uploading SchemaInfo to broker?

GitHub user bigbang489 created a discussion: [Pulsar Client] Is there a way to specify a schema version instead of uploading SchemaInfo to broker? **Is your enhancement request related to a problem? Please describe.** When conumser/producer connect to broker, it has to upload the schema info

[GitHub] [pulsar] vitosans added a comment to the discussion: Messages lost with new topic and regex subscription

GitHub user vitosans added a comment to the discussion: Messages lost with new topic and regex subscription @sijie - What is the official position on this? Is it suggested to use earliest? I see it has gone stale and has not been updated for two years. We are running into this issue, which

[GitHub] [pulsar] baynes added a comment to the discussion: Messages lost with new topic and regex subscription

GitHub user baynes added a comment to the discussion: Messages lost with new topic and regex subscription I guess that would do it. It changes the behavior for topics created while the client is down or before it is started for the first time. It is probably useful to read all the messages

[GitHub] [pulsar] sijie added a comment to the discussion: Messages lost with new topic and regex subscription

GitHub user sijie added a comment to the discussion: Messages lost with new topic and regex subscription @baynes noted. GitHub link: https://github.com/apache/pulsar/discussions/18458#discussioncomment-4133561 This is an automatically sent email for dev@pulsar.apache.org. To

[GitHub] [pulsar] BewareMyPower added a comment to the discussion: Messages lost with new topic and regex subscription

GitHub user BewareMyPower added a comment to the discussion: Messages lost with new topic and regex subscription The same applies to a partitioned consumer. IMO, when a consumer found new topics/partitions, the subscription initial position **should be changed to earliest** no matter what

[GitHub] [pulsar] baynes added a comment to the discussion: Messages lost with new topic and regex subscription

GitHub user baynes added a comment to the discussion: Messages lost with new topic and regex subscription Might be some overlap with #6531 GitHub link: https://github.com/apache/pulsar/discussions/18458#discussioncomment-4133557 This is an automatically sent email for

[GitHub] [pulsar] sijie added a comment to the discussion: Messages lost with new topic and regex subscription

GitHub user sijie added a comment to the discussion: Messages lost with new topic and regex subscription @baynes You can specify SubscriptionInitialPosition.earliest when you create the regex subscription. GitHub link:

[GitHub] [pulsar] baynes added a comment to the discussion: Messages lost with new topic and regex subscription

GitHub user baynes added a comment to the discussion: Messages lost with new topic and regex subscription > > > Might be some overlap with #6531 Further thoughts. I think that is more to do with topics created before function creation/start - and the initial_position option of the consumer

[GitHub] [pulsar] baynes created a discussion: Messages lost with new topic and regex subscription

GitHub user baynes created a discussion: Messages lost with new topic and regex subscription **Describe the bug** When a new topic is detected by a regexp subscription it takes time before the subscriptions cursor is set up for that topic. As the cursor is set to the end of the topic this

[GitHub] [pulsar] kitekite2020 added a comment to the discussion: Active-active and no data loss. Is it possible?

GitHub user kitekite2020 added a comment to the discussion: Active-active and no data loss. Is it possible? > Yes, the metrics related to the replication is described at > https://pulsar.apache.org/docs/en/reference-metrics/#replication-metrics-1, > and

[GitHub] [pulsar] kitekite2020 added a comment to the discussion: Active-active and no data loss. Is it possible?

GitHub user kitekite2020 added a comment to the discussion: Active-active and no data loss. Is it possible? Would like to clarify for geo-replication message delivery. Is it more like TCP, every message got acknowledged when it is replicated successfully to the other site, or more like UDP,

[GitHub] [pulsar] codelipenghui added a comment to the discussion: Active-active and no data loss. Is it possible?

GitHub user codelipenghui added a comment to the discussion: Active-active and no data loss. Is it possible? Yes, it is stored in bookies(data is replicated in the certain bookies). So you can increase the data replicas to avoid data loss. GitHub link:

[GitHub] [pulsar] codelipenghui added a comment to the discussion: Active-active and no data loss. Is it possible?

GitHub user codelipenghui added a comment to the discussion: Active-active and no data loss. Is it possible? Yes, the metrics related to the replication is described at https://pulsar.apache.org/docs/en/reference-metrics/#replication-metrics-1, and

[GitHub] [pulsar] kitekite2020 added a comment to the discussion: Active-active and no data loss. Is it possible?

GitHub user kitekite2020 added a comment to the discussion: Active-active and no data loss. Is it possible? > Yes, it is stored in bookies(data is replicated in the certain bookies). So > you can increase the data replicas to avoid data loss. Thank you. So I understand from you that there is

[GitHub] [pulsar] kitekite2020 added a comment to the discussion: Active-active and no data loss. Is it possible?

GitHub user kitekite2020 added a comment to the discussion: Active-active and no data loss. Is it possible? > Is the message replication guaranteed? The replication state is stored in a cursor(ledger backend) -> Does this mean it is stored in bookie? What if this bookie crashes? GitHub

[GitHub] [pulsar] kitekite2020 added a comment to the discussion: Active-active and no data loss. Is it possible?

GitHub user kitekite2020 added a comment to the discussion: Active-active and no data loss. Is it possible? We would like to have pulsar clusters in two data centers. They are active and active. And we would like to replicate messages from one site to the other. i.e. the subscriber from one

[GitHub] [pulsar] codelipenghui added a comment to the discussion: Active-active and no data loss. Is it possible?

GitHub user codelipenghui added a comment to the discussion: Active-active and no data loss. Is it possible? > We are thinking of geo replication. But since geo replication is > asynchronous, we are worried about message loss. Every single message cannot > be lost. The geo-replication is

[GitHub] [pulsar] kitekite2020 created a discussion: Active-active and no data loss. Is it possible?

GitHub user kitekite2020 created a discussion: Active-active and no data loss. Is it possible? **Is your feature request related to a problem? Please describe.** A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] **Describe the solution you'd like**

Re: Releasing current master as Pulsar 2.11.0 ?

I found out that several PRs have been unable to cherry-pick to 2.11 today. I agree to cut the new branch based on the master and turn off the new/unstable features in branch-2.11. Regards Tboy On Fri, Nov 4, 2022 at 1:00 PM Dave Fisher wrote: > Inline > > Sent from my iPhone > > > On Nov

Re: [ANNOUNCE] New Committer: Lin Chen

Congrats! Lin Best, Zixuan Enrico Olivelli 于2022年11月14日周一 15:06写道： > Congratulations > > Enrico > > Il Lun 14 Nov 2022, 07:30 Qiang Huang ha > scritto: > > > Congratulations! > > > > houxiaoyu 于2022年11月14日周一 13:40写道： > > > > > Congrats! > > > > > > Best, > > > Xiaoyu Hou > > > > > > PengHui

Re: [ANNOUNCE] New Committer: Zili Chen

Congrats! Zili Best, Zixuan Qiang Huang 于2022年11月14日周一 14:31写道： > Congratulations! > > Zike Yang 于2022年11月14日周一 13:45写道： > > > Hi, tison, > > > > Congratulations and welcome! > > > > BR, > > Zike Yang > > > > On Mon, Nov 14, 2022 at 12:58 PM Kai Wang > > wrote: > > > > > > Congratulations!

Re: [DISCUSS] PIP-221: Make TableView support read the non-persistent topic

Il giorno lun 14 nov 2022 alle ore 05:57 Kai Wang ha scritto: > > Hi, pulsar-dev community, > > Since the non-persistent topic support doesn't require API changes. I have > pushed a PR to implement it, which has already been merged. > > See: https://github.com/apache/pulsar/pull/18375 Perfect

[GitHub] [pulsar] tisonkun added a comment to the discussion: Support for long running message consumer

GitHub user tisonkun added a comment to the discussion: Support for long running message consumer I'm moving this discussion to the Discussions forum since it's an open-ended discussion instead of an actionable task :) GitHub link:

[GitHub] [pulsar] harissecic added a comment to the discussion: Support for long running message consumer

GitHub user harissecic added a comment to the discussion: Support for long running message consumer Stumbled upon this looking for another answer. However in case it helps anyone I'll leave a comment. For such cases I guess it's possible to combine DLQ with ackTimeout. Default value I use is

[GitHub] [pulsar] tisonkun added a comment to the discussion: Support for long running message consumer

GitHub user tisonkun added a comment to the discussion: Support for long running message consumer Closed as answered. GitHub link: https://github.com/apache/pulsar/discussions/18456#discussioncomment-4133183 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe,

[GitHub] [pulsar] harissecic added a comment to the discussion: Support for long running message consumer

GitHub user harissecic added a comment to the discussion: Support for long running message consumer > There is still no good solution for retrying long running jobs. I think there's plenty of _good enough_ workarounds but I agree there should be one optimal for long running consumers

[GitHub] [pulsar] benbro added a comment to the discussion: Support for long running message consumer

GitHub user benbro added a comment to the discussion: Support for long running message consumer There is still no good solution for retrying long running jobs. GitHub link: https://github.com/apache/pulsar/discussions/18456#discussioncomment-4133184 This is an automatically sent email

[GitHub] [pulsar] sijie added a comment to the discussion: Support for long running message consumer

GitHub user sijie added a comment to the discussion: Support for long running message consumer @shubham-Shole4ever does Matteo's comment make sense to you? GitHub link: https://github.com/apache/pulsar/discussions/18456#discussioncomment-4133180 This is an automatically sent email for

[GitHub] [pulsar] shubham-Shole4ever added a comment to the discussion: Support for long running message consumer

GitHub user shubham-Shole4ever added a comment to the discussion: Support for long running message consumer @sijie I can do with the workaround suggested by @codelipenghui and @merlimat for the time being. However, as mentioned, the solution will not work in case I also have a need of

[GitHub] [pulsar] merlimat added a comment to the discussion: Support for long running message consumer

GitHub user merlimat added a comment to the discussion: Support for long running message consumer @shubham-Shole4ever When a consumer crashes, or the TCP connection is broken, the messages that were delivered to this consumer and not acked, will be replayed to another available consumer (in

[GitHub] [pulsar] shubham-Shole4ever added a comment to the discussion: Support for long running message consumer

GitHub user shubham-Shole4ever added a comment to the discussion: Support for long running message consumer @codelipenghui But if my application crashes while it is processing the message, it'll never be able to ack/negative ack that message ever. This'll result in that message never being

[GitHub] [pulsar] codelipenghui added a comment to the discussion: Support for long running message consumer

GitHub user codelipenghui added a comment to the discussion: Support for long running message consumer @shubham-Shole4ever You can disable ack timeout, just use ack/negative ack. It means explicitly telling the broker that the process failed and then the broker redeliver this message, if

[GitHub] [pulsar] codelipenghui added a comment to the discussion: Support for long running message consumer

GitHub user codelipenghui added a comment to the discussion: Support for long running message consumer Please take a look at this document which may help you. http://pulsar.apache.org/docs/en/concepts-messaging/#negative-acknowledgement GitHub link:

[GitHub] [pulsar] shubham-Shole4ever added a comment to the discussion: Support for long running message consumer

GitHub user shubham-Shole4ever added a comment to the discussion: Support for long running message consumer @codelipenghui I had a look at the negative-acknowledgement. This will still not work if my ackTimeout is set to 10 mins and the message I am consuming is taking 30 mins (for e.g.).

[GitHub] [pulsar] shubham-Shole4ever created a discussion: Support for long running message consumer

GitHub user shubham-Shole4ever created a discussion: Support for long running message consumer **Is your feature request related to a problem? Please describe.** The ackTimeout is set at the consumer level and is valid for all the messages that consumer handles. We have a case where the