date:20221013

Re: [ANNOUNCE] New Committer: Qiang Huang

2022-10-13 Thread Max Xu

Congratulations! Qiang

Best,
Max Xu


On Fri, Oct 14, 2022 at 12:04 PM guo jiwei  wrote:

> The Project Management Committee (PMC) for Apache Pulsar has invited
> Qiang Huang (https://github.com/hqebupt) to become a committer and
> we are pleased to announce that he has accepted.
>
> Qiang Huang (with Github id: hqebupt) contributed many improvements
> and bug fixes to Pulsar.
>
> Being a committer enables easier contribution to the project since
> there is no need to go via the patch submission process. This should
> enable better productivity.
>
> Welcome and Congratulations, Qiang Huang!
>
> Please join us in congratulating and welcoming Qiang Huang onboard!
>
>
> Regards
> Jiwei Guo on behalf of the Pulsar PMC
>

Re: [ANNOUNCE] New Committer: Qiang Huang

2022-10-13 Thread houxiaoyu

Congratulations Qiang Huang

Thanks,
Xiaoyu Hou

guo jiwei  于2022年10月14日周五 12:04写道：

> The Project Management Committee (PMC) for Apache Pulsar has invited
> Qiang Huang (https://github.com/hqebupt) to become a committer and
> we are pleased to announce that he has accepted.
>
> Qiang Huang (with Github id: hqebupt) contributed many improvements
> and bug fixes to Pulsar.
>
> Being a committer enables easier contribution to the project since
> there is no need to go via the patch submission process. This should
> enable better productivity.
>
> Welcome and Congratulations, Qiang Huang!
>
> Please join us in congratulating and welcoming Qiang Huang onboard!
>
>
> Regards
> Jiwei Guo on behalf of the Pulsar PMC
>

Re: [DISCUSS] Pulsar Metrics - Current State and Future Directions

2022-10-13 Thread Michael Marshall

Hi Asaf,

This is a great topic for discussion, and your document is extremely
thorough! I agree with the general proposal to improve Pulsar's
metrics.

> *Metrics Cardinality: *

+100 if we want to scale Pulsar (and I do!) we need to make this manageable

> *Consolidate into a single library:*

This makes sense to me, and it ensures that new metrics will not be in
one API but not another.

I haven't read the whole doc, but I did read the suggested
improvements. Here are some additional improvements that I've thought
about before.

Are there any metrics we can drop? This would definitely require a
community effort to verify, but I think it could prove valuable.

Can we make the number of histogram buckets configurable? I proposed
this here [0].

Would it be possible to produce a script to help users convert
existing grafana dashboards to work with the new metrics?

Finally, it'd be great to create a metrics section in the contributors
guide when you've completed your work. That will help existing and new
contributors adjust to the new style.

Thanks,
Michael

[0] https://github.com/apache/pulsar/issues/12069

On Mon, Oct 3, 2022 at 3:36 AM Asaf Mesika  wrote:
>
> Hi All,
>
> I would like to share with you a document I wrote during the last months
> titled Pulsar Metrics - Current State and Future Directions
> ,
> and most importantly *get your feedback.*
>
> The initial motivation is to rethink/refactor the way metrics are used in
> Pulsar codebase to solve two large pain points:
>
> 1. *Metrics Cardinality: *As Pulsar can support up to 1M topics
> cluster-wide, this translates into ~100M unique time series, which becomes
> both an impossible cost and affects query performance and general usability
> of metrics. This issue starts surfacing even at 50k-100k topics.
>
> Today users work-around it by disabling topic-granularity metrics and
> scripting their own ETL for generating metrics they can use (based on admin
> stats API), switching between granular topic-level metrics to a group-by
> view of their choosing.
>
> The document outlines a solution built upon the notion of Groups, in which
> users can define a group of metrics, and specify if they wish to define a
> roll-up on it (i.e. remove labels) and filter (i.e. remove specific
> metrics).
> The solution should be able to bring the granularity from topic level (1M)
> to group level (1000).
>
> 2. *Consolidate into a single library:* Today there are 4 different metrics
> libraries/systems in Pulsar. This creates lots of confusion and unhappy
> developer experience, among other impacts. Also achieving (1) requires
> having (2).
>
> The document outlines the different libraries, their functionality and the
> problems they create. The doc also describes one idea for such a library,
> but it still requires a POC.
>
>
> The main goal of the document is mainly to garner feedback to see if the
> directions stipulated there are agreed upon, and if there is any other
> problem missing or existing functionality missed as it serves as the basis
> for the requirements for the solution that will be chosen.
>
> Thanks!
>
> Asaf Mesika
>
> Document link:
> https://docs.google.com/document/d/1vke4w1nt7EEgOvEerPEUS-Al3aqLTm9cl2wTBkKNXUA/edit?usp=sharing

[ANNOUNCE] New Committer: Qiang Huang

2022-10-13 Thread guo jiwei

The Project Management Committee (PMC) for Apache Pulsar has invited
Qiang Huang (https://github.com/hqebupt) to become a committer and
we are pleased to announce that he has accepted.

Qiang Huang (with Github id: hqebupt) contributed many improvements
and bug fixes to Pulsar.

Being a committer enables easier contribution to the project since
there is no need to go via the patch submission process. This should
enable better productivity.

Welcome and Congratulations, Qiang Huang!

Please join us in congratulating and welcoming Qiang Huang onboard!


Regards
Jiwei Guo on behalf of the Pulsar PMC

Pulsar Community Meeting Notes 2022/10/13, (8:30 AM PST)

2022-10-13 Thread Michael Marshall

Hi Pulsar Community,

Here are the notes from the community meeting two weeks ago. We had
several very good discussions.

I want to emphasize that we do not make any official decisions during
these meetings. It is an essential part of the Apache Way [0] to make
project decisions on the dev mailing list. I share the meeting notes
to archive insights, context, and ideas from the meetings that might
hopefully be valuable as we work to improve Pulsar.

Disclaimer: if something is misattributed or misrepresented, please
send a correction to this list.

Source google doc:
https://docs.google.com/document/d/19dXkVXeU2q_nHmkG8zURjKnYlvD96TbKf5KjYyASsOE

[0] https://www.apache.org/theapacheway/

Thanks,
Michael

-2022/10/13, (8:30 AM PST)
- Attendees:
- Matteo Merli
- Michael Marshall
- Lari Hotari
- Dave Fisher
- Andrey Yegorov
- Rajan Dhabalia

- Discussions/PIPs/PRs (Generally discussed in order they appear)

- Michael: what is the current flow for getting PRs to run CI? Lari:
CI will run when a PR is either (approved or has the ready-to-test
label) and the “/pulsarbot rerun-failure-scenarios” comment is added
to the PR. Also, we have had the flaky tests failing for a while and
we don’t have enough attention on the flaky tests. We do seem to have
more resources. Dave: we could could use ASF infra. Matteo: we
switched because the experience there was terrible. Dave: there have
been a lot of updates in the past couple years, though I haven’t
tested it. Matteo: also, switching CI jobs is a ton of work. There was
resource contention previously on the ASF infra. Dave: you can get
your own dedicated works. Companies do fund project related nodes.
Matteo: you can also do this with GitHub actions. Dave: there are some
security concerns. When will the ASF infra trust runners. Matteo: one
of the things in GitHub actions is to have MacOS and Windows runners.
Michael: the flaky test reports are really helpful. Lari: that is
something I handed off to Nicolò. At Apache Con, Gradle announced that
there will be gradle enterprise integration. This will give us live
reports on the build, flaky tests, and more. Matteo: is it a runner or
uploading the build data? Lari: it is an upload. Dave: there is also a
speculative test run feature. Matteo: we’d need to build with gradle.
Lari: they have maven plugins that integrate into maven. Matteo: I
would still support switching to gradle. There were issues in the
bookkeeper project. Lari: I was late to it, but I was disappointed to
see that initiative fall through. Matteo: I feel similarly. We’ll need
strong support from multiple committers to get it done. It would
improve developer experience. It would be a lot of work and we need to
decide that we want to get it done and we should be very committed to
get it done. Dave: I’ve seen projects where there were big problems in
the community and if the change doesn’t have complete buy in, it’ll
cause fracture. Matteo: we also need to make sure that multiple people
will push on the work. If there is enough community support, we can
get this done. Rajan: is the main benefit of switching speed? Lari:
one of the first benefits is that is has a correct incremental build.
Many of the benefits are around that. Build caching is correct and
consistent. Rajan: so the speed? Lari: it’s also the correctness.
Often with maven, you need to do a clean install because you don’t
know that it is correct. Andrey: was correctness the issue in the
bookkeeper project? Matteo: it was related to the build. Andrey: it
had to do with the binary artifacts and issues with how they were
produced. Then, the people who contributed it were unavailable to
support it. Matteo: that is why we want to make sure we have buy in.
The other benefit is the preciseness of the dependencies. In maven, we
are using indirect dependencies, so we don’t have a strong sense of
what we need. With gradle, by default, you have to declare
dependencies. This helps clarify the classpath.

- Rajan: https://github.com/apache/pulsar/pull/17962
- blue/green cluster migration support. PIP 188. This is the first
part which is most of the changes. The second part requires some other
changes to maintain ordering. I plan to implement that once this first
PR is merged. Please review it if you have time. Lari: this is late to
share, but I came across another submission of a similar feature, but
it wasn’t submitted to the project itself.
https://github.com/pkumar-singh/pulsar/wiki/PIP-95:-Live-migration-of-producer-consumer-or-reader-from-one-Pulsar-cluster-to-another
Matteo: that design is from splunk. Rajan: that was incomplete, and
PIP 188 was made as a kind of successor.

- Michael: just want to check in on
https://github.com/apache/pulsar/issues/18012. There was a question
about multiple zookeeper instances (see issue for context). Will that
be a blocker for this PIP? Matteo: it should not be a blocker. That is
a non-standard deployment (though some users do take that approach).

Re: [VOTE] Pulsar Node.js Client Release 1.7.0 Candidate 2

2022-10-13 Thread Yuri Mizushima

Hi Masahiro,

It seems the C++ client causes the AlreadyClosed issue.
https://github.com/apache/pulsar-client-cpp/issues/51

I think this issue doesn't block releasing new Node.js client.
What do you think?

Thanks,
--
Yuri Mizushima
yumiz...@yahoo-corp.jp


"Masahiro Sakamoto"  wrote:

-1 (binding)

I almost always get "Failed to send message: AlreadyClosed" errors when running 
the following code:
##
$ cat producer.js
const Pulsar = require('./index.js');

(async () => {
  const client = new Pulsar.Client({
serviceUrl: 'pulsar://localhost:6650',
  });

  const producer = await client.createProducer({
topic: 'persistent://public/default/t1',
batchingEnabled: false,
  });

  for (let i = 0; i < 10; i += 1) {
const msg = `my-message-${i}`;
producer.send({
  data: Buffer.from(msg),
});
console.log(`Sent message: ${msg}`);
  }
  await producer.flush();

  await producer.close();
  await client.close();
})();
$ node producer.js
Sent message: my-message-0
Sent message: my-message-1
Sent message: my-message-2
Sent message: my-message-3
Sent message: my-message-4
Sent message: my-message-5
Sent message: my-message-6
Sent message: my-message-7
Sent message: my-message-8
Sent message: my-message-9
(node:18572) UnhandledPromiseRejectionWarning: Error: Failed to send message: 
AlreadyClosed
(Use `node --trace-warnings ...` to show where the warning was created)
(node:18572) UnhandledPromiseRejectionWarning: Unhandled promise rejection. 
This error originated either by throwing inside of an async function without a 
catch block, or by rejecting a promise which was not handled with .catch(). To 
terminate the node process on unhandled promise rejection, use the CLI flag 
`--unhandled-rejections=strict` (see 
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnodejs.org%2Fapi%2Fcli.html%23cli_unhandled_rejections_modedata=05%7C01%7Cyumizush%40yahoo-corp.jp%7Cad44a31ff988434d3f1f08daa82142b0%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C638007158045043149%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=d31GWwFv0GBd3ggHyez5T5YWTPq0SFxW%2B6YqTSOnyDw%3Dreserved=0).
 (rejection id: 1)
(node:18572) [DEP0018] DeprecationWarning: Unhandled promise rejections are 
deprecated. In the future, promise rejections that are not handled will 
terminate the Node.js process with a non-zero exit code.
(node:18572) UnhandledPromiseRejectionWarning: Error: Failed to send message: 
AlreadyClosed
(node:18572) UnhandledPromiseRejectionWarning: Unhandled promise rejection. 
This error originated either by throwing inside of an async function without a 
catch block, or by rejecting a promise which was not handled with .catch(). To 
terminate the node process on unhandled promise rejection, use the CLI flag 
`--unhandled-rejections=strict` (see 
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnodejs.org%2Fapi%2Fcli.html%23cli_unhandled_rejections_modedata=05%7C01%7Cyumizush%40yahoo-corp.jp%7Cad44a31ff988434d3f1f08daa82142b0%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C638007158045043149%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=d31GWwFv0GBd3ggHyez5T5YWTPq0SFxW%2B6YqTSOnyDw%3Dreserved=0).
 (rejection id: 2)
(node:18572) UnhandledPromiseRejectionWarning: Error: Failed to send message: 
AlreadyClosed
(node:18572) UnhandledPromiseRejectionWarning: Unhandled promise rejection. 
This error originated either by throwing inside of an async function without a 
catch block, or by rejecting a promise which was not handled with .catch(). To 
terminate the node process on unhandled promise rejection, use the CLI flag 
`--unhandled-rejections=strict` (see 
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnodejs.org%2Fapi%2Fcli.html%23cli_unhandled_rejections_modedata=05%7C01%7Cyumizush%40yahoo-corp.jp%7Cad44a31ff988434d3f1f08daa82142b0%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C638007158045043149%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=d31GWwFv0GBd3ggHyez5T5YWTPq0SFxW%2B6YqTSOnyDw%3Dreserved=0).
 (rejection id: 3)
(node:18572) UnhandledPromiseRejectionWarning: Error: Failed to send message: 
AlreadyClosed
(node:18572) UnhandledPromiseRejectionWarning: Unhandled promise rejection. 
This error originated either by throwing inside of an async function without a 
catch block, or by rejecting a promise which was not handled with .catch(). To 
terminate the node process on unhandled promise rejection, use the CLI flag 
`--unhandled-rejections=strict` (see

Pulsar Community Meeting Notes 2022/09/29, (8:30 AM PST)

2022-10-13 Thread Michael Marshall

Hi Pulsar Community,

Here are the notes from the community meeting two weeks ago. It was a
brief meeting.

Disclaimer: if something is misattributed or misrepresented, please
send a correction to this list.

Source google doc:
https://docs.google.com/document/d/19dXkVXeU2q_nHmkG8zURjKnYlvD96TbKf5KjYyASsOE

Thanks,
Michael

2022/09/29, (8:30 AM PST)
-   Attendees:
-   Matteo Merli
-   Nicolò Boschi
-   Michael Marshall
-   Ayman Khalil

-   Discussions/PIPs/PRs (Generally discussed in order they appear)

-   Nicolò: CI is broken right now, there is a PR to fix it, and it is
still making it through the CI before it can be merged.

-   Matteo: started moving the C++ code out of the build. There is
still more work, but once it is done, we can do the cut off. The code
is not yet removed. The new repo will have different commit hashes.
The history will be intact. We’ll make sure no commits get missed.

46 matches

Mail list logo