Re: Suggestions on GitHub labels and issue templates

[Kiryl Valkovich]

Hi PengHui,
Sure. If the PR is welcome here, I’ll submit it in a few days.


Best,
Kiryl

> On Mar 25, 2024, at 6:07 AM, PengHui Li  wrote:
> 
> Hi Kiryl,
> 
> Thanks for your suggestions, and they are looking good to me
> I'll follow your suggestions on renaming or deprecating the labels.
> 
> For the label automation, do you want to push a PR to add it?
> 
> Regards,
> Penghui
> 
> 
> On Mon, Mar 18, 2024 at 9:39 PM Kiryl Valkovich 
> wrote:
> 
>> Comment with better formatting on GitHub:
>> https://github.com/apache/pulsar/issues/22277#issuecomment-2002553745
>> 
>>• Deprecate java label. Pulsar is written in Java and most PRs update
>> Java code.
>>• Instead of removing labels, deprecate them by renaming them to
>> deprecated/. Probably pick another prefix that is
>> alphabetically closer to the end of the alphabet to reduce noise.
>>• Add go label automatically using labeler:
>> https://github.com/apache/pulsar/blob/master/.github/labeler.yml
>> go:
>> - changed-files:
>> - any-glob-to-any-file: '**/*.go'
>> 
>>• Add component/* labels automatically based on the file path
>> component/config:
>> - changed-files:
>> - any-glob-to-any-file: 'conf/**/*'
>> - any-glob-to-any-file: 'pulsar-config-validation/**/*'
>> component/client:
>> - changed-files:
>> - any-glob-to-any-file: 'pulsar-client/**/*'
>> - any-glob-to-any-file: 'pulsar-client-*/**/*'
>> ...
>> 
>>• Rename bug label to type/bug for consistency. Keep the red color.
>>• (?) Rename component/* => area/* for shorter names. The
>> https://github.com/kubernetes/kubernetes/labels has such naming.
>>• Rename doc-required label to type/doc. Relabel open issues and PRs
>> with doc labels to the type/doc.
>>• Deprecate all other doc-* labels. If it is needed for some kind of
>> workflow, simply use the board project with ToDo -> In Progress -> Done
>> states.
>>• (?) Probably it makes sense to enable and track website and docs
>> issues in apache/pulsar-site repository. And add a good visible link to
>> apache/pulsar README.md.
>>• Deprecate the question label. Instead, move such issues to
>> Discussions -> Q
>>• Migrate issues with the enhancement either to type/feature label or
>> Discussions. Add a new Suggest an idea issue template that redirects to the
>> Discussions -> Ideas
>>• (?) Rename PIP => type/PIP for consistency
>>• Rename flaky-test => type/flaky-test to consistency
>>• Deprecate lifecycle/stale label. Use Stale instead. Rename Stale =>
>> stale for consistency.
>>• Add the ability to pick an area/* label from the dropdown on issue
>> creation.
>> systemd/systemd and a few other projects use this action for that:
>> https://github.com/redhat-plumbers-in-action/advanced-issue-labeler?tab=readme-ov-file#real-life-examples
>> 
>> 
>> Best,
>> Kiryl
>> 
>> 

Suggestions on GitHub labels and issue templates

[Kiryl Valkovich]

Comment with better formatting on GitHub: 
https://github.com/apache/pulsar/issues/22277#issuecomment-2002553745

• Deprecate java label. Pulsar is written in Java and most PRs update Java 
code.
• Instead of removing labels, deprecate them by renaming them to 
deprecated/. Probably pick another prefix that is alphabetically 
closer to the end of the alphabet to reduce noise.
• Add go label automatically using labeler: 
https://github.com/apache/pulsar/blob/master/.github/labeler.yml
go:
- changed-files:
- any-glob-to-any-file: '**/*.go'

• Add component/* labels automatically based on the file path
component/config:
- changed-files:
- any-glob-to-any-file: 'conf/**/*'
- any-glob-to-any-file: 'pulsar-config-validation/**/*'
component/client:
- changed-files:
- any-glob-to-any-file: 'pulsar-client/**/*'
- any-glob-to-any-file: 'pulsar-client-*/**/*'
...

• Rename bug label to type/bug for consistency. Keep the red color.
• (?) Rename component/* => area/* for shorter names. The 
https://github.com/kubernetes/kubernetes/labels has such naming.
• Rename doc-required label to type/doc. Relabel open issues and PRs with 
doc labels to the type/doc.
• Deprecate all other doc-* labels. If it is needed for some kind of 
workflow, simply use the board project with ToDo -> In Progress -> Done states.
• (?) Probably it makes sense to enable and track website and docs issues 
in apache/pulsar-site repository. And add a good visible link to apache/pulsar 
README.md.
• Deprecate the question label. Instead, move such issues to Discussions -> 
Q
• Migrate issues with the enhancement either to type/feature label or 
Discussions. Add a new Suggest an idea issue template that redirects to the 
Discussions -> Ideas
• (?) Rename PIP => type/PIP for consistency
• Rename flaky-test => type/flaky-test to consistency
• Deprecate lifecycle/stale label. Use Stale instead. Rename Stale => stale 
for consistency.
• Add the ability to pick an area/* label from the dropdown on issue 
creation.
systemd/systemd and a few other projects use this action for that: 
https://github.com/redhat-plumbers-in-action/advanced-issue-labeler?tab=readme-ov-file#real-life-examples


Best,
Kiryl

Re: [DISCUSS] Clarify the relation between supported Pulsar versions and versioned docs

[Kiryl Valkovich]

I submitted the PR https://github.com/apache/pulsar-site/pull/843


Best,
Kiryl

> On Mar 10, 2024, at 10:02 PM, Kiryl Valkovich  wrote:
> 
> Asaf and Dave, thank you for your responses.
> 
> The proposed version menu  structure  looks good to me:
> - Next
> - active versions still in support
> - Others
> 
> If there will be no more input, I’m going to submit the PR in the upcoming 
> week.
> 
> I updated the redirect and the release process instructions. Please review 
> the PR:
> https://github.com/apache/pulsar-site/pull/836
> 
> 
> Best,
> Kiryl
> 
>> On Mar 10, 2024, at 9:06 PM, Dave Fisher  wrote:
>> 
>> I like the notion of not updating documentation for versions that are out of 
>> EOL unless the documentation is incorrect.
>> 
>> The only versions that should be on the documentation pop-up on 
>> https:/pulsar.apache.org/docs/x,y,z/ <http://pulsar.apache.org/docs/x,y,z/> 
>> should be:
>> - Next
>> - active versions still in support
>> - Others 
>> 
>> Should the PMC release a version out of support like is happening with 
>> 2.10.x and 2.11.x. Documentation may be updated, but it should remain 
>> “Others”
>> 
>> Also, please look into the redirect on https://pulsar.apache.org/docs/ 
>> <https://pulsar.apache.org/docs/> is currently redirecting to 
>> https://pulsar.apache.org/docs/3.1.x/ 
>> <https://pulsar.apache.org/docs/3.1.x/> is that what is desired?
>> 
>> Best,
>> Dave
>> 
>>> On Mar 6, 2024, at 3:56 AM, Kiryl Valkovich  wrote:
>>> 
>>> Idea: don't require updating versioned docs from contributors.
>>> Making a small documentation fix in a single place is easy, but if we ask 
>>> contributors to fix it in 5-10 places, it may prevent the initiative at all.
>>> 
>>> It could increase the amount of contributions to the documentation.
>>> 
>>> I'm not sure how to better organize this process. Who should do this job - 
>>> the PR reviewer or someone else like a technical writer?
>>> 
>>> 
>>> Best,
>>> Kiryl
>>> 
>>>> On Mar 5, 2024, at 12:15 AM, Kiryl Valkovich  wrote:
>>>> 
>>>> The release policy page states that Pulsar has two supported versions on 
>>>> the current date.
>>>> The documentation site provides four versions to choose from in the 
>>>> dropdown list. If some of them aren't actively supported, should they also 
>>>> be updated?
>>>> 
>>>> GitHub issue with more details and screenshots: 
>>>> https://github.com/apache/pulsar/issues/22177
>>>> 
>>>> 
>>>> Best,
>>>> Kiryl
>>> 
>>> 
>> 
> 

Re: [DISCUSS] Clarify the relation between supported Pulsar versions and versioned docs

[Kiryl Valkovich]

Asaf and Dave, thank you for your responses.

The proposed version menu  structure  looks good to me:
- Next
- active versions still in support
- Others

If there will be no more input, I’m going to submit the PR in the upcoming week.

I updated the redirect and the release process instructions. Please review the 
PR:
https://github.com/apache/pulsar-site/pull/836


Best,
Kiryl

> On Mar 10, 2024, at 9:06 PM, Dave Fisher  wrote:
> 
> I like the notion of not updating documentation for versions that are out of 
> EOL unless the documentation is incorrect.
> 
> The only versions that should be on the documentation pop-up on 
> https:/pulsar.apache.org/docs/x,y,z/ <http://pulsar.apache.org/docs/x,y,z/> 
> should be:
> - Next
> - active versions still in support
> - Others 
> 
> Should the PMC release a version out of support like is happening with 2.10.x 
> and 2.11.x. Documentation may be updated, but it should remain “Others”
> 
> Also, please look into the redirect on https://pulsar.apache.org/docs/ 
> <https://pulsar.apache.org/docs/> is currently redirecting to 
> https://pulsar.apache.org/docs/3.1.x/ <https://pulsar.apache.org/docs/3.1.x/> 
> is that what is desired?
> 
> Best,
> Dave
> 
>> On Mar 6, 2024, at 3:56 AM, Kiryl Valkovich  wrote:
>> 
>> Idea: don't require updating versioned docs from contributors.
>> Making a small documentation fix in a single place is easy, but if we ask 
>> contributors to fix it in 5-10 places, it may prevent the initiative at all.
>> 
>> It could increase the amount of contributions to the documentation.
>> 
>> I'm not sure how to better organize this process. Who should do this job - 
>> the PR reviewer or someone else like a technical writer?
>> 
>> 
>> Best,
>> Kiryl
>> 
>>> On Mar 5, 2024, at 12:15 AM, Kiryl Valkovich  wrote:
>>> 
>>> The release policy page states that Pulsar has two supported versions on 
>>> the current date.
>>> The documentation site provides four versions to choose from in the 
>>> dropdown list. If some of them aren't actively supported, should they also 
>>> be updated?
>>> 
>>> GitHub issue with more details and screenshots: 
>>> https://github.com/apache/pulsar/issues/22177
>>> 
>>> 
>>> Best,
>>> Kiryl
>> 
>> 
> 

Re: [DISCUSS] Clarify the relation between supported Pulsar versions and versioned docs

[Kiryl Valkovich]

Idea: don't require updating versioned docs from contributors.
Making a small documentation fix in a single place is easy, but if we ask 
contributors to fix it in 5-10 places, it may prevent the initiative at all.

It could increase the amount of contributions to the documentation.

I'm not sure how to better organize this process. Who should do this job - the 
PR reviewer or someone else like a technical writer?


Best,
Kiryl

> On Mar 5, 2024, at 12:15 AM, Kiryl Valkovich  wrote:
> 
> The release policy page states that Pulsar has two supported versions on the 
> current date.
> The documentation site provides four versions to choose from in the dropdown 
> list. If some of them aren't actively supported, should they also be updated?
> 
> GitHub issue with more details and screenshots: 
> https://github.com/apache/pulsar/issues/22177
> 
> 
> Best,
> Kiryl

[DISCUSS] Clarify the relation between supported Pulsar versions and versioned docs

[Kiryl Valkovich]

The release policy page states that Pulsar has two supported versions on the 
current date.
The documentation site provides four versions to choose from in the dropdown 
list. If some of them aren't actively supported, should they also be updated?

GitHub issue with more details and screenshots: 
https://github.com/apache/pulsar/issues/22177


Best,
Kiryl

[DISCUSS] Add more actionable elements to the site homepage

[Kiryl Valkovich]

• Call to star Pulsar on GitHub ⭐️
• Add a few lines of code that show how to launch Pulsar
• Call to join Slack

GitHub Discussion with example screenshots and description: 
https://github.com/apache/pulsar/discussions/22189


Best,
Kiryl

Idea: Add a "From zero to hero" contributor roadmap article

[Kiryl Valkovich]

Imagine that you are a Pulsar user or interested in data streaming in general 
and would like to contribute to some related open-source project.
You have some prior programming experience but you've worked on other kinds of 
projects and never developed distributed streaming systems.
The current documentation doesn't describe the prior knowledge list recommended 
to contribute to Pulsar.
Having a clear roadmap of what to learn and where to start would bring more 
contributors to the project.
It may have a similar format:
•  Java
Recommended resources:
...
•  Concurrency and Parallelism
Recommended resources:
...
•  Networking
Recommended resources:
...
•  Data streaming
Recommended resources:
...
•  Cyber-security
Recommended resources:
...
•  Storage and Bookkeeper
Recommended resources:
...
•  Pulsar internals
Recommended resources:
...
•  Important libraries to know
•  Netty
•  Jetty
•  Lombok
...
•  Issue list for starting contributing
...
Of course, it takes time to learn these things and can add some additional load 
on the PR reviewers' shoulders at first. But it also could help to grow a new 
wave of contributors who are loyal to the project from the beginning, which is 
important for growing the Pulsar contributors community.

Discussion on GitHub: https://github.com/apache/pulsar/discussions/22176


Best,
Kiryl

Re: [ANNOUNCE] New Committer: Asaf Mesika

[Kiryl Valkovich]

Congratulations, Asaf!

From: Lari Hotari 
Sent: Tuesday, February 20, 2024 5:50 PM
To: dev@pulsar.apache.org 
Subject: [ANNOUNCE] New Committer: Asaf Mesika

The Apache Pulsar Project Management Committee (PMC) has invited
Asaf Mesika https://github.com/asafm to become a committer and we
are pleased to announce that he has accepted.

Welcome and Congratulations, Asaf Mesika!

Please join us in congratulating and welcoming Asaf onboard!

Best Regards,

Lari Hotari
on behalf of the Pulsar PMC

Re: WIKIPEDIA PAGE

[Kiryl Valkovich]

Hi Brolly,

The last time you and I discussed it in December, you suggested submitting it 
into the Simple English section and wanted to charge for that. In my personal 
opinion, Pulsar deserves an article on a regular English Wikipedia, not in the 
Simple English section. Many smaller ASF projects have such articles.

I still hope that someone from the community will take the initiative and write 
such an article that will pass all the Wiki's requirements like lack of 
personal interest and sufficient amount of references to reliable independent 
resources.


Best,
Kiryl

From: Brolly 
Sent: Saturday, February 17, 2024 11:58 AM
To: Dev 
Subject: WIKIPEDIA PAGE

Hi,

I hope you are well. This is Brolly, I am a Wikipedia moderator.

I just saw your Wikipedia page on the Wikipedia page rejection list.

I would like to offer my assistance in helping you successfully publish
your Wikipedia page.

If you are interested in getting your Wikipedia page done please let me
know.

I look forward to hearing from you.

Thanks

Brolly
[image: beacon]

Re: WIKIPEDIA PAGE NEEDED

[Kiryl Valkovich]

Hi Brolly,

You’re probably talking about my attempt to create an article about Apache 
Pulsar this November.

I’m sure that the Pulsar and the broader data streaming communities would be 
very grateful if you could help.

What would be the next steps for us to take?

Sent a duplicate to your email.


Best,
Kiryl

From: Brolly 
Date: Saturday, December 16, 2023 at 11:47 PM
To: dev@pulsar.apache.org 
Subject: WIKIPEDIA PAGE NEEDED
Hi,

I hope you are well. This is Brolly, I am a Wikipedia moderator.

I just saw your Wikipedia page Apache Pulsar on the Wikipedia page
rejection list.

I would like to offer my assistance in helping you successfully publish
your Wikipedia page.

If you are interested in getting your Wikipedia page done please let me
know.

I look forward to hearing from you.

Thanks

Brolly

Re: [ANNOUNCE] Apache Pulsar 3.0.2 released

[Kiryl Valkovich]

Hi, Zike

Thanks for fixing it.

One more thing to fix. v3.0.2 release is missing on the Downloads page. I 
raised one more issue here:
https://github.com/apache/pulsar/issues/21694


From: Zike Yang 
Date: Wednesday, December 6, 2023 at 1:08 PM
To: dev@pulsar.apache.org 
Subject: Re: [ANNOUNCE] Apache Pulsar 3.0.2 released
Hi, Kiryl

Thanks for your catch. I have submitted a PR to fix it:
https://github.com/apache/pulsar-site/pull/746 PTAL.

Thanks,
Zike Yang

On Wed, Dec 6, 2023 at 4:41 PM Kiryl Valkovich
 wrote:
>
> Congrats on the release!
>
> Small fixes in the Release table are needed. I raised the issue here:
> https://github.com/apache/pulsar/issues/21677
>
>
> Best,
> Kiryl
>
> From: Yubiao Feng 
> Date: Wednesday, December 6, 2023 at 8:17 AM
> To: dev@pulsar.apache.org , us...@pulsar.apache.org 
> , annou...@apache.org 
> Subject: [ANNOUNCE] Apache Pulsar 3.0.2 released
> The Apache Pulsar team is proud to announce Apache Pulsar version 3.0.2.
>
> Pulsar is a highly scalable, low-latency messaging platform running on
> commodity hardware. It provides simple pub-sub semantics over topics,
> guaranteed at-least-once delivery of messages, automatic cursor management
> for
> subscribers, and cross-datacenter replication.
>
> For Pulsar release details and downloads, visit:
>
> https://pulsar.apache.org/download
>
> Release Notes are at:
> https://pulsar.apache.org/release-notes
>
> We would like to thank the contributors that made the release possible.
>
> Regards,
>
> The Pulsar Team

Re: [ANNOUNCE] Apache Pulsar 3.0.2 released

[Kiryl Valkovich]

Congrats on the release!

Small fixes in the Release table are needed. I raised the issue here:
https://github.com/apache/pulsar/issues/21677


Best,
Kiryl

From: Yubiao Feng 
Date: Wednesday, December 6, 2023 at 8:17 AM
To: dev@pulsar.apache.org , us...@pulsar.apache.org 
, annou...@apache.org 
Subject: [ANNOUNCE] Apache Pulsar 3.0.2 released
The Apache Pulsar team is proud to announce Apache Pulsar version 3.0.2.

Pulsar is a highly scalable, low-latency messaging platform running on
commodity hardware. It provides simple pub-sub semantics over topics,
guaranteed at-least-once delivery of messages, automatic cursor management
for
subscribers, and cross-datacenter replication.

For Pulsar release details and downloads, visit:

https://pulsar.apache.org/download

Release Notes are at:
https://pulsar.apache.org/release-notes

We would like to thank the contributors that made the release possible.

Regards,

The Pulsar Team

Missing Pulsar entry in Apache Projects list page

[Kiryl Valkovich]

The project has an entry here:
https://projects.apache.org/projects.html?name

but hasn't here: https://projects.apache.org/projects.html?language
and here: https://projects.apache.org/projects.html?category

Maybe someone who knows where to fix it can fix it.


Best,
Kiryl

Writing the Pulsar article for Wikipedia. Looking for in-depth, reliable, secondary, independent resources on the subject.

[Kiryl Valkovich]

Hi everyone! I think that Pulsar deserves an article on Wikipedia.

The problem is that it’s surprisingly hard to find something in-depth, 
reliable, secondary, and independent on the subject.
Most of the good information is in Pulsar documentation, StreamNative blog, 
DataStax blog, Apache Foundation resources, and on other blogs mostly from 
authors who are now or have been employees in companies that are interested in 
Pulsar's success.
By Wikipedia rules all such sources are not “independent”.
Any info from the documentation isn’t “secondary”.
Talks on YouTube aren’t “reliable” sources.

I pointed the moderator to the https://en.wikipedia.org/wiki/Apache_Kafka 
article has about the same quality of sources. The answer was:

  *   I’ve marked that article as needing assistance, as it has similar 
problems to your draft.

Maybe someone can point me to such resources?

If you have experience writing articles for Wikipedia and know its rules and 
how to better deal with moderators, I won't refuse help.

Draft of the article. It became quite brief after all the edits: 
https://en.wikipedia.org/wiki/Draft:Apache_Pulsar
Talk: https://en.wikipedia.org/wiki/User_talk:Visortelle


Best,
Kiryl

Re: Question about Pulsar gRPC client(s)

[Kiryl Valkovich]

  *   We could have less errors by adopting an API-first approach (generate
server stubs from the OpenAPI specs).
With Protobuf + gRPC I would be all hands up for the API-first approach.
Does code generation work well with OpenAPI? I tried some Swagger => code 
generators a while ago for Node.js + TypeScript, and Go, and they didn’t work 
well. Maybe it’s a better situation now, or better for Java.

IMHO, due to OpenAPI yaml syntax is much more verbose than proto3, it’s harder 
to write and read.
Hard to write + people are lazy = API design suffers in some cases: 0, 1, -1, 
null special values instead of explicit human-understandable types, etc.


  *   simpler to deal with, usable with simple CLI tools, browsers,
etc...
Yes, simpler. Also, it's much simpler to make mistakes as a client, and much 
harder to maintain overall, and backward compatibility in particular as an API 
developer. ;)
With grpc-web or newer connectrpc by buf.build, you already can use gRPC in a 
browser.
Tooling is also here: grpc-ui and grcpcurl.


  *   and it cannot be embedded in the broker
as it's written in Go.
It may be tricky, but it’s possible. Because it’s easy to cross-compile 
anything written in Go, and a result is a small static binary, you can put it 
with the app distro, detect an os and arch on start, write the binary to the 
user cache dir (if not exists), and then run as a forked process. Cgroup will 
remain the same, as for the parent process. gRPC socket <--mTLS--> gateway 
socket. Am I missing something? Maybe zombies? ;)

I understand that unlikely will be some movement in this direction at this 
moment, just shared some thoughts.


  *   Yes. A gRPC proxy as an alternative to the WS proxy makes sense. It
could be developed as a proxy extension.
I am all hands up for having it! Maybe anyone else?


Best,
Kiryl


From: Christophe Bornet 
Date: Wednesday, November 1, 2023 at 10:33 PM
To: dev@pulsar.apache.org 
Subject: Re: Question about Pulsar gRPC client(s)
Le mer. 1 nov. 2023 à 05:16, Kiryl Valkovich
 a écrit :
>
>   *   So I think we could not replace completely the Pulsar protocol by gRPC.
>
> Sure, I understand that it’s unlikely possible to implement a full 
> replacement.
> I’m rather talking about it as a better, more type-safe alternative to 
> WebSocket and REST APIs, with a similar amount of supported features.
>
> https://pulsar.apache.org/docs/3.1.x/client-libraries-websocket/
> https://pulsar.apache.org/docs/3.1.x/client-libraries-rest/
>
> I’m not talking about the performance at all. I think, in the worst case it 
> should be at least the same performance as the WebSocket protocol 
> implementation.
> From my understanding, it should be simpler to maintain it than REST or 
> WebSocket protocols due to type safety, isn’t it?
>

Yes. A gRPC proxy as an alternative to the WS proxy makes sense. It
could be developed as a proxy extension.

>
>   *   You have some kind of type hinting with the
> OpenAPI spec that you can use to generate client SDKs (eg. with
> openapi-generator.
> Sure, but openapi + Java annotations by its nature are very error-prone. 
> About a year ago when I needed to use Pulsar Admin API via REST, I reported 4 
> bugs about wrong fields in the response object and wrong status codes. I 
> suspect there are many more such errors.
>
> The current approach I see in the following way:
> We use strongly typed Java => then we remove all type information and 
> re-define it semi-manually via OpenAPI annotations (and introduce tons of 
> errors) => then again trying to make it strongly typed by running OpenAPI 
> client generator.
>
> I’d prefer the approach where it’s easy and safe to generate a client for any 
> language without bugs introduced by Java API => REST => OpenAPI client 
> conversions.

We could have less errors by adopting an API-first approach (generate
server stubs from the OpenAPI specs). The source of truth for the API
is then the OpenAPI spec. This wouldn't help with wrong status codes
but you would have the same issue with gRPC anyway.

>
> In rare cases where REST is still needed (its bash scripts mostly, or am I 
> missing anything?), it’s not hard to run/embed 
> https://github.com/grpc-ecosystem/grpc-gateway
>
> I think it’s much easier to implement it once in Java, and then generate a 
> client for target languages. The amount of work should be about the same as 
> implementing it for any other language, like here 
> https://github.com/apache/pulsar-client-go/tree/master/pulsaradmin/pkg/admin
> The difference is that this way you don’t need to re-implement it for any 
> other language anymore.
>

I like gRPC a lot, especially for efficient bidirectional-streaming
communication. But for simple unary RPC where performance is not
required, the world has chosen HTTP+JSON for a variety of reasons
(simpler to d

Re: Question about Pulsar gRPC client(s)

[Kiryl Valkovich]

  *   So I think we could not replace completely the Pulsar protocol by gRPC.

Sure, I understand that it’s unlikely possible to implement a full replacement.
I’m rather talking about it as a better, more type-safe alternative to 
WebSocket and REST APIs, with a similar amount of supported features.

https://pulsar.apache.org/docs/3.1.x/client-libraries-websocket/
https://pulsar.apache.org/docs/3.1.x/client-libraries-rest/

I’m not talking about the performance at all. I think, in the worst case it 
should be at least the same performance as the WebSocket protocol 
implementation.
From my understanding, it should be simpler to maintain it than REST or 
WebSocket protocols due to type safety, isn’t it?


  *   You have some kind of type hinting with the
OpenAPI spec that you can use to generate client SDKs (eg. with
openapi-generator.
Sure, but openapi + Java annotations by its nature are very error-prone. About 
a year ago when I needed to use Pulsar Admin API via REST, I reported 4 bugs 
about wrong fields in the response object and wrong status codes. I suspect 
there are many more such errors.

The current approach I see in the following way:
We use strongly typed Java => then we remove all type information and re-define 
it semi-manually via OpenAPI annotations (and introduce tons of errors) => then 
again trying to make it strongly typed by running OpenAPI client generator.

I’d prefer the approach where it’s easy and safe to generate a client for any 
language without bugs introduced by Java API => REST => OpenAPI client 
conversions.

In rare cases where REST is still needed (its bash scripts mostly, or am I 
missing anything?), it’s not hard to run/embed 
https://github.com/grpc-ecosystem/grpc-gateway

I think it’s much easier to implement it once in Java, and then generate a 
client for target languages. The amount of work should be about the same as 
implementing it for any other language, like here 
https://github.com/apache/pulsar-client-go/tree/master/pulsaradmin/pkg/admin
The difference is that this way you don’t need to re-implement it for any other 
language anymore.

Best,
Kiryl

From: Christophe Bornet 
Date: Wednesday, November 1, 2023 at 12:43 AM
To: dev@pulsar.apache.org 
Subject: Re: Question about Pulsar gRPC client(s)
Hi Kiryl,

Thanks for mentioning pulsar-grpc.
Indeed, using gRPC simplifies the implementation of the networking logic
(keep-alive, reconnection, flow control,…). On the other hand, the Java
gRPC implementation makes a lot of buffer copies to cleanly separate the
network and app layers but that takes a toll on performance. Compared to
that, the broker Pulsar protocol impl is optimized to not do copies between
the consumer/producer endpoints and the bookkeeper client.
So I think we could not replace completely the Pulsar protocol by gRPC.
We could maybe support both but it’s a lot of work to maintain both
protocols. (I kind of gave up maintaining pulsar-grpc because of the amount
of work compared to the number of users, but if there’s interest I can
reconsider).
Another possibility would be to do a proxy instead of a low-level protocol
handler. A bit like the WebSocket proxy. This would be far less work to
maintain as it would use the Pulsar protocol to communicate with the
brokers. It could be done as a Proxy extension. Compared to the WS proxy,
this would provide type safety, discovery, and so on…
As for the Admin, it’s a bit the same. It would be a bunch of work to
support both gRPC and REST. You have some kind of type hinting with the
OpenAPI spec that you can use to generate client SDKs (eg. with
openapi-generator.
I wonder what others have to say.

Christophe


Le mar. 31 oct. 2023 à 19:57, Kiryl Valkovich 
a écrit :

> Hi! Am I understanding it right, that if this project
> https://github.com/cbornet/pulsar-grpc is merged to the apache/pulsar
> repo, we could easily cover non-mainstream platforms that are supported by
> gRPC, but don't have ready-to-use Pulsar clients?
>
> https://github.com/apache/pulsar/wiki/PIP-59:-gPRC-Protocol-Handler
>
> Similar to already supported language-agnostic client interfaces - REST
> and WebSocket.
>
> Actively maintained gRPC libraries I found (19, or 15 if considering JVM
> languages and web as duplicates):
> - [C# / .NET](https://grpc.io/docs/languages/csharp/)
> - [C++](https://grpc.io/docs/languages/cpp/)
> - [Dart](https://grpc.io/docs/languages/dart/)
> - [Go](https://grpc.io/docs/languages/go/)
> - [Java](https://grpc.io/docs/languages/java/)
> - [Kotlin](https://grpc.io/docs/languages/kotlin/)
> - [Node](https://grpc.io/docs/languages/node/)
> - [Objective-C](https://grpc.io/docs/languages/objective-c/)
> - [PHP](https://grpc.io/docs/languages/php/)
> - [Python](https://grpc.io/docs/languages/python/)
> - [Ruby](https://grpc.io/docs/languages/ruby/)
> - [OCaml](https://github.com/dialohq/ocaml-grpc)
> - [Haskell](https://github.com/

Question about Pulsar gRPC client(s)

[Kiryl Valkovich]

Hi! Am I understanding it right, that if this project 
https://github.com/cbornet/pulsar-grpc is merged to the apache/pulsar repo, we 
could easily cover non-mainstream platforms that are supported by gRPC, but 
don't have ready-to-use Pulsar clients?

https://github.com/apache/pulsar/wiki/PIP-59:-gPRC-Protocol-Handler

Similar to already supported language-agnostic client interfaces - REST and 
WebSocket.

Actively maintained gRPC libraries I found (19, or 15 if considering JVM 
languages and web as duplicates):
- [C# / .NET](https://grpc.io/docs/languages/csharp/)
- [C++](https://grpc.io/docs/languages/cpp/)
- [Dart](https://grpc.io/docs/languages/dart/)
- [Go](https://grpc.io/docs/languages/go/)
- [Java](https://grpc.io/docs/languages/java/)
- [Kotlin](https://grpc.io/docs/languages/kotlin/)
- [Node](https://grpc.io/docs/languages/node/)
- [Objective-C](https://grpc.io/docs/languages/objective-c/)
- [PHP](https://grpc.io/docs/languages/php/)
- [Python](https://grpc.io/docs/languages/python/)
- [Ruby](https://grpc.io/docs/languages/ruby/)
- [OCaml](https://github.com/dialohq/ocaml-grpc)
- [Haskell](https://github.com/awakesecurity/gRPC-haskell)
- [Elixir](https://github.com/elixir-grpc/grpc)
- [Rust](https://github.com/hyperium/tonic)
- [Scala](https://scalapb.github.io/)
- [Swift](https://github.com/grpc/grpc-swift)
- Web client: https://github.com/grpc/grpc-web
- Web client 2: https://connectrpc.com/docs/web/getting-started

Actively maintained Pulsar libraries (8):
- Java
- C++
- Python
- Go
- Node.js
- C#
- PHP
- Rust

Is there any reason for not merging it to the apache/pulsar?

I would definitely prefer to work with a statically typed gRPC client instead 
of REST or WebSocket.

By the way, the same can work for the Pulsar Admin API. Implement the gRPC 
server once in Java, and we have full-featured native statically typed (where 
applicable :)) Pulsar Admin clients for any platform.

Best,
Kiryl

Re: [DISCUSS] Consistent code style (esp. ws/indent) and autotools

[Kiryl Valkovich]

My mistake. It looks that for some reason Spotless supports .editorconfig only 
for ktlint.

Best,
Kiryl

From: Kiryl Valkovich 
Date: Friday, June 30, 2023 at 6:45 AM
To: dev@pulsar.apache.org 
Subject: Re: [DISCUSS] Consistent code style (esp. ws/indent) and autotools
Hi,

tison, are you going to use .editorconfig for specifying indent rules?

https://editorconfig.org/

It looks like Spotless supports it. 
https://github.com/diffplug/spotless/issues/734

Flink and many other ASF projects use it. 
https://github.com/apache/flink/blob/21eba4ca4cb235a2189c94cdbf3abcec5cde1e6e/.editorconfig
https://github.com/search?q=org%3Aapache%20.editorconfig=code

Unlike Spotless, the .editorconfig works out of the box in most of the modern 
code editors.

Best,
Kiryl

From: tison 
Date: Friday, June 30, 2023 at 3:58 AM
To: Dev 
Subject: [DISCUSS] Consistent code style (esp. ws/indent) and autotools
Hi,

I'd like to propose applying a consistent code style (especially whitespace
and indent) with an autotool Spotless.

// Background

Over and over we argue contributors reformat their patch manually for
checkstyle violations, or even whitespace changes that are not detected by
checkstyle. [1]

A common reason is that such style-only changes increase the burden to do
cherry-pick if a later bug fix is made around the code while we often do
not pick the style change barely or even it's coupled with an unpickable
commit.

CheckStyle helps in some cases while we don't have a strong rule set nor
even apply it to tests (porting bug fix actually include the test).

Manually fixing style violations can be boring and even annoying. That's
why it's effectively "suppressed" in mind.

An autotool to do the formatting work can help and here comes Spotless[2].
Flink and Curator use it and Flink actually migrated from CheckStyle to
Spotless for its more strict consistent rule set and oneliner format. See
their original discussion for the context[3].

// Proprosal

Using Spotless as the auto-formatting tool in all around apache/pulsar.
Since it's an auto tool I can cover the task solely.

// Concerns

1. Won't it be yet another noise to the codebase?

Yes and no. I suggest we pick this change to all maintained versions so
that all of them align with the consistent style. Then from now on, no more
style conflict.

Flink used the same strategy[4] and even we can do it starting from 2.10 as
Lari proposed to apply commits from the least recent maintained version. I
understand the maintained versions are: 2.10, 2.11, 3.0, and master.

2. Can it cover all the rule sets we use in CheckStyle now?

Let's say we almost don't care what the style is but it's consistent and
"not awful".

The default Google style takes line length = 80 which can be a
disappointment so in Curator I use the palantir style[5] which takes line
length = 120 which should keep consistent with current settings.

A downside is that Spotless Maven plugin cannot detect "forbidden imports"
where we can still use CheckStyle[6] - this is a low-traffic path and it
should be fine.

// Implementation

1. Introduce Spotless in the project and apply it around the codebase, for
all maintained versions.
2. Remove the then redundant CheckStyle rules, while retaining the
forbidden imports part as it's still useful.

Looking forward to your feedback!

Best,
tison.

[1]
https://github.com/apache/pulsar/pull/20642/files#diff-cb9b09b67f54fccdd5155dbbcedd50970ee93d9ee85ade1e6b6984cab64dab5d
[2] https://github.com/diffplug/spotless
[3] https://lists.apache.org/thread/3kjkcz9gj6f8j477d1t3gnbkl61hsb7z
[4] https://lists.apache.org/thread/nxdm0pg84q913w0kxszm502myqcg3db0
[5] https://github.com/palantir/palantir-java-format
[6] https://issues.apache.org/jira/browse/FLINK-32154

Re: [DISCUSS] Consistent code style (esp. ws/indent) and autotools

[Kiryl Valkovich]

Hi,

tison, are you going to use .editorconfig for specifying indent rules?

https://editorconfig.org/

It looks like Spotless supports it. 
https://github.com/diffplug/spotless/issues/734

Flink and many other ASF projects use it. 
https://github.com/apache/flink/blob/21eba4ca4cb235a2189c94cdbf3abcec5cde1e6e/.editorconfig
https://github.com/search?q=org%3Aapache%20.editorconfig=code

Unlike Spotless, the .editorconfig works out of the box in most of the modern 
code editors.

Best,
Kiryl

From: tison 
Date: Friday, June 30, 2023 at 3:58 AM
To: Dev 
Subject: [DISCUSS] Consistent code style (esp. ws/indent) and autotools
Hi,

I'd like to propose applying a consistent code style (especially whitespace
and indent) with an autotool Spotless.

// Background

Over and over we argue contributors reformat their patch manually for
checkstyle violations, or even whitespace changes that are not detected by
checkstyle. [1]

A common reason is that such style-only changes increase the burden to do
cherry-pick if a later bug fix is made around the code while we often do
not pick the style change barely or even it's coupled with an unpickable
commit.

CheckStyle helps in some cases while we don't have a strong rule set nor
even apply it to tests (porting bug fix actually include the test).

Manually fixing style violations can be boring and even annoying. That's
why it's effectively "suppressed" in mind.

An autotool to do the formatting work can help and here comes Spotless[2].
Flink and Curator use it and Flink actually migrated from CheckStyle to
Spotless for its more strict consistent rule set and oneliner format. See
their original discussion for the context[3].

// Proprosal

Using Spotless as the auto-formatting tool in all around apache/pulsar.
Since it's an auto tool I can cover the task solely.

// Concerns

1. Won't it be yet another noise to the codebase?

Yes and no. I suggest we pick this change to all maintained versions so
that all of them align with the consistent style. Then from now on, no more
style conflict.

Flink used the same strategy[4] and even we can do it starting from 2.10 as
Lari proposed to apply commits from the least recent maintained version. I
understand the maintained versions are: 2.10, 2.11, 3.0, and master.

2. Can it cover all the rule sets we use in CheckStyle now?

Let's say we almost don't care what the style is but it's consistent and
"not awful".

The default Google style takes line length = 80 which can be a
disappointment so in Curator I use the palantir style[5] which takes line
length = 120 which should keep consistent with current settings.

A downside is that Spotless Maven plugin cannot detect "forbidden imports"
where we can still use CheckStyle[6] - this is a low-traffic path and it
should be fine.

// Implementation

1. Introduce Spotless in the project and apply it around the codebase, for
all maintained versions.
2. Remove the then redundant CheckStyle rules, while retaining the
forbidden imports part as it's still useful.

Looking forward to your feedback!

Best,
tison.

[1]
https://github.com/apache/pulsar/pull/20642/files#diff-cb9b09b67f54fccdd5155dbbcedd50970ee93d9ee85ade1e6b6984cab64dab5d
[2] https://github.com/diffplug/spotless
[3] https://lists.apache.org/thread/3kjkcz9gj6f8j477d1t3gnbkl61hsb7z
[4] https://lists.apache.org/thread/nxdm0pg84q913w0kxszm502myqcg3db0
[5] https://github.com/palantir/palantir-java-format
[6] https://issues.apache.org/jira/browse/FLINK-32154

[Site] New "How does Pulsar work" homepage section

[Kiryl Valkovich]

Hi, Pulsar developers’ community!



At the recent Pulsar Virtual Summit Europe 2023, we introduced the major 
pulsar.apache.org site redesign.

We believe that it makes a good first impression of the Apache Pulsar and helps 
to attract more users and contributors to the project in the long term.



In the next few days, we are going to introduce a new "How does Pulsar work" 
section to the homepage.



Beautiful informative illustration designed by Emidio Cardeira gives a 
high-level understanding of Pulsar building blocks and their relations.

Under the illustration, we placed a brief description purpose of each of the 
building blocks.



Live demo: https://pulsar-site-git-how-pulsar-work-screen-teal-tools.vercel.app/

PR: https://github.com/apache/pulsar-site/pull/614



Thanks Asaf Mesika for curating the site redesign project and tison for 
reviewing the PRs.



Currently, we are working on new Pulsar Community and Pulsar Features pages.

You can track the design progress and leave your comments in Figma:



- 
https://www.figma.com/file/0jERoA2DTlfQoj1uH09FxQ/Pulsar-website?node-id=368-658=8e5jjfIqCrgtRy9w-0



- 
https://www.figma.com/file/0jERoA2DTlfQoj1uH09FxQ/Pulsar-website?type=design=1223-1864=aoWstTjkAYfo5Top-0



Also, if you want to share your thoughts on content, and how to show and 
explain Pulsar concepts in an easy to understand way, please do it in this 
mailing list or join the #pip_249 channel in the Apache Pulslar community Slack.



Best,

Kiryl

Re: [pulsar-site] branch main updated: fix revert to use the latest stable version by default

[Kiryl Valkovich]

I submitted a PR that fixes both:
- link to the latest version
- two highlighted menu items at the same time, that was the original reason for 
the changes in docusaurus.config.json

https://github.com/apache/pulsar-site/pull/587


Best,
Kiryl


From: tison 
Date: Friday, May 26, 2023 at 5:02 AM
To: dev@pulsar.apache.org 
Subject: Re: [pulsar-site] branch main updated: fix revert to use the latest 
stable version by default
This is actually the manner before we merge the redesign PR - I miss this
part when reviewing the redesign PR >_<

Best,
tison.


Dave Fisher  于2023年5月26日周五 10:54写道：

> Hi -
>
> I like this change!
>
> Thanks,
> Dave
>
> Sent from my iPhone
>
> > On May 25, 2023, at 7:36 PM, ti...@apache.org wrote:
> >
> > This is an automated email from the ASF dual-hosted git repository.
> >
> > tison pushed a commit to branch main
> > in repository https://gitbox.apache.org/repos/asf/pulsar-site.git
> >
> >
> > The following commit(s) were added to refs/heads/main by this push:
> > new b30efcce68c fix revert to use the latest stable version by
> default
> > b30efcce68c is described below
> >
> > commit b30efcce68ceeda50f960cec2a22fe0f1194d233
> > Author: tison 
> > AuthorDate: Fri May 26 10:36:33 2023 +0800
> >
> >fix revert to use the latest stable version by default
> >
> >Signed-off-by: tison 
> > ---
> > docusaurus.config.js | 8 
> > 1 file changed, 4 insertions(+), 4 deletions(-)
> >
> > diff --git a/docusaurus.config.js b/docusaurus.config.js
> > index d6484c7c78a..9eba10a8727 100644
> > --- a/docusaurus.config.js
> > +++ b/docusaurus.config.js
> > @@ -173,13 +173,13 @@ module.exports = {
> >   position: "left",
> >   items: [
> > {
> > -  to: "/docs/next/concepts-overview/",
> > -  activeBaseRegex: "docs/next/concepts-overview/$",
> > +  type: 'doc',
> > +  docId: 'concepts-overview',
> >   label: "Concepts",
> > },
> > {
> > -  to: "/docs/next/",
> > -  activeBaseRegex: "docs/next/$",
> > +  type: 'doc',
> > +  docId: 'about',
> >   label: "Quickstart",
> > },
> > {
> >
>
>

Re: Question about Apache Foundation content policies on Pusar website

[Kiryl Valkovich]

Enrico, thank you for your response!
Glad to hear that.

I plan to start the work on these pages this weekend.
If anyone has anything to add, please let me know before that.

Best,
Kiryl


From: Enrico Olivelli 
Date: Thursday, May 25, 2023 at 1:19 PM
To: Dev 
Subject: Re: Question about Apache Foundation content policies on Pusar website
Kiryl,

Il Gio 25 Mag 2023, 13:00 Kiryl Valkovich  ha
scritto:

> Hi,
> I’d like to add information about available trainings, courses, and
> service providers to the Pulsar website. https://pulsar.apache.org/
>
> Like what Kubernetes do at Linux Foundation:
> https://kubernetes.io/training/
> https://kubernetes.io/partners/
>
> The availability of educational programs and managed solutions make it
> much easier to start with new technology.
> I believe that showing it on the official site would lead to better Pulsar
> user acquisition.
>
> Would like to learn if Apache Foundation has some policies on such content
> and where to read more about it.
>

I think that it is mostly up to the community to accept this kind of links,
that may read as advertisement.

There are some Apache project that have them.


Enrico



> Best,
> Kiryl
>
>

Question about Apache Foundation content policies on Pusar website

[Kiryl Valkovich]

Hi,
I’d like to add information about available trainings, courses, and service 
providers to the Pulsar website. https://pulsar.apache.org/

Like what Kubernetes do at Linux Foundation:
https://kubernetes.io/training/
https://kubernetes.io/partners/

The availability of educational programs and managed solutions make it much 
easier to start with new technology.
I believe that showing it on the official site would lead to better Pulsar user 
acquisition.

Would like to learn if Apache Foundation has some policies on such content and 
where to read more about it.

Best,
Kiryl

Look at the fresh Apache Pulsar website!

[Kiryl Valkovich]

Hi,

In PIP-249<https://github.com/apache/pulsar/issues/19611>, Asaf Mesika proposed 
the Pulsar website redesign plan.
In short, the point is that the website should reflect that Pulsar is a modern 
and well-maintained technology.

Emidio Cardeira prepared the home page design, and we at Teal 
Tools<https://teal.tools/> took care of the coding part.
Big thanks to tison for the PR review and technical support.

The current changes affect only the visual look.
Further site content improvements will be done iteratively. I invite you to 
look at PIP-261<https://github.com/apache/pulsar/issues/19912> by Asaf.

Feel free to express your thoughts and tell us about some bugs.
The PR will be merged in the next few days.

Links:
- Demo: https://pulsar-site-pip-249.vercel.app/
- PIP-249: https://github.com/apache/pulsar/issues/19611
- PIP-249 PR: https://github.com/apache/pulsar-site/pull/560


Best regards
--
Kiryl Valkovich
Teal Tools Inc.

Re: [DISCUSS] PIP-249: Pulsar website redesign

[Kiryl Valkovich]

Asaf, okay. I see you have some plan.

I’m just a bit confused with the steps ordering.
Usually, such work starts with “What content do we want to show?” That implies 
content changes.
Then “How can we display this content in the best way?”. That implies the work 
on visual design and content adjustment.
Of course, this order is not mandatory, but it usually allows to save time on 
the number of iterations.
The “Homepage structure” point is defined in the PIP’s scope, so I thought it 
meant content changing is implied at this step.
Waiting for the finished site. :)
By the way, does someone have any more opinions on the topic?

From: Asaf Mesika 
Date: Friday, February 24, 2023 at 3:06 PM
To: dev@pulsar.apache.org 
Subject: Re: [DISCUSS] PIP-249: Pulsar website redesign
I totally agree with your point Kyril, and as I wrote in the welcome email
- I want to do all you wrote, but it has to be in bite size pieces. This
piece is about theme and general look and feel. We "force" ourselves, by
design, to avoid any content change - just like in code in this proposal -
so people can focus only on the theme change. It's like in code - you take
an epic, break it down to stories, and each is broken down to small PRs.

So yes, I have a pipeline of content changes:
* Landing page:
** "Tell me about the pulsar community" section - containing exactly what
you wanted. We gather great statistics that show the growing community in
nice numbers, which can also feature company names
** "Use cases and applications" - so people can know how Pulsar is used by
companies (small and big)
and more.

* Documentation:
   As you probably know - the documentation itself is an encyclopedia, so I
planned next week a PIP presenting a new structure (super high level) and
presenting in detail the first section I'd like to focus on: Quick-start
Guide.

Each such change will be open for discussion to make it the best possible
addition to the site.

I'm happy you liked the look and feel (design) proposed (Emidio is happy
too).


On Fri, Feb 24, 2023 at 10:21 AM Kiryl Valkovich 
wrote:

> While I find the proposed visual changes okay, I'm unsure about the
> homepage structure.
>
> I'd add one obvious point to the Goal section that can be formulated
> differently, but eventually is:
> - **The goal is to get more companies to start using Pulsar.**
>
> In other words, the landing page should **sell** the project to potential
> users.
>
> The defined scope of this PIP includes the "Homepage structure" point. I
> will try to express my thoughts on this.
>
> ## Add "Trusted by" or "Used in production" section
>
> One of the best hooks to attract new customers is to show that their peers
> or more prominent players use it.
>
> This is why I think it's essential to show the list of companies that use
> Pulsar as closer to the page top, as possible.
>
> *Kafka in some form has this info on the first screen*
>  https://user-images.githubusercontent.com/9302460/221116492-b77cb8e9-eca1-46bd-b457-6889f4e708b6.png
> >
>
> *Confluent has it on the second screen*
>  https://user-images.githubusercontent.com/9302460/221114783-4f22394e-d1fd-4640-8264-3b137f44e2a1.png
> >
>
> *StreamNative has it on the second screen*
>  https://user-images.githubusercontent.com/9302460/221116665-6749212f-749f-4221-b175-07261a6d2ed5.png
> >
>
> *Redpanda has it on the second screen*
>  https://user-images.githubusercontent.com/9302460/221118114-aabaee99-f6ef-4586-b7df-02c5ca3b24a6.png
> >
>
> The homepage structure proposed in the PIP doesn't have a list of the
> companies on the page itself. Instead, it has a testimonials list at the
> **bottom** of the page with a link to case-studies.
>
> ## Too much text on the second screen
>
> Users don't like to read long texts on landing pages. They'll just skip it.
> Show the same or similar text in a bullet-point-like style would be better.
>
>  https://user-images.githubusercontent.com/9302460/221119160-5a39c7c9-ad68-4332-961a-e941f3a76693.png
> >
>
> ## The Pulsar Features section
>
> Questions the potential user implicitly asks when they scroll the landing
> page sound like:
>
> > 樂 How can this project help me to solve my pain points?
> > 樂 Why should I "buy" this?
>
> The answers to this question aren't limited to the technical features list.
>
> The current PIP doesn't seem to answer these questions.
>
> ---
>
> ## Alternative page structure proposal
>
> 陋 Here it is: https://pulsar-site-three.vercel.app/
>
> I would be happy to work with the designer if most users prefer the PIP's
> visual style.
>
> My goal is not to compete here "just for fun". My company works on a
> product for Pulsar users, so I'm in some ki

Re: [DISCUSS] PIP-249: Pulsar website redesign

[Kiryl Valkovich]

While I find the proposed visual changes okay, I'm unsure about the homepage 
structure.

I'd add one obvious point to the Goal section that can be formulated 
differently, but eventually is:
- **The goal is to get more companies to start using Pulsar.**

In other words, the landing page should **sell** the project to potential users.

The defined scope of this PIP includes the "Homepage structure" point. I will 
try to express my thoughts on this.

## Add "Trusted by" or "Used in production" section

One of the best hooks to attract new customers is to show that their peers or 
more prominent players use it.

This is why I think it's essential to show the list of companies that use 
Pulsar as closer to the page top, as possible.

*Kafka in some form has this info on the first screen*
https://user-images.githubusercontent.com/9302460/221116492-b77cb8e9-eca1-46bd-b457-6889f4e708b6.png>

*Confluent has it on the second screen*
https://user-images.githubusercontent.com/9302460/221114783-4f22394e-d1fd-4640-8264-3b137f44e2a1.png>

*StreamNative has it on the second screen*
https://user-images.githubusercontent.com/9302460/221116665-6749212f-749f-4221-b175-07261a6d2ed5.png>

*Redpanda has it on the second screen*
https://user-images.githubusercontent.com/9302460/221118114-aabaee99-f6ef-4586-b7df-02c5ca3b24a6.png>

The homepage structure proposed in the PIP doesn't have a list of the companies 
on the page itself. Instead, it has a testimonials list at the **bottom** of 
the page with a link to case-studies.

## Too much text on the second screen

Users don't like to read long texts on landing pages. They'll just skip it.
Show the same or similar text in a bullet-point-like style would be better.

https://user-images.githubusercontent.com/9302460/221119160-5a39c7c9-ad68-4332-961a-e941f3a76693.png>

## The Pulsar Features section

Questions the potential user implicitly asks when they scroll the landing page 
sound like:

> 樂 How can this project help me to solve my pain points?
> 樂 Why should I "buy" this?

The answers to this question aren't limited to the technical features list.

The current PIP doesn't seem to answer these questions.

---

## Alternative page structure proposal

陋 Here it is: https://pulsar-site-three.vercel.app/

I would be happy to work with the designer if most users prefer the PIP's 
visual style.

My goal is not to compete here "just for fun". My company works on a product 
for Pulsar users, so I'm in some kind of dependent on a wider Pulsar adoption.

Thank you.


From: Yu 
Date: Friday, February 24, 2023 at 3:55 AM
To: dev@pulsar.apache.org 
Subject: Re: [DISCUSS] PIP-249: Pulsar website redesign
Hi @asafm and @emidio-cardeira thanks for your great work!

You've contributed new designs (with colors/elements/... that have not been
used) for Pulsar, which is a big step for the community.

Preiviously we've collected some rules at Pulsar Design Style Guide [1],
which are used to create illustrations in docs.

If this new look is accepted, can you document the design rules/best
practice? They should be available in the Contribution Guide [2].

In this way, other contributors can follow and create designs in high
quality and consistent style.

Thanks!

~

[1]
https://docs.google.com/document/d/16Hp7Sc86MQtL0m8fc2w_TrcKXAuglwRwHmdmwfk00mI/edit#heading=h.b8ogodj5sj

[2] https://pulsar.apache.org/contribute/


On Fri, Feb 24, 2023 at 5:27 AM Asaf Mesika  wrote:

> Hi,
>
> PIP issue: https://github.com/apache/pulsar/issues/19611
>
> One of the key elements that attracted me to Pulsar was how awesome it is -
> built in 2012 yet fits the new strategy of being cloud-native - simply
> incredible. It's not only that - it has so many outstanding features,
> making it a cutting edge, modern powerhouse in the messaging system
> category.
>
> Yet, its site does not reflect that.
> It does the opposite.
> People new to Pulsar, not knowing it as we do in the community, would think
> it's an old, unmaintained technology, and maybe not even bother reading the
> landing page.
>
> This is why I embarked on a journey to improve Pulsar web-site, in bite
> sizes. My first step was improving the landing page copy for the elevator
> pitch, opening paragraph, and the feature descriptions (In this PR
> )
>
> Today, together with my friend, co-worker and designer Emidio, I would like
> to propose the second step: revamp the landing page structure and most
> importantly - the website look and feel.
>
> The PIP  contains:
> * The screenshots for it
> * A "live" Figma view for it
> * Detailed explanation of what was changed and why.
>
> I think you'll love it.
>
> Happy to hear your thoughts.
>
>
> Asaf Mesika
>

Re: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema compatibility checks without using avro-protobuf

[Kiryl Valkovich]

[Edit] Sorry, it’s documented here: 
https://pulsar.apache.org/docs/2.11.x/schema-understand/#schema-compatibility-check

From: Kiryl Valkovich 
Date: Monday, February 20, 2023 at 3:48 PM
To: dev@pulsar.apache.org 
Subject: Re: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema compatibility 
checks without using avro-protobuf
Hm. I tend to think that for my Pulsar use case, it would be good to have the 
ability to implement a custom schema compatibility checker.

For example, buf.build (a popular Protobuf linter and build) offers the 
following list of breaking rules. Half of them prefixed with 
FIELD_SAME_XXLANG_PACKAGE_ aren’t relevant.

And I would want to use a subset of them if we are checking a single message 
descriptor. Most likely:
ENUM_VALUE_NO_DELETE
FIELD_NO_DELETE
FIELD_SAME_TYPE
ONEOF_NO_DELETE
FIELD_SAME_LABEL
RESERVED_ENUM_NO_DELETE
RESERVED_MESSAGE_NO_DELETE

I found out that the Pulsar broker has the following configuration option: 
schemaRegistryCompatibilityCheckers
[org.apache.pulsar.broker.service.schema.JsonSchemaCompatibilityCheck, 
org.apache.pulsar.broker.service.schema.AvroSchemaCompatibilityCheck, 
org.apache.pulsar.broker.service.schema.ProtobufNativeSchemaCompatibilityCheck]

But I can’t find documentation for it.

At first look, it seems that for my need it would be enough to have a such 
custom configurable Protobuf message descriptor checker class that implements 
SchemaCompatibilityCheck.

https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/ProtobufNativeSchemaCompatibilityCheck.java#L31

[cid:image002.png@01D94541.991807E0]

[Text  Description automatically generated]

From: SiNan Liu 
Date: Monday, February 20, 2023 at 1:41 PM
To: dev@pulsar.apache.org 
Subject: Re: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema compatibility 
checks without using avro-protobuf
It seems that we should only warn the user about changes to the field type,
not define a compatibility check, or throw an exception.
*Just like this: *
*log.warn("proto.read.ProtobufSchema.uint64Field field type for uint64, was
changed into a uint32");*

I will update this in the PIP issue Alternatives and discuss both designs
with other people.


Kiryl Valkovich  于2023年2月20日周一 18:14写道：

> 1. Sure, I didn’t mean that it's only about the required fields.
> 2. I found the page you are referring to.
> https://protobuf.dev/programming-guides/proto3/#updating
>
> QUOTE START
> If a number is parsed from the wire which doesn’t fit in the corresponding
> type, you will get the same effect as if you had cast the number to that
> type in C++ (for example, if a 64-bit number is read as an int32, it will
> be truncated to 32 bits).
> QUOTE END
>
> It’s discussable. Maybe I’m just missing something.
>
> I personally wouldn’t rely on such compatibility guarantees in a real
> application.
> If my check amount (> int32 lol) would be truncated because someone
> changed the field type in a schema, I would be quite upset.
>
> From: SiNan Liu 
> Date: Monday, February 20, 2023 at 2:30 AM
> To: dev@pulsar.apache.org 
> Subject: Re: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema
> compatibility checks without using avro-protobuf
> Thank you for your suggestions and questions.
> 1. Your first question. It's not just a matter of the required field. There
> should be many differences between proto3 and proto2. I will later test the
> current code for compatibility checks in proto3, and also look at
> compatibility between different protobuf versions.
> 2. On your second question. This is the compatibility rule for field type
> changes I found on the official website. You mean that this rule should not
> pass the compatibility check. Or should an exception be thrown whenever the
> field type changes?
>
>
> Kiryl Valkovich  于 2023年2月20日周一 上午12:31写道：
>
> > First, thank you for looking into it!
> >
> > There is a thing caught my eye:
> >
> > > The writtenSchema cannot add required fields, …
> >
> > As far as I know, the required fields were removed in Protocol Buffers
> v3.
> >
> > I see proto3 usage in existing schema registry tests:
> >
> >
> https://github.com/apache/pulsar/blob/6704f12104219611164aa2bb5bbdfc929613f1bf/pulsar-broker/src/test/proto/ProtobufSchemaTest.proto#L19
> >
> >
> >
> https://github.com/apache/pulsar/blob/1ea4ad814f5f30b8c371db2a86626cd568ace553/pulsar-sql/presto-pulsar/src/test/java/org/apache/pulsar/sql/presto/decoder/protobufnative/TestMsg.proto#L19
> >
> > I see proto2 usage in the proposed changes:
> >
> >
> >
> https://github.com/apache/pulsar/pull/19566/files#diff-f2f7463a3df6dc6366f3ee3a416707e655f0d5b8d2ae623a3f05a209c1d6ec88R19
> >
> >
> &

Re: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema compatibility checks without using avro-protobuf

[Kiryl Valkovich]

Hm. I tend to think that for my Pulsar use case, it would be good to have the 
ability to implement a custom schema compatibility checker.

For example, buf.build (a popular Protobuf linter and build) offers the 
following list of breaking rules. Half of them prefixed with 
FIELD_SAME_XXLANG_PACKAGE_ aren’t relevant.

And I would want to use a subset of them if we are checking a single message 
descriptor. Most likely:
ENUM_VALUE_NO_DELETE
FIELD_NO_DELETE
FIELD_SAME_TYPE
ONEOF_NO_DELETE
FIELD_SAME_LABEL
RESERVED_ENUM_NO_DELETE
RESERVED_MESSAGE_NO_DELETE

I found out that the Pulsar broker has the following configuration option: 
schemaRegistryCompatibilityCheckers
[org.apache.pulsar.broker.service.schema.JsonSchemaCompatibilityCheck, 
org.apache.pulsar.broker.service.schema.AvroSchemaCompatibilityCheck, 
org.apache.pulsar.broker.service.schema.ProtobufNativeSchemaCompatibilityCheck]

But I can’t find documentation for it.

At first look, it seems that for my need it would be enough to have a such 
custom configurable Protobuf message descriptor checker class that implements 
SchemaCompatibilityCheck.

https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/ProtobufNativeSchemaCompatibilityCheck.java#L31

[cid:image002.png@01D94541.991807E0]

[Text  Description automatically generated]

From: SiNan Liu 
Date: Monday, February 20, 2023 at 1:41 PM
To: dev@pulsar.apache.org 
Subject: Re: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema compatibility 
checks without using avro-protobuf
It seems that we should only warn the user about changes to the field type,
not define a compatibility check, or throw an exception.
*Just like this: *
*log.warn("proto.read.ProtobufSchema.uint64Field field type for uint64, was
changed into a uint32");*

I will update this in the PIP issue Alternatives and discuss both designs
with other people.


Kiryl Valkovich  于2023年2月20日周一 18:14写道：

> 1. Sure, I didn’t mean that it's only about the required fields.
> 2. I found the page you are referring to.
> https://protobuf.dev/programming-guides/proto3/#updating
>
> QUOTE START
> If a number is parsed from the wire which doesn’t fit in the corresponding
> type, you will get the same effect as if you had cast the number to that
> type in C++ (for example, if a 64-bit number is read as an int32, it will
> be truncated to 32 bits).
> QUOTE END
>
> It’s discussable. Maybe I’m just missing something.
>
> I personally wouldn’t rely on such compatibility guarantees in a real
> application.
> If my check amount (> int32 lol) would be truncated because someone
> changed the field type in a schema, I would be quite upset.
>
> From: SiNan Liu 
> Date: Monday, February 20, 2023 at 2:30 AM
> To: dev@pulsar.apache.org 
> Subject: Re: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema
> compatibility checks without using avro-protobuf
> Thank you for your suggestions and questions.
> 1. Your first question. It's not just a matter of the required field. There
> should be many differences between proto3 and proto2. I will later test the
> current code for compatibility checks in proto3, and also look at
> compatibility between different protobuf versions.
> 2. On your second question. This is the compatibility rule for field type
> changes I found on the official website. You mean that this rule should not
> pass the compatibility check. Or should an exception be thrown whenever the
> field type changes?
>
>
> Kiryl Valkovich  于 2023年2月20日周一 上午12:31写道：
>
> > First, thank you for looking into it!
> >
> > There is a thing caught my eye:
> >
> > > The writtenSchema cannot add required fields, …
> >
> > As far as I know, the required fields were removed in Protocol Buffers
> v3.
> >
> > I see proto3 usage in existing schema registry tests:
> >
> >
> https://github.com/apache/pulsar/blob/6704f12104219611164aa2bb5bbdfc929613f1bf/pulsar-broker/src/test/proto/ProtobufSchemaTest.proto#L19
> >
> >
> >
> https://github.com/apache/pulsar/blob/1ea4ad814f5f30b8c371db2a86626cd568ace553/pulsar-sql/presto-pulsar/src/test/java/org/apache/pulsar/sql/presto/decoder/protobufnative/TestMsg.proto#L19
> >
> > I see proto2 usage in the proposed changes:
> >
> >
> >
> https://github.com/apache/pulsar/pull/19566/files#diff-f2f7463a3df6dc6366f3ee3a416707e655f0d5b8d2ae623a3f05a209c1d6ec88R19
> >
> >
> >
> https://github.com/apache/pulsar/pull/19566/files#diff-f2f7463a3df6dc6366f3ee3a416707e655f0d5b8d2ae623a3f05a209c1d6ec88R19
> >
> > Also, I’m not sure about this:
> >
> > > int32, uint32, int64, uint64, and bool are all compatible �C this means
> > you can change a field from one of th

Re: Does anyone build UI for Pulsar?

[Kiryl Valkovich]

Enrico, it’s easy.

When I tried it, the basic functionality just didn’t work.

Just make a side-by-side comparison of Pulsar Manager with any of the following 
options:
- Conduktor (Commercial).
- Kafka UI by Provectus (Apache License 2.0).
- Redpanda Console (ex-Kowl) (Mixed license).

I see a significant gap in user experience between any of them and the Pulsar 
Manager.
Also, I just don’t see much contribution activity.

Redpanda Console and Kafka UI, each have about x10 times more commits in a 
year, than Pulsar Manager has in the last 3 years. Therefore I can’t conclude 
that the project is alive.

Regarding the “Why from scratch?” question:
- I found it easier for me personally because of the different tech stack I use.
- Starting from scratch is a good opportunity to learn Pulsar itself.
- I’d want to make a product around that.

I'm not claiming that my result will be 100% better, but why not try?

If someone is already doing or plans to do similar work, let’s collaborate!

From: Enrico Olivelli 
Date: Monday, February 20, 2023 at 2:04 PM
To: dev@pulsar.apache.org 
Subject: Re: Does anyone build UI for Pulsar?
Kiryl,

Il giorno lun 20 feb 2023 alle ore 12:18 Kiryl Valkovich
 ha scritto:
>
> Enrico, it seems you read only the mail message title.

Sorry about that,
I have re-read the message, and I have just realised that I skipped
the very first line :-)

I have one question.
IIUC you started from scratch a new UI, could you please explain why
Pulsar Manager doesn't work for you?


Cheers
Enrico

>
> From: Enrico Olivelli 
> Date: Monday, February 20, 2023 at 11:59 AM
> To: dev@pulsar.apache.org 
> Subject: Re: Does anyone build UI for Pulsar?
> Kiryl,
>
> You can use the official Apache Pulsar Manager the is maintained by
> this community
> https://github.com/apache/pulsar-manager
>
> At DataStax we also maintain this other UI that is also 100% opensource
> https://github.com/datastax/pulsar-admin-console
>
> For the BookKeeper part there is BKVM (BookKeeper Visual Manager)
> https://github.com/diennea/bookkeeper-visual-manager
> This is also bundled with Apache Pulsar Manager
>
>
> I hope that helps
>
> Enrico
>
>
>
> Il giorno lun 20 feb 2023 alle ore 10:51 Kiryl Valkovich
>  ha scritto:
> >
> > Hi everyone!
> >
> > Does anyone personally or some company work on UI for Pulsar other than 
> > pulsar-manager or pulsar-admin-console?
> >
> > I understand that StreamNative and DataStax have managed solutions and 
> > obviously work on their UI.
> >
> > I rather looking for an open-source or commercial tool that can be used in 
> > pair with any Pulsar deployment.
> >
> > I spent some time implementing UI for Apache Pulsar. It’s not ready to 
> > release yet. As usual, the most difficult 20% of the work remained.
> >
> > I’m asking that to avoid situations when several people or teams are doing 
> > the same thing.
> > Recently I proposed the site redesign and some teams are already doing that 
> > as it’s figured out.
> >
> > See Asaf’s comment:
> > https://github.com/apache/pulsar-site/pull/426#issuecomment-1436589392
> >
> > Please at least DM me if it’s not public information yet.
> >
> > Also DM me if you think that we can try to cooperate.
> >
> > Thank you.

Re: Does anyone build UI for Pulsar?

[Kiryl Valkovich]

Enrico, it seems you read only the mail message title.

From: Enrico Olivelli 
Date: Monday, February 20, 2023 at 11:59 AM
To: dev@pulsar.apache.org 
Subject: Re: Does anyone build UI for Pulsar?
Kiryl,

You can use the official Apache Pulsar Manager the is maintained by
this community
https://github.com/apache/pulsar-manager

At DataStax we also maintain this other UI that is also 100% opensource
https://github.com/datastax/pulsar-admin-console

For the BookKeeper part there is BKVM (BookKeeper Visual Manager)
https://github.com/diennea/bookkeeper-visual-manager
This is also bundled with Apache Pulsar Manager


I hope that helps

Enrico



Il giorno lun 20 feb 2023 alle ore 10:51 Kiryl Valkovich
 ha scritto:
>
> Hi everyone!
>
> Does anyone personally or some company work on UI for Pulsar other than 
> pulsar-manager or pulsar-admin-console?
>
> I understand that StreamNative and DataStax have managed solutions and 
> obviously work on their UI.
>
> I rather looking for an open-source or commercial tool that can be used in 
> pair with any Pulsar deployment.
>
> I spent some time implementing UI for Apache Pulsar. It’s not ready to 
> release yet. As usual, the most difficult 20% of the work remained.
>
> I’m asking that to avoid situations when several people or teams are doing 
> the same thing.
> Recently I proposed the site redesign and some teams are already doing that 
> as it’s figured out.
>
> See Asaf’s comment:
> https://github.com/apache/pulsar-site/pull/426#issuecomment-1436589392
>
> Please at least DM me if it’s not public information yet.
>
> Also DM me if you think that we can try to cooperate.
>
> Thank you.

Re: Force redirect questions from Slack to GitHub Discussions or StackOverflow?

[Kiryl Valkovich]

Do you mean, to do it for all messages in the #general channel (maybe only for 
messages that contain the question mark)?

I think it makes sense.

From: Asaf Mesika 
Date: Sunday, February 19, 2023 at 11:54 AM
To: dev@pulsar.apache.org 
Subject: Re: Force redirect questions from Slack to GitHub Discussions or 
StackOverflow?
I would have the bot open a Thread for the message, *suggesting* the user
to click to convert this question into a GitHub Discussion question. This
way you can have the actual GitHub user asking the question and not a bot
one.

On Fri, Feb 17, 2023 at 10:59 PM Kiryl Valkovich 
wrote:

> What about such wording?
>
> ---
> Your question was moved here:
> https://github.com/apache/pulsar/discussions/123
>
> Please consider asking new questions here:
>
>   *   At StackOverflow using apache-pulsar tag.
>   *   In the Q category at GitHub Discussions.
>   *   Apache Pulsar User Mailing List.
>
>
> It will make it searchable by others. Also, this way we can collect a
> knowledge base outside of Slack over time.
>
> I can’t see how the words “please consider” force the user to do something.
>
> Users who have an account on StackOverflow or GitHub can use these
> platforms next time.
> Others can send their question via the mailing list.
>
> From: Dave Fisher 
> Date: Friday, February 17, 2023 at 9:28 PM
> To: dev@pulsar.apache.org 
> Subject: Re: Force redirect questions from Slack to GitHub Discussions or
> StackOverflow?
> My concern is that users should have a choice on where to post their
> questions. They might have concerns about GitHub’s terms and conditions. We
> can pin a message to slack pointing to GitHub discussions and stackoverflow.
>
> Best,
> Dave
>
> Sent from my iPhone
>
> > On Feb 17, 2023, at 9:22 AM, Kiryl Valkovich 
> wrote:
> >
> > I’m the owner of this account.
> > The goal is to test drive duplicating Slack questions to the GitHub
> discussions.
> > With the current level of activity in Slack it’s not so hard to do it
> manually.
> >
> > I’m in CET now. I can share the account credentials with people who can
> post questions to GitHub Discussions on behalf of this account in other
> time zones.
> > Or I can do it once a day.
> >
> > If someone doesn’t find it useful or has ideas on how to do it in a
> better way, just say it directly.
> >
> > From: Enrico Olivelli 
> > Date: Friday, February 17, 2023 at 3:43 PM
> > To: dev@pulsar.apache.org 
> > Subject: Re: Force redirect questions from Slack to GitHub Discussions
> or StackOverflow?
> > Hello,
> > I see that some "Pulsar Community Bot" appeared in Slack
> >
> > it is connected to this email address "pulsar.community@gmail.com"
> >
> > While I find this thing "amazing"I wonder if I missed something,
> > who is the owner of this "bot" ?
> >
> >
> > Enrico
> >
> >> Il giorno gio 16 feb 2023 alle ore 16:03 Kiryl Valkovich
> >>  ha scritto:
> >>
> >> Played with Slack and StackOverflow APIs a bit.
> >>
> >> The Slack API works as expected. After clicking the message button, it
> sends a message descriptor to my app where I can do anything with its
> content.
> >>
> >> It’s possible to post messages via the StackOverflow API, but it’s
> unlikely that any Slack message can be converted to a StackOverflow
> question.
> >>
> >> I encountered several types of validation errors:
> >>
> >> -  This question body does not meet our quality standards.
> Please make sure that it completely describes your problem - including what
> you have already tried - and is written using proper grammar.
> >>
> >>  *   Something similar to “This message looks like a duplicate of
> another message”.
> >>
> >> I believe, GitHub Discussions don’t have such kind of “quality
> standards” validation.
> >>
> >> From: Kiryl Valkovich 
> >> Date: Thursday, February 16, 2023 at 1:33 PM
> >> To: dev@pulsar.apache.org 
> >> Subject: Re: Force redirect questions from Slack to GitHub Discussions
> or StackOverflow?
> >> If there are no hidden obstacles, we can implement it.
> >> StackOverflow supports question creation using REST API:
> https://api.stackexchange.com/docs/create-question
> >>
> >> From: Zike Yang 
> >> Date: Thursday, February 16, 2023 at 11:41 AM
> >> To: dev@pulsar.apache.org 
> >> Subject: Re: Force redirect questions from Slack to GitHub Discussions
> or StackOverflow?
> >> +1, That sounds great to 

Re: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema compatibility checks without using avro-protobuf

[Kiryl Valkovich]

1. Sure, I didn’t mean that it's only about the required fields.
2. I found the page you are referring to.
https://protobuf.dev/programming-guides/proto3/#updating

QUOTE START
If a number is parsed from the wire which doesn’t fit in the corresponding 
type, you will get the same effect as if you had cast the number to that type 
in C++ (for example, if a 64-bit number is read as an int32, it will be 
truncated to 32 bits).
QUOTE END

It’s discussable. Maybe I’m just missing something.

I personally wouldn’t rely on such compatibility guarantees in a real 
application.
If my check amount (> int32 lol) would be truncated because someone changed the 
field type in a schema, I would be quite upset.

From: SiNan Liu 
Date: Monday, February 20, 2023 at 2:30 AM
To: dev@pulsar.apache.org 
Subject: Re: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema compatibility 
checks without using avro-protobuf
Thank you for your suggestions and questions.
1. Your first question. It's not just a matter of the required field. There
should be many differences between proto3 and proto2. I will later test the
current code for compatibility checks in proto3, and also look at
compatibility between different protobuf versions.
2. On your second question. This is the compatibility rule for field type
changes I found on the official website. You mean that this rule should not
pass the compatibility check. Or should an exception be thrown whenever the
field type changes?


Kiryl Valkovich  于 2023年2月20日周一 上午12:31写道：

> First, thank you for looking into it!
>
> There is a thing caught my eye:
>
> > The writtenSchema cannot add required fields, …
>
> As far as I know, the required fields were removed in Protocol Buffers v3.
>
> I see proto3 usage in existing schema registry tests:
>
> https://github.com/apache/pulsar/blob/6704f12104219611164aa2bb5bbdfc929613f1bf/pulsar-broker/src/test/proto/ProtobufSchemaTest.proto#L19
>
>
> https://github.com/apache/pulsar/blob/1ea4ad814f5f30b8c371db2a86626cd568ace553/pulsar-sql/presto-pulsar/src/test/java/org/apache/pulsar/sql/presto/decoder/protobufnative/TestMsg.proto#L19
>
> I see proto2 usage in the proposed changes:
>
>
> https://github.com/apache/pulsar/pull/19566/files#diff-f2f7463a3df6dc6366f3ee3a416707e655f0d5b8d2ae623a3f05a209c1d6ec88R19
>
>
> https://github.com/apache/pulsar/pull/19566/files#diff-f2f7463a3df6dc6366f3ee3a416707e655f0d5b8d2ae623a3f05a209c1d6ec88R19
>
> Also, I’m not sure about this:
>
> > int32, uint32, int64, uint64, and bool are all compatible �C this means
> you can change a field from one of these types to another without breaking
> forwards- or backwards-compatibility.
>
> It leads to JS/PHP-like implicit conversions if I understand it right.
>
> From: SiNan Liu 
> Date: Sunday, February 19, 2023 at 1:59 PM
> To: dev@pulsar.apache.org 
> Subject: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema compatibility
> checks without using avro-protobuf
> Hi all,
>
> I made a PIP to discuss: https://github.com/apache/pulsar/issues/19565.
>
> We can talk about the current design here. Especially for the field type
> change check rules, please give your valuable advice.
>
> Thanks,
> Sinan
>

Does anyone build UI for Pulsar?

[Kiryl Valkovich]

Hi everyone!

Does anyone personally or some company work on UI for Pulsar other than 
pulsar-manager or pulsar-admin-console?

I understand that StreamNative and DataStax have managed solutions and 
obviously work on their UI.

I rather looking for an open-source or commercial tool that can be used in pair 
with any Pulsar deployment.

I spent some time implementing UI for Apache Pulsar. It’s not ready to release 
yet. As usual, the most difficult 20% of the work remained.

I’m asking that to avoid situations when several people or teams are doing the 
same thing.
Recently I proposed the site redesign and some teams are already doing that as 
it’s figured out.

See Asaf’s comment:
https://github.com/apache/pulsar-site/pull/426#issuecomment-1436589392

Please at least DM me if it’s not public information yet.

Also DM me if you think that we can try to cooperate.

Thank you.

Re: Modern look for the website

[Kiryl Valkovich]

I asked only about the home/index/root (/) page look.

Other pages just aren't re-implemented.

I don't know the history behind the previous redesign. I met many situations 
when people didn't want to consider changes for some reason.

Therefore, I must be sure that the proposed changes most likely will be 
accepted before investing my time.


From: PengHui Li 
Date: Monday, February 20, 2023 at 4:51 AM
To: dev@pulsar.apache.org 
Subject: Re: Modern look for the website
It looks super cool!

I just found some links that don't work.

The button "Learn more" and "QuickStart" in the index page will forward to 404 
page
[cid:ii_leca41v80]

And the case study page looks not coordinated well.

https://pulsar-site-three.vercel.app/case-studies

Thanks for the great job.


Penghui

On Mon, Feb 20, 2023 at 6:38 AM Kiryl Valkovich  
wrote:
Hi everyone!



I don't think anyone will argue, it’s important to have a fresh-looking website 
to attract new users.



I decided to try to refresh it a little bit.



Over the weekend I managed to make a home page. I think that I can finish other 
pages in 1-2 weeks, depending on my workload.



Current version: https://pulsar.apache.org/

New version: https://pulsar-site-three.vercel.app



Draft PR: https://github.com/apache/pulsar-site/pull/426



After the redesign, I plan to add a few more pages.



Should I continue?

Modern look for the website

[Kiryl Valkovich]

Hi everyone!



I don't think anyone will argue, it’s important to have a fresh-looking website 
to attract new users.



I decided to try to refresh it a little bit.



Over the weekend I managed to make a home page. I think that I can finish other 
pages in 1-2 weeks, depending on my workload.



Current version: https://pulsar.apache.org/

New version: https://pulsar-site-three.vercel.app



Draft PR: https://github.com/apache/pulsar-site/pull/426



After the redesign, I plan to add a few more pages.



Should I continue?

Re: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema compatibility checks without using avro-protobuf

[Kiryl Valkovich]

First, thank you for looking into it!

There is a thing caught my eye:

> The writtenSchema cannot add required fields, …

As far as I know, the required fields were removed in Protocol Buffers v3.

I see proto3 usage in existing schema registry tests:
https://github.com/apache/pulsar/blob/6704f12104219611164aa2bb5bbdfc929613f1bf/pulsar-broker/src/test/proto/ProtobufSchemaTest.proto#L19

https://github.com/apache/pulsar/blob/1ea4ad814f5f30b8c371db2a86626cd568ace553/pulsar-sql/presto-pulsar/src/test/java/org/apache/pulsar/sql/presto/decoder/protobufnative/TestMsg.proto#L19

I see proto2 usage in the proposed changes:

https://github.com/apache/pulsar/pull/19566/files#diff-f2f7463a3df6dc6366f3ee3a416707e655f0d5b8d2ae623a3f05a209c1d6ec88R19

https://github.com/apache/pulsar/pull/19566/files#diff-f2f7463a3df6dc6366f3ee3a416707e655f0d5b8d2ae623a3f05a209c1d6ec88R19

Also, I’m not sure about this:

> int32, uint32, int64, uint64, and bool are all compatible – this means you 
> can change a field from one of these types to another without breaking 
> forwards- or backwards-compatibility.

It leads to JS/PHP-like implicit conversions if I understand it right.

From: SiNan Liu 
Date: Sunday, February 19, 2023 at 1:59 PM
To: dev@pulsar.apache.org 
Subject: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema compatibility 
checks without using avro-protobuf
Hi all,

I made a PIP to discuss: https://github.com/apache/pulsar/issues/19565.

We can talk about the current design here. Especially for the field type
change check rules, please give your valuable advice.

Thanks,
Sinan

Re: Force redirect questions from Slack to GitHub Discussions or StackOverflow?

[Kiryl Valkovich]

What about such wording?

---
Your question was moved here: https://github.com/apache/pulsar/discussions/123

Please consider asking new questions here:

  *   At StackOverflow using apache-pulsar tag.
  *   In the Q category at GitHub Discussions.
  *   Apache Pulsar User Mailing List.


It will make it searchable by others. Also, this way we can collect a knowledge 
base outside of Slack over time.

I can’t see how the words “please consider” force the user to do something.

Users who have an account on StackOverflow or GitHub can use these platforms 
next time.
Others can send their question via the mailing list.

From: Dave Fisher 
Date: Friday, February 17, 2023 at 9:28 PM
To: dev@pulsar.apache.org 
Subject: Re: Force redirect questions from Slack to GitHub Discussions or 
StackOverflow?
My concern is that users should have a choice on where to post their questions. 
They might have concerns about GitHub’s terms and conditions. We can pin a 
message to slack pointing to GitHub discussions and stackoverflow.

Best,
Dave

Sent from my iPhone

> On Feb 17, 2023, at 9:22 AM, Kiryl Valkovich  
> wrote:
>
> I’m the owner of this account.
> The goal is to test drive duplicating Slack questions to the GitHub 
> discussions.
> With the current level of activity in Slack it’s not so hard to do it 
> manually.
>
> I’m in CET now. I can share the account credentials with people who can post 
> questions to GitHub Discussions on behalf of this account in other time zones.
> Or I can do it once a day.
>
> If someone doesn’t find it useful or has ideas on how to do it in a better 
> way, just say it directly.
>
> From: Enrico Olivelli 
> Date: Friday, February 17, 2023 at 3:43 PM
> To: dev@pulsar.apache.org 
> Subject: Re: Force redirect questions from Slack to GitHub Discussions or 
> StackOverflow?
> Hello,
> I see that some "Pulsar Community Bot" appeared in Slack
>
> it is connected to this email address "pulsar.community@gmail.com"
>
> While I find this thing "amazing"I wonder if I missed something,
> who is the owner of this "bot" ?
>
>
> Enrico
>
>> Il giorno gio 16 feb 2023 alle ore 16:03 Kiryl Valkovich
>>  ha scritto:
>>
>> Played with Slack and StackOverflow APIs a bit.
>>
>> The Slack API works as expected. After clicking the message button, it sends 
>> a message descriptor to my app where I can do anything with its content.
>>
>> It’s possible to post messages via the StackOverflow API, but it’s unlikely 
>> that any Slack message can be converted to a StackOverflow question.
>>
>> I encountered several types of validation errors:
>>
>> -  This question body does not meet our quality standards. Please 
>> make sure that it completely describes your problem - including what you 
>> have already tried - and is written using proper grammar.
>>
>>  *   Something similar to “This message looks like a duplicate of another 
>> message”.
>>
>> I believe, GitHub Discussions don’t have such kind of “quality standards” 
>> validation.
>>
>> From: Kiryl Valkovich 
>> Date: Thursday, February 16, 2023 at 1:33 PM
>> To: dev@pulsar.apache.org 
>> Subject: Re: Force redirect questions from Slack to GitHub Discussions or 
>> StackOverflow?
>> If there are no hidden obstacles, we can implement it.
>> StackOverflow supports question creation using REST API: 
>> https://api.stackexchange.com/docs/create-question
>>
>> From: Zike Yang 
>> Date: Thursday, February 16, 2023 at 11:41 AM
>> To: dev@pulsar.apache.org 
>> Subject: Re: Force redirect questions from Slack to GitHub Discussions or 
>> StackOverflow?
>> +1, That sounds great to me.
>>
>>> 2. You click the three dots button on his message -> “Convert to a GitHub 
>>> discussion”.
>>
>> Is it a similar workflow for converting to a StackOverflow question?
>>
>> BR,
>> Zike Yang
>>
>>> On Thu, Feb 16, 2023 at 12:14 PM  wrote:
>>>
>>> +1
>>>
>>> Since web pages are more easily and public.
>>>
>>> Best
>>> Mattison
>>> On Feb 16, 2023, 07:58 +0800, Christophe Bornet , 
>>> wrote:
>>>> +100
>>>> Also note that for good or bad reasons, the number of questions on
>>>> StaOverflow is often used as a metric for the popularity of a project.
>>>>
>>>> Le mer. 15 févr. 2023 à 13:12, Kiryl Valkovich 
>>>> a écrit :
>>>>
>>>>> Hi everyone! Enrico Olivelli asked me to repost it to the mailing list.
>>>>>
>>>>> --- Me
&

Re: Force redirect questions from Slack to GitHub Discussions or StackOverflow?

[Kiryl Valkovich]

I’m the owner of this account.
The goal is to test drive duplicating Slack questions to the GitHub discussions.
With the current level of activity in Slack it’s not so hard to do it manually.

I’m in CET now. I can share the account credentials with people who can post 
questions to GitHub Discussions on behalf of this account in other time zones.
Or I can do it once a day.

If someone doesn’t find it useful or has ideas on how to do it in a better way, 
just say it directly.

From: Enrico Olivelli 
Date: Friday, February 17, 2023 at 3:43 PM
To: dev@pulsar.apache.org 
Subject: Re: Force redirect questions from Slack to GitHub Discussions or 
StackOverflow?
Hello,
I see that some "Pulsar Community Bot" appeared in Slack

it is connected to this email address "pulsar.community@gmail.com"

While I find this thing "amazing"I wonder if I missed something,
who is the owner of this "bot" ?


Enrico

Il giorno gio 16 feb 2023 alle ore 16:03 Kiryl Valkovich
 ha scritto:
>
> Played with Slack and StackOverflow APIs a bit.
>
> The Slack API works as expected. After clicking the message button, it sends 
> a message descriptor to my app where I can do anything with its content.
>
> It’s possible to post messages via the StackOverflow API, but it’s unlikely 
> that any Slack message can be converted to a StackOverflow question.
>
> I encountered several types of validation errors:
>
> -  This question body does not meet our quality standards. Please 
> make sure that it completely describes your problem - including what you have 
> already tried - and is written using proper grammar.
>
>   *   Something similar to “This message looks like a duplicate of another 
> message”.
>
> I believe, GitHub Discussions don’t have such kind of “quality standards” 
> validation.
>
> From: Kiryl Valkovich 
> Date: Thursday, February 16, 2023 at 1:33 PM
> To: dev@pulsar.apache.org 
> Subject: Re: Force redirect questions from Slack to GitHub Discussions or 
> StackOverflow?
> If there are no hidden obstacles, we can implement it.
> StackOverflow supports question creation using REST API: 
> https://api.stackexchange.com/docs/create-question
>
> From: Zike Yang 
> Date: Thursday, February 16, 2023 at 11:41 AM
> To: dev@pulsar.apache.org 
> Subject: Re: Force redirect questions from Slack to GitHub Discussions or 
> StackOverflow?
> +1, That sounds great to me.
>
> > 2. You click the three dots button on his message -> “Convert to a GitHub 
> > discussion”.
>
> Is it a similar workflow for converting to a StackOverflow question?
>
> BR,
> Zike Yang
>
> On Thu, Feb 16, 2023 at 12:14 PM  wrote:
> >
> > +1
> >
> > Since web pages are more easily and public.
> >
> > Best
> > Mattison
> > On Feb 16, 2023, 07:58 +0800, Christophe Bornet , 
> > wrote:
> > > +100
> > > Also note that for good or bad reasons, the number of questions on
> > > StaOverflow is often used as a metric for the popularity of a project.
> > >
> > > Le mer. 15 févr. 2023 à 13:12, Kiryl Valkovich 
> > > 
> > > a écrit :
> > >
> > > > Hi everyone! Enrico Olivelli asked me to repost it to the mailing list.
> > > >
> > > > --- Me
> > > > I’m very worried that good answers from David Kjerrumgaard and others
> > > > won’t be googleable because it’s in Slack. To make any search you need 
> > > > to
> > > > pay a monthly subscription.
> > > >
> > > > In my opinion, it would be wiser to make StackOverflow, Discuss, or 
> > > > GitHub
> > > > discussions a place for questions. And strictly redirect people who ask 
> > > > any
> > > > question in Slack to the right place.
> > > > In GitHub discussions, you also can manage categories, as well as you 
> > > > can
> > > > manage channels in Slack.
> > > >
> > > > Subscription to a specific category is coming soon.
> > > > https://github.com/community/community/discussions/3951#discussioncomment-3879939
> > > > The most important that it’s searchable.
> > > >
> > > > Examples:
> > > > https://github.com/community/community/discussions
> > > > https://github.com/apache/superset/discussions
> > > >
> > > > --- Later me to David Kjerrumgaard
> > > > What do you think about the following workflow:
> > > > 1. The user asks a new question in Slack.
> > > > 2. You click the three dots button on his message -> “Convert to a 
> > > > GitHub
> > > > discu

Re: Force redirect questions from Slack to GitHub Discussions or StackOverflow?

[Kiryl Valkovich]

Played with Slack and StackOverflow APIs a bit.

The Slack API works as expected. After clicking the message button, it sends a 
message descriptor to my app where I can do anything with its content.

It’s possible to post messages via the StackOverflow API, but it’s unlikely 
that any Slack message can be converted to a StackOverflow question.

I encountered several types of validation errors:

-  This question body does not meet our quality standards. Please make 
sure that it completely describes your problem - including what you have 
already tried - and is written using proper grammar.

  *   Something similar to “This message looks like a duplicate of another 
message”.

I believe, GitHub Discussions don’t have such kind of “quality standards” 
validation.

From: Kiryl Valkovich 
Date: Thursday, February 16, 2023 at 1:33 PM
To: dev@pulsar.apache.org 
Subject: Re: Force redirect questions from Slack to GitHub Discussions or 
StackOverflow?
If there are no hidden obstacles, we can implement it.
StackOverflow supports question creation using REST API: 
https://api.stackexchange.com/docs/create-question

From: Zike Yang 
Date: Thursday, February 16, 2023 at 11:41 AM
To: dev@pulsar.apache.org 
Subject: Re: Force redirect questions from Slack to GitHub Discussions or 
StackOverflow?
+1, That sounds great to me.

> 2. You click the three dots button on his message -> “Convert to a GitHub 
> discussion”.

Is it a similar workflow for converting to a StackOverflow question?

BR,
Zike Yang

On Thu, Feb 16, 2023 at 12:14 PM  wrote:
>
> +1
>
> Since web pages are more easily and public.
>
> Best
> Mattison
> On Feb 16, 2023, 07:58 +0800, Christophe Bornet , 
> wrote:
> > +100
> > Also note that for good or bad reasons, the number of questions on
> > StaOverflow is often used as a metric for the popularity of a project.
> >
> > Le mer. 15 févr. 2023 à 13:12, Kiryl Valkovich 
> > a écrit :
> >
> > > Hi everyone! Enrico Olivelli asked me to repost it to the mailing list.
> > >
> > > --- Me
> > > I’m very worried that good answers from David Kjerrumgaard and others
> > > won’t be googleable because it’s in Slack. To make any search you need to
> > > pay a monthly subscription.
> > >
> > > In my opinion, it would be wiser to make StackOverflow, Discuss, or GitHub
> > > discussions a place for questions. And strictly redirect people who ask 
> > > any
> > > question in Slack to the right place.
> > > In GitHub discussions, you also can manage categories, as well as you can
> > > manage channels in Slack.
> > >
> > > Subscription to a specific category is coming soon.
> > > https://github.com/community/community/discussions/3951#discussioncomment-3879939
> > > The most important that it’s searchable.
> > >
> > > Examples:
> > > https://github.com/community/community/discussions
> > > https://github.com/apache/superset/discussions
> > >
> > > --- Later me to David Kjerrumgaard
> > > What do you think about the following workflow:
> > > 1. The user asks a new question in Slack.
> > > 2. You click the three dots button on his message -> “Convert to a GitHub
> > > discussion”.
> > > 3. The bot creates a new GitHub discussion in the “Q” category with the
> > > original message’s content.
> > > 4. The bot converts the original message to a thread and posts a message
> > > like:
> > > > > In order to keep history and make your question searchable by other
> > > Apache Pulsar users, it has been converted to a GitHub discussion.
> > > > > Further conversation is available by this link:
> > > https://github.com/apache/pulsar/discussions/18766.
> > > 5. You answer the question on GitHub.
> > >
> > > How it may work:
> > > https://api.slack.com/best-practices/blueprints/actions-and-dialogs
> > > I can try to implement it makes any sense for you. (edited)
> > >
> > > Full original thread:
> > > https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1676449603034309
> > >
> > >

Re: Force redirect questions from Slack to GitHub Discussions or StackOverflow?

[Kiryl Valkovich]

If there are no hidden obstacles, we can implement it.
StackOverflow supports question creation using REST API: 
https://api.stackexchange.com/docs/create-question

From: Zike Yang 
Date: Thursday, February 16, 2023 at 11:41 AM
To: dev@pulsar.apache.org 
Subject: Re: Force redirect questions from Slack to GitHub Discussions or 
StackOverflow?
+1, That sounds great to me.

> 2. You click the three dots button on his message -> “Convert to a GitHub 
> discussion”.

Is it a similar workflow for converting to a StackOverflow question?

BR,
Zike Yang

On Thu, Feb 16, 2023 at 12:14 PM  wrote:
>
> +1
>
> Since web pages are more easily and public.
>
> Best
> Mattison
> On Feb 16, 2023, 07:58 +0800, Christophe Bornet , 
> wrote:
> > +100
> > Also note that for good or bad reasons, the number of questions on
> > StaOverflow is often used as a metric for the popularity of a project.
> >
> > Le mer. 15 févr. 2023 à 13:12, Kiryl Valkovich 
> > a écrit :
> >
> > > Hi everyone! Enrico Olivelli asked me to repost it to the mailing list.
> > >
> > > --- Me
> > > I’m very worried that good answers from David Kjerrumgaard and others
> > > won’t be googleable because it’s in Slack. To make any search you need to
> > > pay a monthly subscription.
> > >
> > > In my opinion, it would be wiser to make StackOverflow, Discuss, or GitHub
> > > discussions a place for questions. And strictly redirect people who ask 
> > > any
> > > question in Slack to the right place.
> > > In GitHub discussions, you also can manage categories, as well as you can
> > > manage channels in Slack.
> > >
> > > Subscription to a specific category is coming soon.
> > > https://github.com/community/community/discussions/3951#discussioncomment-3879939
> > > The most important that it’s searchable.
> > >
> > > Examples:
> > > https://github.com/community/community/discussions
> > > https://github.com/apache/superset/discussions
> > >
> > > --- Later me to David Kjerrumgaard
> > > What do you think about the following workflow:
> > > 1. The user asks a new question in Slack.
> > > 2. You click the three dots button on his message -> “Convert to a GitHub
> > > discussion”.
> > > 3. The bot creates a new GitHub discussion in the “Q” category with the
> > > original message’s content.
> > > 4. The bot converts the original message to a thread and posts a message
> > > like:
> > > > > In order to keep history and make your question searchable by other
> > > Apache Pulsar users, it has been converted to a GitHub discussion.
> > > > > Further conversation is available by this link:
> > > https://github.com/apache/pulsar/discussions/18766.
> > > 5. You answer the question on GitHub.
> > >
> > > How it may work:
> > > https://api.slack.com/best-practices/blueprints/actions-and-dialogs
> > > I can try to implement it makes any sense for you. (edited)
> > >
> > > Full original thread:
> > > https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1676449603034309
> > >
> > >

Force redirect questions from Slack to GitHub Discussions or StackOverflow?

[Kiryl Valkovich]

Hi everyone! Enrico Olivelli asked me to repost it to the mailing list.

--- Me
I’m very worried that good answers from David Kjerrumgaard and others won’t be 
googleable because it’s in Slack. To make any search you need to pay a monthly 
subscription.

In my opinion, it would be wiser to make StackOverflow, Discuss, or GitHub 
discussions a place for questions. And strictly redirect people who ask any 
question in Slack to the right place.
In GitHub discussions, you also can manage categories, as well as you can 
manage channels in Slack.

Subscription to a specific category is coming soon. 
https://github.com/community/community/discussions/3951#discussioncomment-3879939
The most important that it’s searchable.

Examples:
https://github.com/community/community/discussions
https://github.com/apache/superset/discussions

--- Later me to David Kjerrumgaard
What do you think about the following workflow:
1. The user asks a new question in Slack.
2. You click the three dots button on his message -> “Convert to a GitHub 
discussion”.
3. The bot creates a new GitHub discussion in the “Q” category with the 
original message’s content.
4. The bot converts the original message to a thread and posts a message like:
> In order to keep history and make your question searchable by other Apache 
> Pulsar users, it has been converted to a GitHub discussion.
> Further conversation is available by this link: 
> https://github.com/apache/pulsar/discussions/18766.
5. You answer the question on GitHub.

How it may work: 
https://api.slack.com/best-practices/blueprints/actions-and-dialogs
I can try to implement it makes any sense for you. (edited)

Full original thread: 
https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1676449603034309