+1 for starting with pulsar sink and catalog.
Sijie
On Tue, Sep 24, 2019 at 3:36 PM Jark Wu wrote:
> I agree that we can start from Pulsar sink and Pulsar Catalog.
> When we finish that, maybe FLIP-27 is ready then.
>
> Best,
> Jark
>
> On Tue, 24 Sep 2019 at 17:06, Becket Qin wrote:
>
>> Hi
I agree that we can start from Pulsar sink and Pulsar Catalog.
When we finish that, maybe FLIP-27 is ready then.
Best,
Jark
On Tue, 24 Sep 2019 at 17:06, Becket Qin wrote:
> Hi Till,
>
> My understanding is that we are going to hold back a little bit on the
> Pulsar source connector until
Hi Till,
My understanding is that we are going to hold back a little bit on the
Pulsar source connector until FLIP-27 is ready. If we check in the old
Pulsar source connector right away, we will have to maintain that old
connector for quite a while (a year or more), which is something we don't
What is the conclusion of this discussion? Are we gonna merge the old
source/sink or do we believe that FLIP-27 will be completed in time so that
we can merge the new source right away?
Cheers,
Till
On Tue, Sep 24, 2019 at 3:43 AM 郭士榕 wrote:
>
>
> unscribe
>
>
>
>
>
>
> At 2019-09-24 08:53:25,
Thanks, Stephan.
Sounds good to me. We can still try our best to get new Pulsar connector in
Flink 1.10. In case we do not have time to do that, we will prominently
link the Pulsar connector from the Flink connector docs.
Thanks,
Jiangjie (Becket) Qin
On Mon, Sep 23, 2019 at 4:11 PM Stephan
Okay, I see your point, Becket.
Then let us prominently link the Pulsar connector from the Flink connector
docs then, so that users can find it easily.
As soon as FLIP 27 is done, we reach out the Pulsar folks to contribute a
new connector.
On Mon, Sep 23, 2019 at 3:11 AM Becket Qin wrote:
>
Hi Stephan,
I have no doubt about the value of adding Pulsar connector to Flink repo.
My concern is about how exactly we are going to do it.
As mentioned before, I believe that we can handle connectors more
> pragmatically and less strict than the core of Flink, if it helps unlocking
> users.
I
My assumption is as Sijie's, that once the connector is either part of
Flink, or part of the streamnative repo. No double maintenance.
I feel this discussion is very much caught in problems that are all
solvable if we want to solve them.
Maybe we can think what our goal for users and the
Thanks Becket.
I think it is better for the Flink community to judge the benefits of doing
this. I was trying to provide some views from outsiders.
Thanks,
Sijie
On Fri, Sep 20, 2019 at 10:25 AM Becket Qin wrote:
> Hi Sijie,
>
> Yes, we will have to support existing old connectors and new
Hi Sijie,
Yes, we will have to support existing old connectors and new connectors in
parallel for a while. We have to take that maintenance overhead because
existing connectors have been used by the users for a long time. I guess It
may take at least a year for us to fully remove the old
Thanks everyone here. Sorry for jumping into the discussion here.
I am not very familiar about the deprecation process in Flink. If I
misunderstood the process, please fix me.
As far as I understand, FLIP-27 is introducing a new unified API for
connectors. After it introduces the new API
and
Thanks for the explanation, Stephan. I have a few questions / thoughts.
So that means we will remove the old connector without a major version
bump, is that correct?
I am not 100% sure if mixing 1.10 connectors with 1.11 connectors will
always work because we saw some dependency class collisions
My take would be the following:
- If we merge the connector now and replace it with a FLIP-27 version
before the 1.10 release, then we need no deprecation process
- If we don't manage to replace it with a FLIP-27 version before the 1.10
release, than it is good that we have the other version,
Hi,
You should have the necessary permissions.
Let me know if something doesn't work.
Cheers, Fabian
Am Do., 19. Sept. 2019 um 10:56 Uhr schrieb Yijie Shen <
henry.yijies...@gmail.com>:
> Hi Fabian,
>
> It's yijieshen
>
> Thanks for your help!
>
> On Thu, Sep 19, 2019 at 4:52 PM Fabian
Hi Stephan,
Thanks for the clarification. I completely agree with you and Thomas on the
process of adding connectors to Flink repo. However, I am wondering what is
the deprecation process? Given the main concern here was that we may have
to maintain two Pulsar connector code bases until the old
Hi Fabian,
It's yijieshen
Thanks for your help!
On Thu, Sep 19, 2019 at 4:52 PM Fabian Hueske wrote:
>
> Hi Yijie,
>
> I can give you permission if you tell me your Confluence user name.
>
> Thanks, Fabian
>
> Am Do., 19. Sept. 2019 um 10:49 Uhr schrieb Yijie Shen <
>
Hi Yijie,
I can give you permission if you tell me your Confluence user name.
Thanks, Fabian
Am Do., 19. Sept. 2019 um 10:49 Uhr schrieb Yijie Shen <
henry.yijies...@gmail.com>:
> > > > Could you please follow the FLIP process to start a new FLIP
> [DISCUSSION]
> > > > thread in the mailing
> > > Could you please follow the FLIP process to start a new FLIP [DISCUSSION]
> > > thread in the mailing list?
It seems that I don't have permission to create pages in FLIP.
> > > > Thanks for sharing the pulsar FLIP.
> > > > Would you mind enabling comments/suggestions on the google doc
+1 to what Stephan (and Thomas) said.
Am Do., 19. Sept. 2019 um 09:54 Uhr schrieb Stephan Ewen :
> Some quick thoughts on the connector contribution process. I basically
> reiterate here what Thomas mentioned in another thread about the Kinesis
> connector.
>
> For connectors, we should favor a
; > scalability
> > > > > > > > > > > > > > > > bar.
> > > > > > > > > > > > > > > > > > With more and more connectors coming into
> > > Flink
> > > > > > repo,
> > > > > > > > we
> temporarily
> > > > > > > > > > > > > > > > > on hold due to potential solution to
> shorten
> > the
> > > > > > build
> > > > > > > > time.
> > > > > > > > > > > > >
Given the above efforts, it would be great to
> > > first
> > > > > see
> > > > > > > if we
> > > > > > > > > > can
> > > > > > > >
> > > > > > > with
> > > > > > > > > > > > good
> > > > > > > > > > > > > > > quality and test coverage. If the quality as an
> > > > > ecosystem
> > > > > > > >
t; > > > > > > land in 1.10. Therefore timing wise, if we are
> > going
> > > to
> > > > > have
> > > > > > > > the
> > > > > > > > > > > Pulsar
> > >
t; > > > > >
> > > > > > > > > > > > > On Wed, Sep 4, 2019 at 4:32 PM Chesnay Schepler <
> > > > > > > > > ches...@apache.org>
> > > > > > > > > > > > > wrote:
>
ntribution
> > > > > > > > > > > > > beforehand and getting several +1's from
> committers.
> > > > > > > > > > > > >
> > > > > > > > > > > > > We should really make sure that if we
>
he
> > > > > > community
> > > > > > > > > (what's
> > > > > > > > > > > > the state of that anyway?), this may be a better
> > option.
> > > > > > > > > > > >
> > > > > > > > >
gt; more
> > > > > > > > > > and
> > > > > > > > > > > > more popular and it would further expand Flink's
> > > > > > > interoperability.
> > > > > > > > Also
> > > > > > > > &g
developed components because they need to be
> > kept
> > > in
> > > > > > sync
> > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > > external
; > > it
> > > > > > back
> > > > > > > > > to
> > > > > > > > > > Flink.
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > >
gt; > > >>
> > > > > > > > >> FLINK-9641 and FLINK-9168 was created for bringing Pulsar
> as
> > > > > source
> > > > > > > and
> > > > > > > > >> sink for Flink. The integratio
> > >> Pulsar for Pulsar users to use Flink to process event
> streams
> > in
> > > > > > Pulsar.
> > > > > > > >> (See
> > https://github.com/apache/pulsar/tree/master/pulsar-flink
> > > ).
> > > > The
> > > > > > > Flink
>
t streams
> with
> > > > > Pulsar
> > > > > > >> schema.
> > > > > > >>
> > > > > > >> It led us to rethink how to integrate with Flink in the best
> > way.
> > > Then
> > > > > > we
> >
the
> > > > > >> integration. With that being said, in the new pulsar-flink
> > > > > implementation,
> > > > > >> you can register pulsar as a flink catalog and query / process
> the
> > > > event
> >
614ca9c9/README.md#use-pulsar-catalog
> > > > >>
> > > > >> Yijie has also written a blog post explaining why we re-implement
> the
> > > > flink
> > > > >> connector with Flink 1.9 and what are the changes we made in the
>
; >> We believe Pulsar is not just a simple data sink or source for Flink.
> > It
> > > >> actually can be a fully integrated streaming data storage for Flink in
> > > many
> > > >> areas (sink, source, schema/catalog and state). The combination of
> >
ted to
> > >> maintain, improve and evolve the integration with Flink to help the
> > users
> > >> who use both Flink and Pulsar.
> > >>
> > >> Hope this give you a bit more background about the pulsar flink
> > >> integration. Let me know what are your thoughts.
> > &g
; >>
> >>> Hi Yijie
> >>>
> >>> I can see that Pulsar becomes more and more popular recently and very
> >> glad
> >>> to see more people willing to contribute to Flink ecosystem.
> >>>
> >>> Before any further
From: Yijie Shen
Sent: Tuesday, September 3, 2019 13:57
To: dev@flink.apache.org
Subject: [DISCUSS] Contribute Pulsar Flink connector back to Flink
Dear Flink Community!
I would like to open the discussion of contributing Pulsar Flink
connector [0] back to Flink
fferent implementation?
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-9641
> > [2] https://issues.apache.org/jira/browse/FLINK-9168
> >
> > Best
> > Yun Tang
> >
> > From: Yijie Shen
> > Sent: T
> Sent: Tuesday, September 3, 2019 13:57
> To: dev@flink.apache.org
> Subject: [DISCUSS] Contribute Pulsar Flink connector back to Flink
>
> Dear Flink Community!
>
> I would like to open the discussion of contributing Pulsar Flink
> connector [0] back to Flink.
>
> ## A
: Tuesday, September 3, 2019 13:57
To: dev@flink.apache.org
Subject: [DISCUSS] Contribute Pulsar Flink connector back to Flink
Dear Flink Community!
I would like to open the discussion of contributing Pulsar Flink
connector [0] back to Flink.
## A brief introduction to Apache Pulsar
Apache Pulsar
Dear Flink Community!
I would like to open the discussion of contributing Pulsar Flink
connector [0] back to Flink.
## A brief introduction to Apache Pulsar
Apache Pulsar[1] is a multi-tenant, high-performance distributed
pub-sub messaging system. Pulsar includes multiple features such as
43 matches
Mail list logo
