[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608738#comment-17608738 ] Jiabao Sun commented on FLINK-6573: --- Hi there, The discussion thread was opened on [https://lists.apache.org/thread/bhzj70t9g6ofdk8hqtfjjlxqnl0l4xwn.] Looking forward to any comments or feedback. ;) > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Assignee: Jiabao Sun >Priority: Not a Priority > Labels: pull-request-available, stale-assigned > Attachments: image-2021-11-15-14-41-07-514.png > > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608303#comment-17608303 ] Martijn Visser commented on FLINK-6573: --- [~sarkaramr...@gmail.com] There's no timeline. I would expect that the FLIP can be discussed in the next couple of days, then it can be voted on. That would probably be 2 weeks or so. When that's done, the connector code will be made available in its own repository and released independently of Flink itself. > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Assignee: Jiabao Sun >Priority: Not a Priority > Labels: pull-request-available, stale-assigned > Attachments: image-2021-11-15-14-41-07-514.png > > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608225#comment-17608225 ] Amrit Sarkar commented on FLINK-6573: - Thanks everyone for working on this connector, we are looking forward to it as a team at our workplace. Is there any sort of timelines we have for this particular JIRA to be completed, and ultimately GA? > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Assignee: Jiabao Sun >Priority: Not a Priority > Labels: pull-request-available, stale-assigned > Attachments: image-2021-11-15-14-41-07-514.png > > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608051#comment-17608051 ] Leonard Xu commented on FLINK-6573: --- I've assigned this ticket to [~jiabao.sun] as he has start the great work. > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Assignee: Jiabao Sun >Priority: Not a Priority > Labels: pull-request-available, stale-assigned > Attachments: image-2021-11-15-14-41-07-514.png > > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17607744#comment-17607744 ] Jiabao Sun commented on FLINK-6573: --- Thanks [~chesnay] , the format was updated to conform with the template. > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Assignee: ZhuoYu Chen >Priority: Not a Priority > Labels: pull-request-available, stale-assigned > Attachments: image-2021-11-15-14-41-07-514.png > > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17607733#comment-17607733 ] Chesnay Schepler commented on FLINK-6573: - [~jiabao.sun] Please update your FLIP to conform with the FLIP template: https://cwiki.apache.org/confluence/display/FLINK/FLIP+Connector+Template > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Assignee: ZhuoYu Chen >Priority: Not a Priority > Labels: pull-request-available, stale-assigned > Attachments: image-2021-11-15-14-41-07-514.png > > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17607727#comment-17607727 ] Jiabao Sun commented on FLINK-6573: --- The FLIP-262 draft has been added on [https://cwiki.apache.org/confluence/display/FLINK/FLIP-262%3A+Introduce+MongoDB+connector] There's also a copy on [https://docs.google.com/document/d/1OmMh6Yf6wPEcWlQOP0IMAIyvByf-CFDooV-m6_xztrE/edit?usp=sharing] [~martijnvisser] Please help review it when you have time. > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Assignee: ZhuoYu Chen >Priority: Not a Priority > Labels: pull-request-available, stale-assigned > Attachments: image-2021-11-15-14-41-07-514.png > > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17607552#comment-17607552 ] Martijn Visser commented on FLINK-6573: --- [~jiabao.sun] Great to hear that! I've granted you permissions to add a FLIP, you can read up on the process on https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals. You can use https://cwiki.apache.org/confluence/display/FLINK/FLIP+Connector+Template as a template and https://cwiki.apache.org/confluence/display/FLINK/FLIP-252%3A+Amazon+DynamoDB+Sink+Connector as an example. Looking forward to your contribution! > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Assignee: ZhuoYu Chen >Priority: Not a Priority > Labels: pull-request-available, stale-assigned > Attachments: image-2021-11-15-14-41-07-514.png > > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17607465#comment-17607465 ] Jiabao Sun commented on FLINK-6573: --- Thanks [~martijnvisser] for the feedback I just noticed the mongo-flink repository seems to be implemented via new Source and Sink interfaces in recent commits. Compared to it, the PR-20848 provides more powerful features such as parallel read and write and lookup abilities. If this makes sense I'm willing to created a FLIP to introduce it. > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Assignee: ZhuoYu Chen >Priority: Not a Priority > Labels: pull-request-available, stale-assigned > Attachments: image-2021-11-15-14-41-07-514.png > > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17607404#comment-17607404 ] Martijn Visser commented on FLINK-6573: --- [~arvid] Thanks for the ping [~jiabao.sun] Thanks for reaching out! Great to hear that you would like to introduce a new connector to the Flink ecosystem. In order to introduce a new connector, we'll first need to go through the FLIP process. See https://cwiki.apache.org/confluence/display/FLINK/FLIP-252%3A+Amazon+DynamoDB+Sink+Connector as an example. For MongoDB, a new FLIP would need to be created, discussed and voted on. When the vote has passed, we can create a new repository (like github.com/apache/flink-connector-mongodb) where the source code for that connector can be stored. New connectors aren't currently being merged in Flink's main repo. One question, as I see there's also https://github.com/mongo-flink/mongo-flink - Are you familiar with that connector? > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Assignee: ZhuoYu Chen >Priority: Not a Priority > Labels: pull-request-available, stale-assigned > Attachments: image-2021-11-15-14-41-07-514.png > > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17607298#comment-17607298 ] Jiabao Sun commented on FLINK-6573: --- Thanks [~arvid]. Looking forward to getting some advise from [~martijnvisser] . > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Assignee: ZhuoYu Chen >Priority: Not a Priority > Labels: pull-request-available, stale-assigned > Attachments: image-2021-11-15-14-41-07-514.png > > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606504#comment-17606504 ] Arvid Heise commented on FLINK-6573: [~martijnvisser] [~MartijnVisser] is probably the right person for guiding the new connector. He is currently pushing for the different external connector repos. @Martijn, could you please champion this effort? (Side note: you should get your apache accounts merged through INFRA ;)). > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Assignee: ZhuoYu Chen >Priority: Not a Priority > Labels: pull-request-available, stale-assigned > Attachments: image-2021-11-15-14-41-07-514.png > > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606488#comment-17606488 ] Jiabao Sun commented on FLINK-6573: --- Hi [~arvid], are we going to push this forward? > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Assignee: ZhuoYu Chen >Priority: Not a Priority > Labels: pull-request-available, stale-assigned > Attachments: image-2021-11-15-14-41-07-514.png > > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606024#comment-17606024 ] Jiabao Sun commented on FLINK-6573: --- I noticed that this ticked is reopened. So I submit a new PR 20848 to support MongoDB stream and sql connector by new Source and Sink interface. * Support parallel read and write. * Support lookup table source. * Support scan table source. * Support push limit down. * Support push projection down. > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Assignee: ZhuoYu Chen >Priority: Not a Priority > Labels: pull-request-available, stale-assigned > Attachments: image-2021-11-15-14-41-07-514.png > > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447000#comment-17447000 ] Lai Dai commented on FLINK-6573: [~monster#12] Hi, Could you tell when to open a pr for this mongodb connector > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Assignee: ZhuoYu Chen >Priority: Not a Priority > Labels: stale-assigned > Attachments: image-2021-11-15-14-41-07-514.png > > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[
https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445175#comment-17445175
]
ZhuoYu Chen commented on FLINK-6573:
{color:#4a90e2}[~arvid] {color:#33}Ok, I will submit the code this
Sunday{color}{color}
> Flink MongoDB Connector
> ---
>
> Key: FLINK-6573
> URL: https://issues.apache.org/jira/browse/FLINK-6573
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / Common
>Affects Versions: 1.2.0
> Environment: Linux Operating System, Mongo DB
>Reporter: Nagamallikarjuna
>Assignee: ZhuoYu Chen
>Priority: Not a Priority
> Labels: stale-assigned
> Attachments: image-2021-11-15-14-41-07-514.png
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> Hi Community,
> Currently we are using Flink in the current Project. We have huge amount of
> data to process using Flink which resides in Mongo DB. We have a requirement
> of parallel data connectivity in between Flink and Mongo DB for both
> reads/writes. Currently we are planning to create this connector and
> contribute to the Community.
> I will update the further details once I receive your feedback
> Please let us know if you have any concerns.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17443605#comment-17443605 ] Arvid Heise commented on FLINK-6573: That's good looking already :) Looking forward to your PR. We will work on the external connector repo this week and I can give an update if it's ready for contributions by then. > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Assignee: ZhuoYu Chen >Priority: Not a Priority > Labels: stale-assigned > Attachments: image-2021-11-15-14-41-07-514.png > > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[
https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17443565#comment-17443565
]
ZhuoYu Chen commented on FLINK-6573:
[~arvid] {color:#33}Thank you for reminding me{color}.{color:#33}I have
completed most of the work, but there are some shortcomings :1.
{color}{color:#33}When the data sink to mongodb is an array,mongodb will
return an exception, and you need to transfer the array to List 2 before sink
data.{color}{color:#33}Specify the primary key 3. {color}{color:#33}The
requirement is to organize the code to meet FLink PR specification{color}
!image-2021-11-15-14-41-07-514.png!
> Flink MongoDB Connector
> ---
>
> Key: FLINK-6573
> URL: https://issues.apache.org/jira/browse/FLINK-6573
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / Common
>Affects Versions: 1.2.0
> Environment: Linux Operating System, Mongo DB
>Reporter: Nagamallikarjuna
>Assignee: ZhuoYu Chen
>Priority: Not a Priority
> Labels: stale-assigned
> Attachments: image-2021-11-15-14-35-04-364.png,
> image-2021-11-15-14-35-40-394.png
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> Hi Community,
> Currently we are using Flink in the current Project. We have huge amount of
> data to process using Flink which resides in Mongo DB. We have a requirement
> of parallel data connectivity in between Flink and Mongo DB for both
> reads/writes. Currently we are planning to create this connector and
> contribute to the Community.
> I will update the further details once I receive your feedback
> Please let us know if you have any concerns.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17443278#comment-17443278 ] Arvid Heise commented on FLINK-6573: I have assigned the ticket to you. Please note that we are currently transitioning connectors from Flink's main repo into https://github.com/apache/flink-connectors . I can't give you an ETA when that repo is ready but we can use your contribution as the lackmus test. The advantage of this repo is that your Mongo connector would be available as soon as it's done and users don't need to wait for 1.15. Please also note that we only accept new connectors written against the unified [source|https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/sources/] and [sink|https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API] interfaces. > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Assignee: ZhuoYu Chen >Priority: Not a Priority > Labels: stale-assigned > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[
https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17443126#comment-17443126
]
ZhuoYu Chen commented on FLINK-6573:
[~arvid] {color:#33}I have completed the implementation, you can assign the
task to me, I will sort out the code, and I will submit PR {color}
> Flink MongoDB Connector
> ---
>
> Key: FLINK-6573
> URL: https://issues.apache.org/jira/browse/FLINK-6573
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / Common
>Affects Versions: 1.2.0
> Environment: Linux Operating System, Mongo DB
>Reporter: Nagamallikarjuna
>Priority: Not a Priority
> Labels: stale-assigned
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> Hi Community,
> Currently we are using Flink in the current Project. We have huge amount of
> data to process using Flink which resides in Mongo DB. We have a requirement
> of parallel data connectivity in between Flink and Mongo DB for both
> reads/writes. Currently we are planning to create this connector and
> contribute to the Community.
> I will update the further details once I receive your feedback
> Please let us know if you have any concerns.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17436902#comment-17436902 ] Jiabao Sun commented on FLINK-6573: --- [~monster#12] If you don’t have time, I think I can help finish it. > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Priority: Not a Priority > Labels: stale-assigned > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17436860#comment-17436860 ] Arvid Heise commented on FLINK-6573: [~jiabao.sun], [~monster#12] could you please coordinate who wants to work on it? > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Priority: Not a Priority > Labels: stale-assigned > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17436857#comment-17436857 ] Arvid Heise commented on FLINK-6573: I'll reassign this ticket. It's fair to assume that [~yew1eb] is not working on it anymore after 4 years. Please note that new connectors have to use the Source and Sink interface to be accepted. > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Assignee: Hai Zhou >Priority: Not a Priority > Labels: stale-assigned > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17436846#comment-17436846 ] Till Rohrmann commented on FLINK-6573: -- cc [~arvid] for potential guidance. > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Assignee: Hai Zhou >Priority: Not a Priority > Labels: stale-assigned > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17429139#comment-17429139 ] Jiabao Sun commented on FLINK-6573: --- Can I take this ticket? > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Assignee: Hai Zhou >Priority: Not a Priority > Labels: stale-assigned > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[
https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17416996#comment-17416996
]
zhuoyu Chen commented on FLINK-6573:
[~yew1eb] {color:#33}Hello, the company I work for is using Flink to
connect Mongo to provide data support for business system,
{color}{color:#33}but there is no official connector, so I have realized a
Mongo connector by myself, {color}{color:#33}and now I want to contribute
it to the community.I hope you can assign this task to me.{color}
> Flink MongoDB Connector
> ---
>
> Key: FLINK-6573
> URL: https://issues.apache.org/jira/browse/FLINK-6573
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / Common
>Affects Versions: 1.2.0
> Environment: Linux Operating System, Mongo DB
>Reporter: Nagamallikarjuna
>Assignee: Hai Zhou
>Priority: Not a Priority
> Labels: stale-assigned
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> Hi Community,
> Currently we are using Flink in the current Project. We have huge amount of
> data to process using Flink which resides in Mongo DB. We have a requirement
> of parallel data connectivity in between Flink and Mongo DB for both
> reads/writes. Currently we are planning to create this connector and
> contribute to the Community.
> I will update the further details once I receive your feedback
> Please let us know if you have any concerns.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329551#comment-17329551 ] Flink Jira Bot commented on FLINK-6573: --- This issue is assigned but has not received an update in 7 days so it has been labeled "stale-assigned". If you are still working on the issue, please give an update and remove the label. If you are no longer working on the issue, please unassign so someone else may work on it. In 7 days the issue will be automatically unassigned. > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Assignee: Hai Zhou >Priority: Minor > Labels: stale-assigned, stale-minor > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17321656#comment-17321656 ] Flink Jira Bot commented on FLINK-6573: --- This issue and all of its Sub-Tasks have not been updated for 180 days. So, it has been labeled "stale-minor". If you are still affected by this bug or are still interested in this issue, please give an update and remove the label. In 7 days the issue will be closed automatically. > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Assignee: Hai Zhou >Priority: Minor > Labels: stale-minor > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16780460#comment-16780460 ] Till Rohrmann commented on FLINK-6573: -- Hi [~yanghua], I would bring this discussion to Flink's dev mailing list to see what responses we get. In general it is good to add more connectors to Flink. However, one always needs to consider the long term maintenance of such an effort. Therefore, it would be good if someone using MongoDB could commit to do that. > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Assignee: Hai Zhou >Priority: Minor > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16779985#comment-16779985 ] vinoyang commented on FLINK-6573: - [~Zentol] and [~till.rohrmann] Do you think this is a valuable proposal to implement a mongodb connector for flink, considering that mongodb is a very common storage component (Flink official website only provides [a demo link|https://ci.apache.org/projects/flink/flink-docs-stable/dev/batch/connectors.html#access-mongodb], I think this is not friendly). If you think this makes sense, is it implemented in flink, or in Bahir? > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Assignee: Hai Zhou >Priority: Minor > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16008853#comment-16008853 ] Greg Hogan commented on FLINK-6573: --- [~naagachowdary] glad to hear of your success and thank you for this contributions! The Apache Flink community is adding new connectors to the [bahir-flink|https://github.com/apache/bahir-flink] repo but please continue to make use of the flink-dev mailing list for any questions or discussion. > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Batch Connectors and Input/Output Formats >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Priority: Minor > Fix For: 2.0.0 > > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
