Re: Are the Table API Connectors production ready?

[ravi_suryavanshi.yahoo.com via user]

Thank you All.  

On Tuesday, 14 March, 2023 at 07:14:05 am IST, yuxia 
 wrote:  
 
 The plan shows the filters has been pushed down. But remeber, although pused 
down, the filesystem table won't accept the filter. So, it'll be still like 
scan 
all files.

Best regards,
Yuxia
发件人: "Maryam Moafimadani" 
收件人: "Hang Ruan" 
抄送: "yuxia" , "ravi suryavanshi" 
, "Yaroslav Tkachenko" , 
"Shammon FY" , "User" 
发送时间: 星期一, 2023年 3 月 13日 下午 10:07:57
主题: Re: Are the Table API Connectors production ready?

Hi All,It's exciting to see file filtering in the plan for development. I am 
curious whether the following query on a filesystem connector would actually 
push down the filter on metadata `file.path`?

Select score, `file.path` from MyUserTable WHERE `file.path` LIKE '%prefix_%' 

== Optimized Execution Plan ==
Calc(select=[score, file.path], where=[LIKE(file.path, '%2022070611284%')])
+- TableSourceScan(table=[[default_catalog, default_database, MyUserTable, 
filter=[LIKE(file.path, _UTF-16LE'%2022070611284%')]]], fields=[score, 
file.path])

Thanks,Maryam
On Mon, Mar 13, 2023 at 8:55 AM Hang Ruan  wrote:

Hi, yuxia,I would like to help to complete this task.
Best,Hang
yuxia  于2023年3月13日周一 09:32写道：

Yeah, you're right. We don't provide filtering files with patterns. And 
actually we had already a jira[1] for it.
I was intended to do this in the past, but don't have much time.  Anyone who 
are insterested can take it over. We're 
happy to help review.

[1] https://issues.apache.org/jira/browse/FLINK-17398

Best regards,
Yuxia
发件人: "User" 
收件人: "Yaroslav Tkachenko" , "Shammon FY" 

抄送: "User" 
发送时间: 星期一, 2023年 3 月 13日 上午 12:36:46
主题: Re: Are the Table API Connectors production ready?

 Thanks a lot, Yaroslav and Shammon.I want to use the Filesystem Connector.  I 
tried it works well till it is running. If the job is restarted. It processes 
all the files again.
Could not find the move or delete option after collecting the files. Also, I 
could not find the filtering using patterns.
Pattern matching is required as different files exist in the same folder.
Regards,RaviOn Friday, 10 March, 2023 at 05:47:27 am IST, Shammon FY 
 wrote:  
 
 Hi Ravi
Agree with Yaroslav and if you find any problems in use, you can create an 
issue in jira https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK 
. I have used kafka/jdbc/hive in production too, they work well.
Best,Shammon
On Fri, Mar 10, 2023 at 1:42 AM Yaroslav Tkachenko  wrote:

Hi Ravi,
All of them should be production ready. I've personally used half of them in 
production.
Do you have any specific concerns? 
On Thu, Mar 9, 2023 at 9:39 AM ravi_suryavanshi.yahoo.com via user 
 wrote:

 Hi,Can anyone help me here?
Thanks and regards,Ravi
On Monday, 27 February, 2023 at 09:33:18 am IST, ravi_suryavanshi.yahoo.com 
via user  wrote:  
 
 Hi Team,

In Flink 1.16.0, we would like to use some of the Table API Connectors for 
production. Kindly let me know if the below connectors are production ready or 
only for testing purposes.

| Name | Version | Source | Sink |
| Filesystem |  | Bounded and Unbounded Scan, Lookup | Streaming Sink, Batch 
Sink |
| Elasticsearch | 6.x & 7.x | Not supported | Streaming Sink, Batch Sink |
| Opensearch | 1.x & 2.x | Not supported | Streaming Sink, Batch Sink |
| Apache Kafka | 0.10+ | Unbounded Scan | Streaming Sink, Batch Sink |
| Amazon DynamoDB |  | Not supported | Streaming Sink, Batch Sink |
| Amazon Kinesis Data Streams |  | Unbounded Scan | Streaming Sink |
| Amazon Kinesis Data Firehose |  | Not supported | Streaming Sink |
| JDBC |  | Bounded Scan, Lookup | Streaming Sink, Batch Sink |
| Apache HBase | 1.4.x & 2.2.x | Bounded Scan, Lookup | Streaming Sink, Batch 
Sink |
| Apache Hive |


Thanks and regards
  

  




-- 
Maryam MoafimadaniSenior Data Developer @Shopify
  

Re: Are the Table API Connectors production ready?

[yuxia]

The plan shows the filters has been pushed down. But remeber, although pused 
down, the filesystem table won't accept the filter. So, it'll be still like 
scan 
all files. 

Best regards, 
Yuxia 


发件人: "Maryam Moafimadani"  
收件人: "Hang Ruan"  
抄送: "yuxia" , "ravi suryavanshi" 
, "Yaroslav Tkachenko" , 
"Shammon FY" , "User"  
发送时间: 星期一, 2023年 3 月 13日 下午 10:07:57 
主题: Re: Are the Table API Connectors production ready? 

Hi All, 
It's exciting to see file filtering in the plan for development. I am curious 
whether the following query on a filesystem connector would actually push down 
the filter on metadata `file.path`? 

Select score, `file.path` from MyUserTable WHERE `file.path` LIKE '%prefix_%' 

== Optimized Execution Plan == 
Calc(select=[score, file.path], where=[LIKE(file.path, '%2022070611284%')]) 
+- TableSourceScan(table=[[default_catalog, default_database, MyUserTable, 
filter=[LIKE(file.path, _UTF-16LE'%2022070611284%')]]], fields=[score, 
file.path]) 

Thanks, 
Maryam 

On Mon, Mar 13, 2023 at 8:55 AM Hang Ruan < [ mailto:ruanhang1...@gmail.com | 
ruanhang1...@gmail.com ] > wrote: 



Hi, yuxia, 
I would like to help to complete this task. 

Best, 
Hang 

yuxia < [ mailto:luoyu...@alumni.sjtu.edu.cn | luoyu...@alumni.sjtu.edu.cn ] > 
于2023年3月13日周一 09:32写道： 

BQ_BEGIN

Yeah, you're right. We don't provide filtering files with patterns. And 
actually we had already a jira[1] for it. 
I was intended to do this in the past, but don't have much time. Anyone who are 
insterested can take it over. We're 
happy to help review. 

[1] [ https://issues.apache.org/jira/browse/FLINK-17398 | 
https://issues.apache.org/jira/browse/FLINK-17398 ] 

Best regards, 
Yuxia 


发件人: "User" < [ mailto:user@flink.apache.org | user@flink.apache.org ] > 
收件人: "Yaroslav Tkachenko" < [ mailto:yaros...@goldsky.com | 
yaros...@goldsky.com ] >, "Shammon FY" < [ mailto:zjur...@gmail.com | 
zjur...@gmail.com ] > 
抄送: "User" < [ mailto:user@flink.apache.org | user@flink.apache.org ] > 
发送时间: 星期一, 2023年 3 月 13日 上午 12:36:46 
主题: Re: Are the Table API Connectors production ready? 

Thanks a lot, Yaroslav and Shammon. 
I want to use the Filesystem Connector. I tried it works well till it is 
running. If the job is restarted. It processes all the files again. 

Could not find the move or delete option after collecting the files. Also, I 
could not find the filtering using patterns. 

Pattern matching is required as different files exist in the same folder. 

Regards, 
Ravi 
On Friday, 10 March, 2023 at 05:47:27 am IST, Shammon FY < [ 
mailto:zjur...@gmail.com | zjur...@gmail.com ] > wrote: 


Hi Ravi 

Agree with Yaroslav and if you find any problems in use, you can create an 
issue in jira [ 
https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK | 
https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK ] . I have 
used kafka/jdbc/hive in production too, they work well. 

Best, 
Shammon 

On Fri, Mar 10, 2023 at 1:42 AM Yaroslav Tkachenko < [ 
mailto:yaros...@goldsky.com | yaros...@goldsky.com ] > wrote: 

BQ_BEGIN

Hi Ravi, 

All of them should be production ready. I've personally used half of them in 
production. 

Do you have any specific concerns? 

On Thu, Mar 9, 2023 at 9:39 AM [ http://ravi_suryavanshi.yahoo.com/ | 
ravi_suryavanshi.yahoo.com ] via user < [ mailto:user@flink.apache.org | 
user@flink.apache.org ] > wrote: 

BQ_BEGIN

Hi, 
Can anyone help me here? 

Thanks and regards, 
Ravi 

On Monday, 27 February, 2023 at 09:33:18 am IST, [ 
http://ravi_suryavanshi.yahoo.com/ | ravi_suryavanshi.yahoo.com ] via user < [ 
mailto:user@flink.apache.org | user@flink.apache.org ] > wrote: 


Hi Team, 


In Flink 1.16.0, we would like to use some of the Table API Connectors for 
production. Kindly let me know if the below connectors are production ready or 
only for testing purposes. 

Name Version Source Sink [ 
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/filesystem/
 | Filesystem ]  Bounded and Unbounded Scan, Lookup  Streaming 
Sink, Batch Sink 
[ 
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/elasticsearch/
 | Elasticsearch ] 6.x & 7.x   Not supported   Streaming Sink, Batch 
Sink 
[ 
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/opensearch/
 | Opensearch ]   1.x & 2.x   Not supported   Streaming Sink, Batch Sink 
[ 
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kafka/
 | Apache Kafka ]  0.10+   Unbounded Scan  Streaming Sink, Batch Sink 
[ 
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/dynamodb/
 | Amazon DynamoDB ]Not supported   Streaming Sink, Batch Sink 
[ 
https://nightlies.apache.org/flink/flink-docs-rele

Re: Are the Table API Connectors production ready?

[yuxia]

Thanks Hang for taking it. Assigned to you~ 

Best regards, 
Yuxia 


发件人: "Hang Ruan"  
收件人: "yuxia"  
抄送: "ravi suryavanshi" , "Yaroslav Tkachenko" 
, "Shammon FY" , "User" 
 
发送时间: 星期一, 2023年 3 月 13日 下午 8:54:49 
主题: Re: Are the Table API Connectors production ready? 

Hi, yuxia, 
I would like to help to complete this task. 

Best, 
Hang 

yuxia < [ mailto:luoyu...@alumni.sjtu.edu.cn | luoyu...@alumni.sjtu.edu.cn ] > 
于2023年3月13日周一 09:32写道： 



Yeah, you're right. We don't provide filtering files with patterns. And 
actually we had already a jira[1] for it. 
I was intended to do this in the past, but don't have much time. Anyone who are 
insterested can take it over. We're 
happy to help review. 

[1] [ https://issues.apache.org/jira/browse/FLINK-17398 | 
https://issues.apache.org/jira/browse/FLINK-17398 ] 

Best regards, 
Yuxia 


发件人: "User" < [ mailto:user@flink.apache.org | user@flink.apache.org ] > 
收件人: "Yaroslav Tkachenko" < [ mailto:yaros...@goldsky.com | 
yaros...@goldsky.com ] >, "Shammon FY" < [ mailto:zjur...@gmail.com | 
zjur...@gmail.com ] > 
抄送: "User" < [ mailto:user@flink.apache.org | user@flink.apache.org ] > 
发送时间: 星期一, 2023年 3 月 13日 上午 12:36:46 
主题: Re: Are the Table API Connectors production ready? 

Thanks a lot, Yaroslav and Shammon. 
I want to use the Filesystem Connector. I tried it works well till it is 
running. If the job is restarted. It processes all the files again. 

Could not find the move or delete option after collecting the files. Also, I 
could not find the filtering using patterns. 

Pattern matching is required as different files exist in the same folder. 

Regards, 
Ravi 
On Friday, 10 March, 2023 at 05:47:27 am IST, Shammon FY < [ 
mailto:zjur...@gmail.com | zjur...@gmail.com ] > wrote: 


Hi Ravi 

Agree with Yaroslav and if you find any problems in use, you can create an 
issue in jira [ 
https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK | 
https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK ] . I have 
used kafka/jdbc/hive in production too, they work well. 

Best, 
Shammon 

On Fri, Mar 10, 2023 at 1:42 AM Yaroslav Tkachenko < [ 
mailto:yaros...@goldsky.com | yaros...@goldsky.com ] > wrote: 

BQ_BEGIN

Hi Ravi, 

All of them should be production ready. I've personally used half of them in 
production. 

Do you have any specific concerns? 

On Thu, Mar 9, 2023 at 9:39 AM [ http://ravi_suryavanshi.yahoo.com/ | 
ravi_suryavanshi.yahoo.com ] via user < [ mailto:user@flink.apache.org | 
user@flink.apache.org ] > wrote: 

BQ_BEGIN

Hi, 
Can anyone help me here? 

Thanks and regards, 
Ravi 

On Monday, 27 February, 2023 at 09:33:18 am IST, [ 
http://ravi_suryavanshi.yahoo.com/ | ravi_suryavanshi.yahoo.com ] via user < [ 
mailto:user@flink.apache.org | user@flink.apache.org ] > wrote: 


Hi Team, 


In Flink 1.16.0, we would like to use some of the Table API Connectors for 
production. Kindly let me know if the below connectors are production ready or 
only for testing purposes. 

Name Version Source Sink [ 
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/filesystem/
 | Filesystem ]  Bounded and Unbounded Scan, Lookup  Streaming 
Sink, Batch Sink 
[ 
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/elasticsearch/
 | Elasticsearch ] 6.x & 7.x   Not supported   Streaming Sink, Batch 
Sink 
[ 
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/opensearch/
 | Opensearch ]   1.x & 2.x   Not supported   Streaming Sink, Batch Sink 
[ 
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kafka/
 | Apache Kafka ]  0.10+   Unbounded Scan  Streaming Sink, Batch Sink 
[ 
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/dynamodb/
 | Amazon DynamoDB ]Not supported   Streaming Sink, Batch Sink 
[ 
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kinesis/
 | Amazon Kinesis Data Streams ] Unbounded Scan  Streaming Sink 
[ 
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/firehose/
 | Amazon Kinesis Data Firehose ]   Not supported   Streaming Sink 
[ 
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/jdbc/
 | JDBC ]   Bounded Scan, LookupStreaming Sink, Batch Sink 
[ 
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hbase/
 | Apache HBase ]  1.4.x & 2.2.x   Bounded Scan, LookupStreaming Sink, 
Batch Sink 
[ 
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hive/overview/
 | Apache Hive ] 

Thanks and regards 




BQ_END


BQ_END

Re: Are the Table API Connectors production ready?

[Maryam Moafimadani via user]

Hi All,
It's exciting to see file filtering in the plan for development. I am
curious whether the following query on a filesystem connector would
actually push down the filter on metadata `file.path`?

*Select score, `file.path` from MyUserTable WHERE `file.path` LIKE
'%prefix_%' *

== Optimized Execution Plan ==
Calc(select=[score, file.path], where=[LIKE(file.path, '%2022070611284%')])
+- TableSourceScan(table=[[default_catalog, default_database, MyUserTable,
filter=[LIKE(file.path, _UTF-16LE'%2022070611284%')]]], fields=[score,
file.path])

Thanks,
Maryam

On Mon, Mar 13, 2023 at 8:55 AM Hang Ruan  wrote:

> Hi, yuxia,
> I would like to help to complete this task.
>
> Best,
> Hang
>
> yuxia  于2023年3月13日周一 09:32写道：
>
>> Yeah, you're right. We don't provide filtering files with patterns. And
>> actually we had already a jira[1] for it.
>> I was intended to do this in the past, but don't have much time.  Anyone
>> who are insterested can take it over. We're
>> happy to help review.
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-17398
>>
>> Best regards,
>> Yuxia
>>
>> --
>> *发件人: *"User" 
>> *收件人: *"Yaroslav Tkachenko" , "Shammon FY" <
>> zjur...@gmail.com>
>> *抄送: *"User" 
>> *发送时间: *星期一, 2023年 3 月 13日 上午 12:36:46
>> *主题: *Re: Are the Table API Connectors production ready?
>>
>> Thanks a lot, Yaroslav and Shammon.
>> I want to use the Filesystem Connector.  I tried it works well till it is
>> running. If the job is restarted. It processes all the files again.
>>
>> Could not find the move or delete option after collecting the files.
>> Also, I could not find the filtering using patterns.
>>
>> Pattern matching is required as different files exist in the same folder.
>>
>> Regards,
>> Ravi
>> On Friday, 10 March, 2023 at 05:47:27 am IST, Shammon FY <
>> zjur...@gmail.com> wrote:
>>
>>
>> Hi Ravi
>>
>> Agree with Yaroslav and if you find any problems in use, you can create
>> an issue in jira
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK . I
>> have used kafka/jdbc/hive in production too, they work well.
>>
>> Best,
>> Shammon
>>
>> On Fri, Mar 10, 2023 at 1:42 AM Yaroslav Tkachenko 
>> wrote:
>>
>> Hi Ravi,
>>
>> All of them should be production ready. I've personally used half of them
>> in production.
>>
>> Do you have any specific concerns?
>>
>> On Thu, Mar 9, 2023 at 9:39 AM ravi_suryavanshi.yahoo.com via user <
>> user@flink.apache.org> wrote:
>>
>> Hi,
>> Can anyone help me here?
>>
>> Thanks and regards,
>> Ravi
>>
>> On Monday, 27 February, 2023 at 09:33:18 am IST,
>> ravi_suryavanshi.yahoo.com via user  wrote:
>>
>>
>> Hi Team,
>>
>>
>> In Flink 1.16.0, we would like to use some of the Table API Connectors
>> for production. Kindly let me know if the below connectors are production
>> ready or only for testing purposes.
>>
>> NameVersionSourceSink
>> Filesystem
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/filesystem/>
>>  Bounded
>> and Unbounded Scan, Lookup Streaming Sink, Batch Sink
>> Elasticsearch
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/elasticsearch/>
>>  6.x
>> & 7.x Not supported Streaming Sink, Batch Sink
>> Opensearch
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/opensearch/>
>>  1.x
>> & 2.x Not supported Streaming Sink, Batch Sink
>> Apache Kafka
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kafka/>
>> 0.10+ Unbounded Scan Streaming Sink, Batch Sink
>> Amazon DynamoDB
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/dynamodb/>
>>  Not
>> supported Streaming Sink, Batch Sink
>> Amazon Kinesis Data Streams
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kinesis/>
>>  Unbounded
>> Scan Streaming Sink
>> Amazon Kinesis Data Firehose
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/firehose/>
>>  Not
>> supported Streaming Sink
>> JDBC
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/jdbc/>
>>  Bounded
>> Scan, Lookup Streaming Sink, Batch Sink
>> Apache HBase
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hbase/>
>>  1.4.x
>> & 2.2.x Bounded Scan, Lookup Streaming Sink, Batch Sink
>> Apache Hive
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hive/overview/>
>>
>> Thanks and regards
>>
>>
>>

-- 
Maryam Moafimadani
Senior Data Developer @Shopify <http://www.shopify.com/>

Re: Are the Table API Connectors production ready?

[Hang Ruan]

Hi, yuxia,
I would like to help to complete this task.

Best,
Hang

yuxia  于2023年3月13日周一 09:32写道：

> Yeah, you're right. We don't provide filtering files with patterns. And
> actually we had already a jira[1] for it.
> I was intended to do this in the past, but don't have much time.  Anyone
> who are insterested can take it over. We're
> happy to help review.
>
> [1] https://issues.apache.org/jira/browse/FLINK-17398
>
> Best regards,
> Yuxia
>
> --
> *发件人: *"User" 
> *收件人: *"Yaroslav Tkachenko" , "Shammon FY" <
> zjur...@gmail.com>
> *抄送: *"User" 
> *发送时间: *星期一, 2023年 3 月 13日 上午 12:36:46
> *主题: *Re: Are the Table API Connectors production ready?
>
> Thanks a lot, Yaroslav and Shammon.
> I want to use the Filesystem Connector.  I tried it works well till it is
> running. If the job is restarted. It processes all the files again.
>
> Could not find the move or delete option after collecting the files. Also,
> I could not find the filtering using patterns.
>
> Pattern matching is required as different files exist in the same folder.
>
> Regards,
> Ravi
> On Friday, 10 March, 2023 at 05:47:27 am IST, Shammon FY <
> zjur...@gmail.com> wrote:
>
>
> Hi Ravi
>
> Agree with Yaroslav and if you find any problems in use, you can create an
> issue in jira
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK . I have
> used kafka/jdbc/hive in production too, they work well.
>
> Best,
> Shammon
>
> On Fri, Mar 10, 2023 at 1:42 AM Yaroslav Tkachenko 
> wrote:
>
> Hi Ravi,
>
> All of them should be production ready. I've personally used half of them
> in production.
>
> Do you have any specific concerns?
>
> On Thu, Mar 9, 2023 at 9:39 AM ravi_suryavanshi.yahoo.com via user <
> user@flink.apache.org> wrote:
>
> Hi,
> Can anyone help me here?
>
> Thanks and regards,
> Ravi
>
> On Monday, 27 February, 2023 at 09:33:18 am IST,
> ravi_suryavanshi.yahoo.com via user  wrote:
>
>
> Hi Team,
>
>
> In Flink 1.16.0, we would like to use some of the Table API Connectors for
> production. Kindly let me know if the below connectors are production ready
> or only for testing purposes.
>
> NameVersionSourceSink
> Filesystem
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/filesystem/>
>  Bounded
> and Unbounded Scan, Lookup Streaming Sink, Batch Sink
> Elasticsearch
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/elasticsearch/>
>  6.x
> & 7.x Not supported Streaming Sink, Batch Sink
> Opensearch
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/opensearch/>
>  1.x
> & 2.x Not supported Streaming Sink, Batch Sink
> Apache Kafka
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kafka/>
> 0.10+ Unbounded Scan Streaming Sink, Batch Sink
> Amazon DynamoDB
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/dynamodb/>
>  Not
> supported Streaming Sink, Batch Sink
> Amazon Kinesis Data Streams
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kinesis/>
>  Unbounded
> Scan Streaming Sink
> Amazon Kinesis Data Firehose
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/firehose/>
>  Not
> supported Streaming Sink
> JDBC
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/jdbc/>
>  Bounded
> Scan, Lookup Streaming Sink, Batch Sink
> Apache HBase
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hbase/>
>  1.4.x
> & 2.2.x Bounded Scan, Lookup Streaming Sink, Batch Sink
> Apache Hive
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hive/overview/>
>
> Thanks and regards
>
>
>

Re: Are the Table API Connectors production ready?

[yuxia]

Yeah, you're right. We don't provide filtering files with patterns. And 
actually we had already a jira[1] for it. 
I was intended to do this in the past, but don't have much time. Anyone who are 
insterested can take it over. We're 
happy to help review. 

[1] https://issues.apache.org/jira/browse/FLINK-17398 

Best regards, 
Yuxia 


发件人: "User"  
收件人: "Yaroslav Tkachenko" , "Shammon FY" 
 
抄送: "User"  
发送时间: 星期一, 2023年 3 月 13日 上午 12:36:46 
主题: Re: Are the Table API Connectors production ready? 

Thanks a lot, Yaroslav and Shammon. 
I want to use the Filesystem Connector. I tried it works well till it is 
running. If the job is restarted. It processes all the files again. 

Could not find the move or delete option after collecting the files. Also, I 
could not find the filtering using patterns. 

Pattern matching is required as different files exist in the same folder. 

Regards, 
Ravi 
On Friday, 10 March, 2023 at 05:47:27 am IST, Shammon FY  
wrote: 


Hi Ravi 

Agree with Yaroslav and if you find any problems in use, you can create an 
issue in jira [ 
https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK | 
https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK ] . I have 
used kafka/jdbc/hive in production too, they work well. 

Best, 
Shammon 

On Fri, Mar 10, 2023 at 1:42 AM Yaroslav Tkachenko < [ 
mailto:yaros...@goldsky.com | yaros...@goldsky.com ] > wrote: 



Hi Ravi, 

All of them should be production ready. I've personally used half of them in 
production. 

Do you have any specific concerns? 

On Thu, Mar 9, 2023 at 9:39 AM [ http://ravi_suryavanshi.yahoo.com/ | 
ravi_suryavanshi.yahoo.com ] via user < [ mailto:user@flink.apache.org | 
user@flink.apache.org ] > wrote: 

BQ_BEGIN

Hi, 
Can anyone help me here? 

Thanks and regards, 
Ravi 

On Monday, 27 February, 2023 at 09:33:18 am IST, [ 
http://ravi_suryavanshi.yahoo.com/ | ravi_suryavanshi.yahoo.com ] via user < [ 
mailto:user@flink.apache.org | user@flink.apache.org ] > wrote: 


Hi Team, 


In Flink 1.16.0, we would like to use some of the Table API Connectors for 
production. Kindly let me know if the below connectors are production ready or 
only for testing purposes. 

Name Version Source Sink [ 
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/filesystem/
 | Filesystem ]  Bounded and Unbounded Scan, Lookup  Streaming 
Sink, Batch Sink 
[ 
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/elasticsearch/
 | Elasticsearch ] 6.x & 7.x   Not supported   Streaming Sink, Batch 
Sink 
[ 
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/opensearch/
 | Opensearch ]   1.x & 2.x   Not supported   Streaming Sink, Batch Sink 
[ 
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kafka/
 | Apache Kafka ]  0.10+   Unbounded Scan  Streaming Sink, Batch Sink 
[ 
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/dynamodb/
 | Amazon DynamoDB ]Not supported   Streaming Sink, Batch Sink 
[ 
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kinesis/
 | Amazon Kinesis Data Streams ] Unbounded Scan  Streaming Sink 
[ 
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/firehose/
 | Amazon Kinesis Data Firehose ]   Not supported   Streaming Sink 
[ 
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/jdbc/
 | JDBC ]   Bounded Scan, LookupStreaming Sink, Batch Sink 
[ 
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hbase/
 | Apache HBase ]  1.4.x & 2.2.x   Bounded Scan, LookupStreaming Sink, 
Batch Sink 
[ 
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hive/overview/
 | Apache Hive ] 

Thanks and regards 




BQ_END

Re: Are the Table API Connectors production ready?

[Shammon FY]

Hi ravi

Do you read streaming data with filesystem connector? By default, the file
system is bound. If you need to read streaming data, you can look at the
configuration `source.monitor.interval` in
https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/filesystem/#directory-watching,
and see whether your job is executed in streaming mode and trigger
checkpoint regularly.

Best,
Shammon


On Mon, Mar 13, 2023 at 12:36 AM Shammon FY  wrote:

>
>

Re: Are the Table API Connectors production ready?

[ravi_suryavanshi.yahoo.com via user]

 Thanks a lot, Yaroslav and Shammon.I want to use the Filesystem Connector.  I 
tried it works well till it is running. If the job is restarted. It processes 
all the files again.
Could not find the move or delete option after collecting the files. Also, I 
could not find the filtering using patterns.
Pattern matching is required as different files exist in the same folder.
Regards,RaviOn Friday, 10 March, 2023 at 05:47:27 am IST, Shammon FY 
 wrote:  
 
 Hi Ravi
Agree with Yaroslav and if you find any problems in use, you can create an 
issue in jira https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK 
. I have used kafka/jdbc/hive in production too, they work well.
Best,Shammon
On Fri, Mar 10, 2023 at 1:42 AM Yaroslav Tkachenko  wrote:

Hi Ravi,
All of them should be production ready. I've personally used half of them in 
production.
Do you have any specific concerns? 
On Thu, Mar 9, 2023 at 9:39 AM ravi_suryavanshi.yahoo.com via user 
 wrote:

 Hi,Can anyone help me here?
Thanks and regards,Ravi
On Monday, 27 February, 2023 at 09:33:18 am IST, ravi_suryavanshi.yahoo.com 
via user  wrote:  
 
 Hi Team,

In Flink 1.16.0, we would like to use some of the Table API Connectors for 
production. Kindly let me know if the below connectors are production ready or 
only for testing purposes.

| Name | Version | Source | Sink |
| Filesystem |  | Bounded and Unbounded Scan, Lookup | Streaming Sink, Batch 
Sink |
| Elasticsearch | 6.x & 7.x | Not supported | Streaming Sink, Batch Sink |
| Opensearch | 1.x & 2.x | Not supported | Streaming Sink, Batch Sink |
| Apache Kafka | 0.10+ | Unbounded Scan | Streaming Sink, Batch Sink |
| Amazon DynamoDB |  | Not supported | Streaming Sink, Batch Sink |
| Amazon Kinesis Data Streams |  | Unbounded Scan | Streaming Sink |
| Amazon Kinesis Data Firehose |  | Not supported | Streaming Sink |
| JDBC |  | Bounded Scan, Lookup | Streaming Sink, Batch Sink |
| Apache HBase | 1.4.x & 2.2.x | Bounded Scan, Lookup | Streaming Sink, Batch 
Sink |
| Apache Hive |


Thanks and regards
  

  

Re: Are the Table API Connectors production ready?

[Shammon FY]

Hi Ravi

Agree with Yaroslav and if you find any problems in use, you can create an
issue in jira
https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK . I have
used kafka/jdbc/hive in production too, they work well.

Best,
Shammon

On Fri, Mar 10, 2023 at 1:42 AM Yaroslav Tkachenko 
wrote:

> Hi Ravi,
>
> All of them should be production ready. I've personally used half of them
> in production.
>
> Do you have any specific concerns?
>
> On Thu, Mar 9, 2023 at 9:39 AM ravi_suryavanshi.yahoo.com via user <
> user@flink.apache.org> wrote:
>
>> Hi,
>> Can anyone help me here?
>>
>> Thanks and regards,
>> Ravi
>>
>> On Monday, 27 February, 2023 at 09:33:18 am IST,
>> ravi_suryavanshi.yahoo.com via user  wrote:
>>
>>
>> Hi Team,
>>
>>
>> In Flink 1.16.0, we would like to use some of the Table API Connectors
>> for production. Kindly let me know if the below connectors are production
>> ready or only for testing purposes.
>>
>> NameVersionSourceSink
>> Filesystem
>> 
>>  Bounded
>> and Unbounded Scan, Lookup Streaming Sink, Batch Sink
>> Elasticsearch
>> 
>>  6.x
>> & 7.x Not supported Streaming Sink, Batch Sink
>> Opensearch
>> 
>>  1.x
>> & 2.x Not supported Streaming Sink, Batch Sink
>> Apache Kafka
>> 
>> 0.10+ Unbounded Scan Streaming Sink, Batch Sink
>> Amazon DynamoDB
>> 
>>  Not
>> supported Streaming Sink, Batch Sink
>> Amazon Kinesis Data Streams
>> 
>>  Unbounded
>> Scan Streaming Sink
>> Amazon Kinesis Data Firehose
>> 
>>  Not
>> supported Streaming Sink
>> JDBC
>> 
>>  Bounded
>> Scan, Lookup Streaming Sink, Batch Sink
>> Apache HBase
>> 
>>  1.4.x
>> & 2.2.x Bounded Scan, Lookup Streaming Sink, Batch Sink
>> Apache Hive
>> 
>>
>> Thanks and regards
>>
>

Re: Are the Table API Connectors production ready?

[Yaroslav Tkachenko]

Hi Ravi,

All of them should be production ready. I've personally used half of them
in production.

Do you have any specific concerns?

On Thu, Mar 9, 2023 at 9:39 AM ravi_suryavanshi.yahoo.com via user <
user@flink.apache.org> wrote:

> Hi,
> Can anyone help me here?
>
> Thanks and regards,
> Ravi
>
> On Monday, 27 February, 2023 at 09:33:18 am IST,
> ravi_suryavanshi.yahoo.com via user  wrote:
>
>
> Hi Team,
>
>
> In Flink 1.16.0, we would like to use some of the Table API Connectors for
> production. Kindly let me know if the below connectors are production ready
> or only for testing purposes.
>
> NameVersionSourceSink
> Filesystem
> 
>  Bounded
> and Unbounded Scan, Lookup Streaming Sink, Batch Sink
> Elasticsearch
> 
>  6.x
> & 7.x Not supported Streaming Sink, Batch Sink
> Opensearch
> 
>  1.x
> & 2.x Not supported Streaming Sink, Batch Sink
> Apache Kafka
> 
> 0.10+ Unbounded Scan Streaming Sink, Batch Sink
> Amazon DynamoDB
> 
>  Not
> supported Streaming Sink, Batch Sink
> Amazon Kinesis Data Streams
> 
>  Unbounded
> Scan Streaming Sink
> Amazon Kinesis Data Firehose
> 
>  Not
> supported Streaming Sink
> JDBC
> 
>  Bounded
> Scan, Lookup Streaming Sink, Batch Sink
> Apache HBase
> 
>  1.4.x
> & 2.2.x Bounded Scan, Lookup Streaming Sink, Batch Sink
> Apache Hive
> 
>
> Thanks and regards
>

Re: Are the Table API Connectors production ready?

[ravi_suryavanshi.yahoo.com via user]

 Hi,Can anyone help me here?
Thanks and regards,Ravi
On Monday, 27 February, 2023 at 09:33:18 am IST, ravi_suryavanshi.yahoo.com 
via user  wrote:  
 
 Hi Team,

In Flink 1.16.0, we would like to use some of the Table API Connectors for 
production. Kindly let me know if the below connectors are production ready or 
only for testing purposes.

| Name | Version | Source | Sink |
| Filesystem |  | Bounded and Unbounded Scan, Lookup | Streaming Sink, Batch 
Sink |
| Elasticsearch | 6.x & 7.x | Not supported | Streaming Sink, Batch Sink |
| Opensearch | 1.x & 2.x | Not supported | Streaming Sink, Batch Sink |
| Apache Kafka | 0.10+ | Unbounded Scan | Streaming Sink, Batch Sink |
| Amazon DynamoDB |  | Not supported | Streaming Sink, Batch Sink |
| Amazon Kinesis Data Streams |  | Unbounded Scan | Streaming Sink |
| Amazon Kinesis Data Firehose |  | Not supported | Streaming Sink |
| JDBC |  | Bounded Scan, Lookup | Streaming Sink, Batch Sink |
| Apache HBase | 1.4.x & 2.2.x | Bounded Scan, Lookup | Streaming Sink, Batch 
Sink |
| Apache Hive |


Thanks and regards
  

Are the Table API Connectors production ready?

[ravi_suryavanshi.yahoo.com via user]

Hi Team,

In Flink 1.16.0, we would like to use some of the Table API Connectors for 
production. Kindly let me know if the below connectors are production ready or 
only for testing purposes.

| Name | Version | Source | Sink |
| Filesystem |  | Bounded and Unbounded Scan, Lookup | Streaming Sink, Batch 
Sink |
| Elasticsearch | 6.x & 7.x | Not supported | Streaming Sink, Batch Sink |
| Opensearch | 1.x & 2.x | Not supported | Streaming Sink, Batch Sink |
| Apache Kafka | 0.10+ | Unbounded Scan | Streaming Sink, Batch Sink |
| Amazon DynamoDB |  | Not supported | Streaming Sink, Batch Sink |
| Amazon Kinesis Data Streams |  | Unbounded Scan | Streaming Sink |
| Amazon Kinesis Data Firehose |  | Not supported | Streaming Sink |
| JDBC |  | Bounded Scan, Lookup | Streaming Sink, Batch Sink |
| Apache HBase | 1.4.x & 2.2.x | Bounded Scan, Lookup | Streaming Sink, Batch 
Sink |
| Apache Hive |


Thanks and regards