Hi All, It's exciting to see file filtering in the plan for development. I am curious whether the following query on a filesystem connector would actually push down the filter on metadata `file.path`?
*Select score, `file.path` from MyUserTable WHERE `file.path` LIKE '%prefix_%' * == Optimized Execution Plan == Calc(select=[score, file.path], where=[LIKE(file.path, '%2022070611284%')]) +- TableSourceScan(table=[[default_catalog, default_database, MyUserTable, filter=[LIKE(file.path, _UTF-16LE'%2022070611284%')]]], fields=[score, file.path]) Thanks, Maryam On Mon, Mar 13, 2023 at 8:55 AM Hang Ruan <ruanhang1...@gmail.com> wrote: > Hi, yuxia, > I would like to help to complete this task. > > Best, > Hang > > yuxia <luoyu...@alumni.sjtu.edu.cn> 于2023年3月13日周一 09:32写道： > >> Yeah, you're right. We don't provide filtering files with patterns. And >> actually we had already a jira[1] for it. >> I was intended to do this in the past, but don't have much time. Anyone >> who are insterested can take it over. We're >> happy to help review. >> >> [1] https://issues.apache.org/jira/browse/FLINK-17398 >> >> Best regards, >> Yuxia >> >> ------------------------------ >> *发件人: *"User" <user@flink.apache.org> >> *收件人: *"Yaroslav Tkachenko" <yaros...@goldsky.com>, "Shammon FY" < >> zjur...@gmail.com> >> *抄送: *"User" <user@flink.apache.org> >> *发送时间: *星期一, 2023年 3 月 13日 上午 12:36:46 >> *主题: *Re: Are the Table API Connectors production ready? >> >> Thanks a lot, Yaroslav and Shammon. >> I want to use the Filesystem Connector. I tried it works well till it is >> running. If the job is restarted. It processes all the files again. >> >> Could not find the move or delete option after collecting the files. >> Also, I could not find the filtering using patterns. >> >> Pattern matching is required as different files exist in the same folder. >> >> Regards, >> Ravi >> On Friday, 10 March, 2023 at 05:47:27 am IST, Shammon FY < >> zjur...@gmail.com> wrote: >> >> >> Hi Ravi >> >> Agree with Yaroslav and if you find any problems in use, you can create >> an issue in jira >> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK . I >> have used kafka/jdbc/hive in production too, they work well. >> >> Best, >> Shammon >> >> On Fri, Mar 10, 2023 at 1:42 AM Yaroslav Tkachenko <yaros...@goldsky.com> >> wrote: >> >> Hi Ravi, >> >> All of them should be production ready. I've personally used half of them >> in production. >> >> Do you have any specific concerns? >> >> On Thu, Mar 9, 2023 at 9:39 AM ravi_suryavanshi.yahoo.com via user < >> user@flink.apache.org> wrote: >> >> Hi, >> Can anyone help me here? >> >> Thanks and regards, >> Ravi >> >> On Monday, 27 February, 2023 at 09:33:18 am IST, >> ravi_suryavanshi.yahoo.com via user <user@flink.apache.org> wrote: >> >> >> Hi Team, >> >> >> In Flink 1.16.0, we would like to use some of the Table API Connectors >> for production. Kindly let me know if the below connectors are production >> ready or only for testing purposes. >> >> NameVersionSourceSink >> Filesystem >> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/filesystem/> >> Bounded >> and Unbounded Scan, Lookup Streaming Sink, Batch Sink >> Elasticsearch >> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/elasticsearch/> >> 6.x >> & 7.x Not supported Streaming Sink, Batch Sink >> Opensearch >> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/opensearch/> >> 1.x >> & 2.x Not supported Streaming Sink, Batch Sink >> Apache Kafka >> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kafka/> >> 0.10+ Unbounded Scan Streaming Sink, Batch Sink >> Amazon DynamoDB >> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/dynamodb/> >> Not >> supported Streaming Sink, Batch Sink >> Amazon Kinesis Data Streams >> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kinesis/> >> Unbounded >> Scan Streaming Sink >> Amazon Kinesis Data Firehose >> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/firehose/> >> Not >> supported Streaming Sink >> JDBC >> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/jdbc/> >> Bounded >> Scan, Lookup Streaming Sink, Batch Sink >> Apache HBase >> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hbase/> >> 1.4.x >> & 2.2.x Bounded Scan, Lookup Streaming Sink, Batch Sink >> Apache Hive >> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hive/overview/> >> >> Thanks and regards >> >> >> -- Maryam Moafimadani Senior Data Developer @Shopify <http://www.shopify.com/>
