[jira] [Closed] (FLINK-30000) Introduce FileSystemFactory to create FileSystem from custom configuration
[ https://issues.apache.org/jira/browse/FLINK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-3. Resolution: Fixed > Introduce FileSystemFactory to create FileSystem from custom configuration > -- > > Key: FLINK-3 > URL: https://issues.apache.org/jira/browse/FLINK-3 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Priority: Major > Fix For: table-store-0.4.0 > > > Currently, table store uses static Flink FileSystem. This can not support: > 1. Use another FileSystem different from checkpoint FileSystem. > 2. Use FileSystem in Hive and Spark from custom configuration instead of > using FileSystem.initialize. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30039) Insert overwrite show excetion that query and insert schema do not match
[ https://issues.apache.org/jira/browse/FLINK-30039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-30039. Resolution: Fixed > Insert overwrite show excetion that query and insert schema do not match > > > Key: FLINK-30039 > URL: https://issues.apache.org/jira/browse/FLINK-30039 > Project: Flink > Issue Type: Bug > Components: Table Store >Affects Versions: 1.16.0 > Environment: Flink 1.16.0 >Reporter: StarBoy1005 >Priority: Critical > Attachments: screenshot-2.png, screenshot-3.png > > > After change a table's bucket num in table store,I used a insert overwrite > sql to reorganize records to new files (I guess form old orc file to the new). > Everytime if select column contains the partition column,then get the next > exception: > !screenshot-2.png! > When exclude the partition field,it do perform the overwrite. > I wonder the “EXPR$3: STRING NOT NULL” means what , -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30031) flink table store run abnormally when shade flink
[
https://issues.apache.org/jira/browse/FLINK-30031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee closed FLINK-30031.
Resolution: Fixed
> flink table store run abnormally when shade flink
> -
>
> Key: FLINK-30031
> URL: https://issues.apache.org/jira/browse/FLINK-30031
> Project: Flink
> Issue Type: Bug
> Components: Table Store
>Affects Versions: 1.15.0, table-store-0.2.1
>Reporter: zhenzhenhua
>Priority: Major
>
> {color:#172b4d}I try to sink flink-table-store in Apache SeaTunnel, SeaTunnel
> use the Flink version is 13.6.{color}
> {color:#172b4d}To avoid flink conflict,I use maven-shade-plugin plugin to
> shade flink dependency.{color}
> {color:#172b4d}However, runing build jar to write flink-table-store occur
> error,throw exception:{color}
>
> {code:java}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/flink/table/store/codegen/CodeGenerator
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at
> org.apache.flink.table.store.codegen.ComponentClassLoader.loadClassFromComponentOnly(ComponentClassLoader.java:127)
> at
> org.apache.flink.table.store.codegen.ComponentClassLoader.loadClass(ComponentClassLoader.java:106)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:370)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at java.util.Iterator.forEachRemaining(Iterator.java:116)
> at
> org.apache.flink.table.store.codegen.CodeGenLoader.discover(CodeGenLoader.java:123)
> at
> org.apache.flink.table.store.codegen.CodeGenUtils.generateRecordComparator(CodeGenUtils.java:65)
> at
> org.apache.flink.table.store.file.utils.KeyComparatorSupplier.(KeyComparatorSupplier.java:40)
> at
> org.apache.flink.table.store.file.KeyValueFileStore.(KeyValueFileStore.java:59)
> at
> org.apache.flink.table.store.table.ChangelogWithKeyFileStoreTable.(ChangelogWithKeyFileStoreTable.java:103)
> at
> org.apache.flink.table.store.table.FileStoreTableFactory.create(FileStoreTableFactory.java:72)
> at
> org.apache.flink.table.store.table.FileStoreTableFactory.create(FileStoreTableFactory.java:50)
> at org.example.TestWrite.main(TestWrite.java:24)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.table.store.codegen.CodeGenerator
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at
> org.apache.flink.table.store.codegen.ComponentClassLoader.loadClassFromComponentOnly(ComponentClassLoader.java:127)
> at
> org.apache.flink.table.store.codegen.ComponentClassLoader.loadClass(ComponentClassLoader.java:106)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 27 more
> {code}
>
>
> {code:java}
> // pom.xml
>
>
> org.apache.flink
> flink-table-store-core
> 0.2.1
>
>
> org.apache.flink
> flink-table-store-format
> 0.2.1
>
>
> org.apache.flink
> flink-table-common
> 1.15.0
>
>
> org.apache.flink
> flink-table-runtime
> 1.15.0
>
>
> org.apache.flink
> flink-connector-base
> 1.15.0
>
>
> org.apache.flink
> flink-shaded-jackson
> 2.12.1-13.0
>
>
> org.apache.flink
> flink-shaded-hadoop-2-uber
> 2.7.5-10.0
>
>
> org.apache.commons
> commons-lang3
> 3.3.2
>
>
> log4j
> log4j
> 1.2.17
>
>
>
>
>
> org.apache.maven.plugins
> maven-shade-plugin
> 3.1.1
>
>
> false
>
>
>
[jira] [Closed] (FLINK-29630) Junit 5.8.1 run unit test with temporary directory will occur Failed to delete temp directory.
[ https://issues.apache.org/jira/browse/FLINK-29630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29630. Resolution: Fixed > Junit 5.8.1 run unit test with temporary directory will occur Failed to > delete temp directory. > -- > > Key: FLINK-29630 > URL: https://issues.apache.org/jira/browse/FLINK-29630 > Project: Flink > Issue Type: Improvement > Components: Table Store >Affects Versions: table-store-0.3.0 >Reporter: Aiden Gong >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.4.0 > > Attachments: image-2022-10-14-09-12-33-903.png > > > Junit 5.8.1 run unit test with temporary directory will occur Failed to > delete temp directory. > My local : > windows 10 > jdk1.8 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-26937) Introduce Leveled compression for LSM
[ https://issues.apache.org/jira/browse/FLINK-26937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-26937. Resolution: Fixed > Introduce Leveled compression for LSM > - > > Key: FLINK-26937 > URL: https://issues.apache.org/jira/browse/FLINK-26937 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Assignee: xingyuan cheng >Priority: Minor > Fix For: table-store-0.4.0 > > > Currently ORC is all ZLIB compression by default, in fact the files at level > 0, will definitely be rewritten and we can have different compression for > different levels. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29988) Improve upper case fields for hive metastore
[ https://issues.apache.org/jira/browse/FLINK-29988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29988. Resolution: Fixed > Improve upper case fields for hive metastore > > > Key: FLINK-29988 > URL: https://issues.apache.org/jira/browse/FLINK-29988 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Assignee: Fang Yong >Priority: Major > Labels: pull-request-available > > If the fields in the fts table are uppercase, there will be a mismatched > exception when used in the Hive. > 1. If it is not supported at the beginning, throw an exception when flink > creates a table to the hive metastore. > 2. If it is supported, so that no error is reported in the whole process, but > save lower case in hive metastore. We can check columns with the same name > when creating a table in Flink with hive metastore. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-27814) Add an abstraction layer for connectors to read and write row data instead of key-values
[
https://issues.apache.org/jira/browse/FLINK-27814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee closed FLINK-27814.
Resolution: Fixed
> Add an abstraction layer for connectors to read and write row data instead of
> key-values
>
>
> Key: FLINK-27814
> URL: https://issues.apache.org/jira/browse/FLINK-27814
> Project: Flink
> Issue Type: Improvement
> Components: Table Store
>Affects Versions: table-store-0.2.0
>Reporter: Caizhi Weng
>Assignee: Caizhi Weng
>Priority: Major
> Labels: pull-request-available, stale-assigned
>
> Currently {{FileStore}} exposes an interface for reading and writing
> {{KeyValue}}. However connectors may have different ways to change a
> {{RowData}} to {{KeyValue}} under different {{WriteMode}}. This results in
> lots of {{if...else...}} branches and duplicated code.
> We need to add an abstraction layer for connectors to read and write row data
> instead of key-values.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-27548) Improve quick-start of table store
[ https://issues.apache.org/jira/browse/FLINK-27548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-27548. Resolution: Fixed > Improve quick-start of table store > -- > > Key: FLINK-27548 > URL: https://issues.apache.org/jira/browse/FLINK-27548 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Assignee: ZhuoYu Chen >Priority: Minor > Fix For: table-store-0.4.0 > > > When the quick-start is completed, the stream job needs to be killed on the > flink page and the table needs to be dropped. > But the exiting of the stream job is asynchronous and we need to wait a while > between these two actions. Otherwise there will be residue in the file > directory. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29636) Add micro benchmark module in flink table store
[ https://issues.apache.org/jira/browse/FLINK-29636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29636. Resolution: Fixed > Add micro benchmark module in flink table store > --- > > Key: FLINK-29636 > URL: https://issues.apache.org/jira/browse/FLINK-29636 > Project: Flink > Issue Type: Improvement > Components: Table Store >Affects Versions: table-store-0.3.0, table-store-0.2.2 >Reporter: Fang Yong >Priority: Major > Labels: pull-request-available > > Currently there's a `flink-table-store-benchmark` to run a query in flink > cluster and collect metrics to measure the performance of > `flink-table-store`. There're some key operation steps such as > read/write/compaction in `flink-table-store`, we should add a > `flink-table-store-micro-benchmark` for them -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29290) Produce changelog during full compaction in Table Store
[ https://issues.apache.org/jira/browse/FLINK-29290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29290. Resolution: Fixed > Produce changelog during full compaction in Table Store > --- > > Key: FLINK-29290 > URL: https://issues.apache.org/jira/browse/FLINK-29290 > Project: Flink > Issue Type: New Feature > Components: Table Store >Affects Versions: table-store-0.3.0 >Reporter: Caizhi Weng >Assignee: Caizhi Weng >Priority: Major > > Currently Table Store only produces changelog directly from input. Some > downstream systems, however, require complete changelogs including both > UPDATE_BEFORE and UPDATE_AFTER messages. > We can only get these information during full compaction, so we should add a > feature to produce changelog during full compaction. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29746) Add workflow in github for micro benchmarks
[ https://issues.apache.org/jira/browse/FLINK-29746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29746. Resolution: Fixed > Add workflow in github for micro benchmarks > --- > > Key: FLINK-29746 > URL: https://issues.apache.org/jira/browse/FLINK-29746 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Affects Versions: table-store-0.2.2 >Reporter: Fang Yong >Priority: Major > > Add workflow in github for micro benchmarks project -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-28009) Optimize data split
[ https://issues.apache.org/jira/browse/FLINK-28009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-28009. Resolution: Fixed > Optimize data split > --- > > Key: FLINK-28009 > URL: https://issues.apache.org/jira/browse/FLINK-28009 > Project: Flink > Issue Type: Improvement > Components: Table Store >Affects Versions: table-store-0.1.0 >Reporter: xingyuan cheng >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available > > Optimizing split data logic for large data volumes using parallel streams > https://github.com/apache/flink-table-store/pull/151 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29331) pre-aggregated merge supports changelog inputs
[ https://issues.apache.org/jira/browse/FLINK-29331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29331. Resolution: Fixed > pre-aggregated merge supports changelog inputs > -- > > Key: FLINK-29331 > URL: https://issues.apache.org/jira/browse/FLINK-29331 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Priority: Major > Fix For: table-store-0.4.0 > > > In FLINK-27626 , we have supported pre-agg merge, but no changelog inputs > support. > We can support changelog inputs for some function, like sum/count. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-28718) SinkSavepointITCase.testRecoverFromSavepoint is unstable
[
https://issues.apache.org/jira/browse/FLINK-28718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee closed FLINK-28718.
Resolution: Fixed
> SinkSavepointITCase.testRecoverFromSavepoint is unstable
>
>
> Key: FLINK-28718
> URL: https://issues.apache.org/jira/browse/FLINK-28718
> Project: Flink
> Issue Type: Bug
> Components: Table Store
>Reporter: Jingsong Lee
>Priority: Major
> Fix For: table-store-0.4.0
>
>
> https://github.com/apache/flink-table-store/runs/7537817210?check_suite_focus=true
> {code:java}
> Error: Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 185.274 s <<< FAILURE! - in
> org.apache.flink.table.store.connector.sink.SinkSavepointITCase
> Error: testRecoverFromSavepoint Time elapsed: 180.157 s <<< ERROR!
> org.junit.runners.model.TestTimedOutException: test timed out after 18
> milliseconds
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.flink.table.store.connector.sink.SinkSavepointITCase.testRecoverFromSavepoint(SinkSavepointITCase.java:84)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:750)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-28690) UpdateSchema#fromCatalogTable lost column comment
[ https://issues.apache.org/jira/browse/FLINK-28690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-28690. Resolution: Fixed > UpdateSchema#fromCatalogTable lost column comment > - > > Key: FLINK-28690 > URL: https://issues.apache.org/jira/browse/FLINK-28690 > Project: Flink > Issue Type: Bug > Components: Table Store >Affects Versions: table-store-0.2.0 >Reporter: Jane Chan >Assignee: Jane Chan >Priority: Not a Priority > Fix For: table-store-0.4.0 > > > The reason is that > org.apache.flink.table.api.TableSchema#toPhysicalRowDataType lost column > comments, which leads to comparison failure in > AbstractTableStoreFactory#buildFileStoreTable. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29080) Migrate all tests from managed table to catalog-based tests
[ https://issues.apache.org/jira/browse/FLINK-29080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29080. Resolution: Fixed > Migrate all tests from managed table to catalog-based tests > --- > > Key: FLINK-29080 > URL: https://issues.apache.org/jira/browse/FLINK-29080 > Project: Flink > Issue Type: Improvement > Components: Table Store >Affects Versions: table-store-0.3.0 >Reporter: Jane Chan >Priority: Major > Fix For: table-store-0.4.0 > > > To get rid of ManagedTableFactory and enable test on -Pflink-1.14 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29243) Introduce a COW mode for table store
[ https://issues.apache.org/jira/browse/FLINK-29243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29243. Resolution: Fixed changelog producer = full-compaction is the cow mode. > Introduce a COW mode for table store > > > Key: FLINK-29243 > URL: https://issues.apache.org/jira/browse/FLINK-29243 > Project: Flink > Issue Type: New Feature > Components: Table Store >Reporter: Jingsong Lee >Priority: Major > Fix For: table-store-0.4.0 > > > Currently, we can combine the COW (Copy on write) mode by three options, but > the combination may not be optimal and tedious. > We can introduce a option that specifically turns on the mode of COW. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29272) Document DataStream API (DataStream to Table) for table store
[ https://issues.apache.org/jira/browse/FLINK-29272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29272. Resolution: Fixed > Document DataStream API (DataStream to Table) for table store > - > > Key: FLINK-29272 > URL: https://issues.apache.org/jira/browse/FLINK-29272 > Project: Flink > Issue Type: New Feature > Components: Table Store >Reporter: Jingsong Lee >Priority: Major > Fix For: table-store-0.4.0 > > > We can have documentation to describe how to convert from DataStream to Table > to write to TableStore. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29820) Support to read data from table store with specify snapshot id
[ https://issues.apache.org/jira/browse/FLINK-29820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17702211#comment-17702211 ] Jingsong Lee commented on FLINK-29820: -- We have already supported this. > Support to read data from table store with specify snapshot id > -- > > Key: FLINK-29820 > URL: https://issues.apache.org/jira/browse/FLINK-29820 > Project: Flink > Issue Type: New Feature > Components: Table Store >Affects Versions: table-store-0.3.0, table-store-0.2.2 >Reporter: Fang Yong >Priority: Major > > Currently our queries can only read the latest snapshot from table store. We > need to support reading the data of the specified snapshot, in this way, we > can read historical versions or compare data across versions as needed. In > [#FLINK-29735] we have supported to query snapshots with `SELECT * FROM > MyTable$snapshots`, we can query data from table `wordcount` with `snapshot > 10` as follows: > `SELECT * FROM wordcount$snapshot$10 WHERE count>10` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29820) Support to read data from table store with specify snapshot id
[ https://issues.apache.org/jira/browse/FLINK-29820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29820. Resolution: Fixed > Support to read data from table store with specify snapshot id > -- > > Key: FLINK-29820 > URL: https://issues.apache.org/jira/browse/FLINK-29820 > Project: Flink > Issue Type: New Feature > Components: Table Store >Affects Versions: table-store-0.3.0, table-store-0.2.2 >Reporter: Fang Yong >Priority: Major > > Currently our queries can only read the latest snapshot from table store. We > need to support reading the data of the specified snapshot, in this way, we > can read historical versions or compare data across versions as needed. In > [#FLINK-29735] we have supported to query snapshots with `SELECT * FROM > MyTable$snapshots`, we can query data from table `wordcount` with `snapshot > 10` as follows: > `SELECT * FROM wordcount$snapshot$10 WHERE count>10` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29823) Support get schema for table snapshot
[ https://issues.apache.org/jira/browse/FLINK-29823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29823. Resolution: Fixed > Support get schema for table snapshot > - > > Key: FLINK-29823 > URL: https://issues.apache.org/jira/browse/FLINK-29823 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Affects Versions: table-store-0.3.0, table-store-0.2.2 >Reporter: Fang Yong >Priority: Major > Labels: pull-request-available > > Support to create schema from table snapshot with given snapshot id -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29828) Support to read data from table with given snapshot id
[ https://issues.apache.org/jira/browse/FLINK-29828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29828. Resolution: Fixed > Support to read data from table with given snapshot id > -- > > Key: FLINK-29828 > URL: https://issues.apache.org/jira/browse/FLINK-29828 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Affects Versions: table-store-0.3.0, table-store-0.2.2 >Reporter: Fang Yong >Priority: Major > > Support reading data from table with given snapshot id from table, users can > query table data as follows: > `SELECT t1.word, t1.cnt, t2.cnt FROM word_count$snapshot$10 t1 JOIN > word_count$snapshot$11 t2 ON t1.word = t2.word and t1.cnt != t2.cnt` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29882) LargeDataITCase is not stable
[ https://issues.apache.org/jira/browse/FLINK-29882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29882. Resolution: Fixed > LargeDataITCase is not stable > - > > Key: FLINK-29882 > URL: https://issues.apache.org/jira/browse/FLINK-29882 > Project: Flink > Issue Type: Bug > Components: Table Store >Affects Versions: table-store-0.3.0 >Reporter: Jane Chan >Priority: Major > Fix For: table-store-0.4.0 > > > https://github.com/apache/flink-table-store/actions/runs/3391781964/jobs/5637271002 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29832) Improve switch to default database in docs
[ https://issues.apache.org/jira/browse/FLINK-29832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29832. Resolution: Fixed > Improve switch to default database in docs > -- > > Key: FLINK-29832 > URL: https://issues.apache.org/jira/browse/FLINK-29832 > Project: Flink > Issue Type: Bug > Components: Table Store >Affects Versions: table-store-0.3.0, table-store-0.2.2 >Reporter: Fang Yong >Priority: Major > Labels: pull-request-available > Attachments: image-2022-11-01-16-40-47-539.png > > > `FlinkCatalogFactory` creates a default database named `default` in table > store. The `default` is a keyword in SQL, and when we create a new database, > we cant execute `use default` to switch to `default` directly. We can switch > to default database "use `default`;" in flink table store -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30227) Add LeafEmptyFunction for predicate without fields
[ https://issues.apache.org/jira/browse/FLINK-30227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-30227. Resolution: Fixed > Add LeafEmptyFunction for predicate without fields > -- > > Key: FLINK-30227 > URL: https://issues.apache.org/jira/browse/FLINK-30227 > Project: Flink > Issue Type: Improvement > Components: Table Store >Affects Versions: table-store-0.3.0 >Reporter: Fang Yong >Priority: Major > > In addition to `LeafBinaryFunction` and `LeafUnaryFunction`, we should add > `LeafEmptyFunction` for predicate -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30080) Introduce public programming api and dependency jar for table store
[ https://issues.apache.org/jira/browse/FLINK-30080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-30080. Resolution: Fixed > Introduce public programming api and dependency jar for table store > --- > > Key: FLINK-30080 > URL: https://issues.apache.org/jira/browse/FLINK-30080 > Project: Flink > Issue Type: New Feature > Components: Table Store >Reporter: Jingsong Lee >Priority: Major > > Users need to access tablestore through programming interfaces, but do not > want to use a computing engine such as Flink or Spark. > We can expose the programming api to read and write the tablestore, and also > need to expose the corresponding dependency jar. Note that this dependency > may not conflict with multiple versions of Flink, which is conducive to the > integration of third-party systems. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30276) [umbrella] Flink free for table store core
[ https://issues.apache.org/jira/browse/FLINK-30276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-30276. Resolution: Fixed > [umbrella] Flink free for table store core > -- > > Key: FLINK-30276 > URL: https://issues.apache.org/jira/browse/FLINK-30276 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Fix For: table-store-0.4.0 > > > In FLINK-30080, We need a core that does not rely on specific Flink versions > to support flexible deployment and ecology. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31002) Provide data sampling query
[ https://issues.apache.org/jira/browse/FLINK-31002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31002. Resolution: Fixed this should be implemented by flink sql > Provide data sampling query > --- > > Key: FLINK-31002 > URL: https://issues.apache.org/jira/browse/FLINK-31002 > Project: Flink > Issue Type: New Feature > Components: Table Store >Reporter: Jingsong Lee >Priority: Major > > Want to take several randomly from each partition, but the limit is always > fixed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30628) Kerberos in HiveCatalog is not work
[ https://issues.apache.org/jira/browse/FLINK-30628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-30628. Resolution: Fixed > Kerberos in HiveCatalog is not work > --- > > Key: FLINK-30628 > URL: https://issues.apache.org/jira/browse/FLINK-30628 > Project: Flink > Issue Type: Bug > Components: Table Store >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.4.0, table-store-0.3.1 > > > We should read kerberos keytab from catalog options and doAs for hive > metastore client. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31179) Make data structures serializable
[ https://issues.apache.org/jira/browse/FLINK-31179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31179. Resolution: Fixed > Make data structures serializable > - > > Key: FLINK-31179 > URL: https://issues.apache.org/jira/browse/FLINK-31179 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31322) Improve merge-into action
[ https://issues.apache.org/jira/browse/FLINK-31322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31322. Resolution: Fixed > Improve merge-into action > - > > Key: FLINK-31322 > URL: https://issues.apache.org/jira/browse/FLINK-31322 > Project: Flink > Issue Type: Improvement > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: yuzelin >Priority: Major > > Umbrella issue for improving flink merge-into action. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31315) FlinkActionsE2eTest.testMergeInto is unstable
[
https://issues.apache.org/jira/browse/FLINK-31315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee closed FLINK-31315.
Resolution: Fixed
> FlinkActionsE2eTest.testMergeInto is unstable
> -
>
> Key: FLINK-31315
> URL: https://issues.apache.org/jira/browse/FLINK-31315
> Project: Flink
> Issue Type: Bug
> Components: Table Store
>Reporter: Jingsong Lee
>Priority: Major
> Fix For: table-store-0.4.0
>
>
> {code:java}
> Error: Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 320.272 s <<< FAILURE! - in
> org.apache.flink.table.store.tests.FlinkActionsE2eTest
> 82Error: testMergeInto Time elapsed: 111.826 s <<< FAILURE!
> 83org.opentest4j.AssertionFailedError:
> 84Result is still unexpected after 60 retries.
> 85Expected: {3, v_3, creation, 02-27=1, 2, v_2, creation, 02-27=1, 6, v_6,
> creation, 02-28=1, 1, v_1, creation, 02-27=1, 8, v_8, insert, 02-29=1, 11,
> v_11, insert, 02-29=1, 7, Seven, matched_upsert, 02-28=1, 5, v_5, creation,
> 02-28=1, 10, v_10, creation, 02-28=1, 9, v_9, creation, 02-28=1}
> 86Actual: {4, v_4, creation, 02-27=1, 8, v_8, creation, 02-28=1, 3, v_3,
> creation, 02-27=1, 7, v_7, creation, 02-28=1, 2, v_2, creation, 02-27=1, 6,
> v_6, creation, 02-28=1, 1, v_1, creation, 02-27=1, 5, v_5, creation, 02-28=1,
> 10, v_10, creation, 02-28=1, 9, v_9, creation, 02-28=1}
> 87at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:39)
> 88at org.junit.jupiter.api.Assertions.fail(Assertions.java:134)
> 89at
> org.apache.flink.table.store.tests.E2eTestBase.checkResult(E2eTestBase.java:261)
> 90at
> org.apache.flink.table.store.tests.FlinkActionsE2eTest.testMergeInto(FlinkActionsE2eTest.java:355)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-31338) support infer parallelism for flink table store
[ https://issues.apache.org/jira/browse/FLINK-31338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31338. Resolution: Fixed Just use github pr/issues. > support infer parallelism for flink table store > > > Key: FLINK-31338 > URL: https://issues.apache.org/jira/browse/FLINK-31338 > Project: Flink > Issue Type: Improvement > Components: Table Store >Affects Versions: table-store-0.3.0 >Reporter: Jun Zhang >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.4.0 > > > When using flink to query the fts table, we can config the scan parallelism > by set the scan.parallelism, but the user may do not know how much > parallelism should be used, setting a too large parallelism will cause > resource waste, setting the parallelism too small will cause the query to be > slow, so we can add parallelism infer. > The function is enabled by default. the parallelism is equal to the number of > read splits. Of course, the user can manually turn off the infer function. In > order to prevent too many datafiles from causing excessive parallelism, we > also set a max infer parallelism. When the infer parallelism exceeds the > setting, use the max parallelism. > In addition, we also need to compare with the limit in the select query > statement to get a more appropriate parallelism in the case of limit > pushdown, for example we have a sql select * from table limit 1, and finally > we infer the parallelism is 10, but we only one parallel is needed , besause > we only need one data . -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31417) Hadoop version unknown when TrinoPageSourceBase.getNextPage
[
https://issues.apache.org/jira/browse/FLINK-31417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee closed FLINK-31417.
Resolution: Fixed
> Hadoop version unknown when TrinoPageSourceBase.getNextPage
> ---
>
> Key: FLINK-31417
> URL: https://issues.apache.org/jira/browse/FLINK-31417
> Project: Flink
> Issue Type: Bug
> Components: Table Store
>Affects Versions: table-store-0.4.0
>Reporter: nonggia.liang
>Priority: Major
>
> Exception thrown when quering flink-table-store by trino
> {code:java}
> 2023-03-13T11:46:36.694+0800ERROR SplitRunner-11-113
> io.trino.execution.executor.TaskExecutor Error processing Split
> 20230313_034504_0_jdcet.1.0.0-11 {} (start = 3.599627617710298E10, wall =
> 89264 ms, cpu = 0 ms, wait = 1 ms, calls = 1)java.lang.NoClassDefFoundError:
> Could not initialize class
> org.apache.flink.table.store.shaded.org.apache.orc.impl.RecordReaderUtils
> at
> org.apache.flink.table.store.shaded.org.apache.orc.impl.RecordReaderImpl.(RecordReaderImpl.java:257)
> at
> org.apache.flink.table.store.shaded.org.apache.orc.impl.ReaderImpl.rows(ReaderImpl.java:649)
> at
> org.apache.flink.table.store.format.orc.OrcReaderFactory.createRecordReader(OrcReaderFactory.java:284)
> at
> org.apache.flink.table.store.format.orc.OrcReaderFactory.createReader(OrcReaderFactory.java:98)
> at
> org.apache.flink.table.store.format.orc.OrcReaderFactory.createReader(OrcReaderFactory.java:56)
> at
> org.apache.flink.table.store.file.utils.FileUtils.createFormatReader(FileUtils.java:108)
> at
> org.apache.flink.table.store.file.io.KeyValueDataFileRecordReader.(KeyValueDataFileRecordReader.java:55)
> at
> org.apache.flink.table.store.file.io.KeyValueFileReaderFactory.createRecordReader(KeyValueFileReaderFactory.java:95)
> at
> org.apache.flink.table.store.file.mergetree.MergeTreeReaders.lambda$readerForRun$1(MergeTreeReaders.java:89)
> at
> org.apache.flink.table.store.file.mergetree.compact.ConcatRecordReader.create(ConcatRecordReader.java:50)
> at
> org.apache.flink.table.store.file.mergetree.MergeTreeReaders.readerForRun(MergeTreeReaders.java:92)
> at
> org.apache.flink.table.store.file.mergetree.MergeTreeReaders.readerForSection(MergeTreeReaders.java:74)
> at
> org.apache.flink.table.store.file.operation.KeyValueFileStoreRead.lambda$createReader$2(KeyValueFileStoreRead.java:195)
> at
> org.apache.flink.table.store.file.mergetree.compact.ConcatRecordReader.create(ConcatRecordReader.java:50)
> at
> org.apache.flink.table.store.file.operation.KeyValueFileStoreRead.createReader(KeyValueFileStoreRead.java:204)
> at
> org.apache.flink.table.store.table.source.KeyValueTableRead.createReader(KeyValueTableRead.java:44)
> at
> org.apache.flink.table.store.trino.TrinoPageSourceProvider.createPageSource(TrinoPageSourceProvider.java:76)
> at
> org.apache.flink.table.store.trino.TrinoPageSourceProvider.lambda$createPageSource$0(TrinoPageSourceProvider.java:52)
> at
> org.apache.flink.table.store.trino.ClassLoaderUtils.runWithContextClassLoader(ClassLoaderUtils.java:30)
> at
> org.apache.flink.table.store.trino.TrinoPageSourceProvider.createPageSource(TrinoPageSourceProvider.java:51)
> at
> io.trino.split.PageSourceManager.createPageSource(PageSourceManager.java:68)
> at
> io.trino.operator.TableScanOperator.getOutput(TableScanOperator.java:308)
> at io.trino.operator.Driver.processInternal(Driver.java:388)at
> io.trino.operator.Driver.lambda$processFor$9(Driver.java:292)at
> io.trino.operator.Driver.tryWithLock(Driver.java:685)at
> io.trino.operator.Driver.processFor(Driver.java:285)at
> io.trino.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1076)
> at
> io.trino.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
> at
> io.trino.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:488)
> at io.trino.$gen.Trino_366_020230313_034413_2.run(Unknown Source)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-03-13T11:46:36.775+0800ERROR remote-task-callback-2
> io.trino.execution.scheduler.PipelinedStageExecution Pipelined stage
> execution for stage 20230313_034504_0_jdcet.1
> failedjava.lang.ExceptionInInitializerErrorat
>
[jira] [Closed] (FLINK-31391) Introduce lookup changelog producer
[ https://issues.apache.org/jira/browse/FLINK-31391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31391. Resolution: Fixed > Introduce lookup changelog producer > --- > > Key: FLINK-31391 > URL: https://issues.apache.org/jira/browse/FLINK-31391 > Project: Flink > Issue Type: New Feature > Components: Table Store >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Fix For: table-store-0.4.0 > > > Currently, only full-compaction can produce changelog, some merge-engine must > have changelog producing, for example, partial-update and aggregation. But > full-compaction is very heavy, write amplification is big huge... > We should introduce a new changelog producer, supports lower latency to > produce changelog. > > POC: https://github.com/apache/flink-table-store/pull/590 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31417) Hadoop version unknown when TrinoPageSourceBase.getNextPage
[
https://issues.apache.org/jira/browse/FLINK-31417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17702207#comment-17702207
]
Jingsong Lee commented on FLINK-31417:
--
Thanks [~nonggia] , this has been fixed in 0.3, will cp to 0.4.
> Hadoop version unknown when TrinoPageSourceBase.getNextPage
> ---
>
> Key: FLINK-31417
> URL: https://issues.apache.org/jira/browse/FLINK-31417
> Project: Flink
> Issue Type: Bug
> Components: Table Store
>Affects Versions: table-store-0.4.0
>Reporter: nonggia.liang
>Priority: Major
>
> Exception thrown when quering flink-table-store by trino
> {code:java}
> 2023-03-13T11:46:36.694+0800ERROR SplitRunner-11-113
> io.trino.execution.executor.TaskExecutor Error processing Split
> 20230313_034504_0_jdcet.1.0.0-11 {} (start = 3.599627617710298E10, wall =
> 89264 ms, cpu = 0 ms, wait = 1 ms, calls = 1)java.lang.NoClassDefFoundError:
> Could not initialize class
> org.apache.flink.table.store.shaded.org.apache.orc.impl.RecordReaderUtils
> at
> org.apache.flink.table.store.shaded.org.apache.orc.impl.RecordReaderImpl.(RecordReaderImpl.java:257)
> at
> org.apache.flink.table.store.shaded.org.apache.orc.impl.ReaderImpl.rows(ReaderImpl.java:649)
> at
> org.apache.flink.table.store.format.orc.OrcReaderFactory.createRecordReader(OrcReaderFactory.java:284)
> at
> org.apache.flink.table.store.format.orc.OrcReaderFactory.createReader(OrcReaderFactory.java:98)
> at
> org.apache.flink.table.store.format.orc.OrcReaderFactory.createReader(OrcReaderFactory.java:56)
> at
> org.apache.flink.table.store.file.utils.FileUtils.createFormatReader(FileUtils.java:108)
> at
> org.apache.flink.table.store.file.io.KeyValueDataFileRecordReader.(KeyValueDataFileRecordReader.java:55)
> at
> org.apache.flink.table.store.file.io.KeyValueFileReaderFactory.createRecordReader(KeyValueFileReaderFactory.java:95)
> at
> org.apache.flink.table.store.file.mergetree.MergeTreeReaders.lambda$readerForRun$1(MergeTreeReaders.java:89)
> at
> org.apache.flink.table.store.file.mergetree.compact.ConcatRecordReader.create(ConcatRecordReader.java:50)
> at
> org.apache.flink.table.store.file.mergetree.MergeTreeReaders.readerForRun(MergeTreeReaders.java:92)
> at
> org.apache.flink.table.store.file.mergetree.MergeTreeReaders.readerForSection(MergeTreeReaders.java:74)
> at
> org.apache.flink.table.store.file.operation.KeyValueFileStoreRead.lambda$createReader$2(KeyValueFileStoreRead.java:195)
> at
> org.apache.flink.table.store.file.mergetree.compact.ConcatRecordReader.create(ConcatRecordReader.java:50)
> at
> org.apache.flink.table.store.file.operation.KeyValueFileStoreRead.createReader(KeyValueFileStoreRead.java:204)
> at
> org.apache.flink.table.store.table.source.KeyValueTableRead.createReader(KeyValueTableRead.java:44)
> at
> org.apache.flink.table.store.trino.TrinoPageSourceProvider.createPageSource(TrinoPageSourceProvider.java:76)
> at
> org.apache.flink.table.store.trino.TrinoPageSourceProvider.lambda$createPageSource$0(TrinoPageSourceProvider.java:52)
> at
> org.apache.flink.table.store.trino.ClassLoaderUtils.runWithContextClassLoader(ClassLoaderUtils.java:30)
> at
> org.apache.flink.table.store.trino.TrinoPageSourceProvider.createPageSource(TrinoPageSourceProvider.java:51)
> at
> io.trino.split.PageSourceManager.createPageSource(PageSourceManager.java:68)
> at
> io.trino.operator.TableScanOperator.getOutput(TableScanOperator.java:308)
> at io.trino.operator.Driver.processInternal(Driver.java:388)at
> io.trino.operator.Driver.lambda$processFor$9(Driver.java:292)at
> io.trino.operator.Driver.tryWithLock(Driver.java:685)at
> io.trino.operator.Driver.processFor(Driver.java:285)at
> io.trino.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1076)
> at
> io.trino.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
> at
> io.trino.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:488)
> at io.trino.$gen.Trino_366_020230313_034413_2.run(Unknown Source)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-03-13T11:46:36.775+0800ERROR remote-task-callback-2
> io.trino.execution.scheduler.PipelinedStageExecution Pipelined stage
> execution for stage 20230313_034504_0_jdcet.1
>
[jira] [Closed] (FLINK-31461) Supports schema historical version expiring
[ https://issues.apache.org/jira/browse/FLINK-31461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31461. Resolution: Fixed Use github pr/issues. > Supports schema historical version expiring > --- > > Key: FLINK-31461 > URL: https://issues.apache.org/jira/browse/FLINK-31461 > Project: Flink > Issue Type: Improvement > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: Fang Yong >Priority: Major > Labels: pull-request-available > > Schema evolution will generate multiple versions of schema. When the > specified version of the schema is no longer referenced by snapshot, it > should be deleted -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31462) [Flink] Supports full calculation from specified snapshots in streaming mode
[ https://issues.apache.org/jira/browse/FLINK-31462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31462. Resolution: Fixed Use github pr/issues. > [Flink] Supports full calculation from specified snapshots in streaming mode > > > Key: FLINK-31462 > URL: https://issues.apache.org/jira/browse/FLINK-31462 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Ming Li >Priority: Major > Labels: pull-request-available > > Currently, the table store provides a startup mode for incremental > consumption from a specified snapshot in streaming mode. We can provide a > startup mode for incremental consumption after full calculation from a > specified snapshot. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31429) Support CTAS(create table as) streaming job with schema changes in table store
[ https://issues.apache.org/jira/browse/FLINK-31429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17702206#comment-17702206 ] Jingsong Lee commented on FLINK-31429: -- Please use github pr/issues. > Support CTAS(create table as) streaming job with schema changes in table store > -- > > Key: FLINK-31429 > URL: https://issues.apache.org/jira/browse/FLINK-31429 > Project: Flink > Issue Type: New Feature > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: Caizhi Weng >Assignee: Caizhi Weng >Priority: Major > Fix For: table-store-0.4.0 > > > Currently CDC connectors for Flink has the ability to stream out records > changes and schema changes of a database table. Flink Table Store should have > the ability to directly consume these changes, including schema changes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31451) Flink Table Store Ecosystem: Introduce Presto Reader for table store
[ https://issues.apache.org/jira/browse/FLINK-31451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31451. Resolution: Fixed Use github pr/issues. > Flink Table Store Ecosystem: Introduce Presto Reader for table store > > > Key: FLINK-31451 > URL: https://issues.apache.org/jira/browse/FLINK-31451 > Project: Flink > Issue Type: New Feature > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: Zaihu Shi >Assignee: Zaihu Shi >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.4.0 > > > Introduce Presto Reader for table store -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31479) Close blocking iterators in tests
[ https://issues.apache.org/jira/browse/FLINK-31479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31479. Resolution: Fixed Use github issues. > Close blocking iterators in tests > - > > Key: FLINK-31479 > URL: https://issues.apache.org/jira/browse/FLINK-31479 > Project: Flink > Issue Type: Bug > Components: Table Store >Reporter: Guojun Li >Priority: Major > > Several blocking iterators are not closed in `ContinuousFileStoreITCase` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31467) Support time travel for Spark 3.3
[ https://issues.apache.org/jira/browse/FLINK-31467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-31467. Resolution: Fixed Use github issues. > Support time travel for Spark 3.3 > - > > Key: FLINK-31467 > URL: https://issues.apache.org/jira/browse/FLINK-31467 > Project: Flink > Issue Type: New Feature > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: yuzelin >Priority: Major > Labels: pull-request-available > > Support Spark 3.3 `VERSION AS OF` and `TIMESTAMP AS OF` Syntax -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] xuzhiwen1255 commented on pull request #22208: [FLINK-31499] [table-planner] Move SqlCreateTable conversion logic to SqlCreateTableConverter
xuzhiwen1255 commented on PR #22208: URL: https://github.com/apache/flink/pull/22208#issuecomment-1475073169 Thanks for @wuchong, according to the comments, I reworked the code, PTAL. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #22210: [FLINK-31506][table] Move execution logic of AlterOperation out from TableEnvironmentImpl
flinkbot commented on PR #22210: URL: https://github.com/apache/flink/pull/22210#issuecomment-1475038905 ## CI report: * cdbe35311a6437a0161640ba41f4d97be0495566 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-31506) Move execution logic of AlterOperation out from TableEnvironmentImpl
[
https://issues.apache.org/jira/browse/FLINK-31506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Nuyanzin reassigned FLINK-31506:
---
Assignee: Sergey Nuyanzin
> Move execution logic of AlterOperation out from TableEnvironmentImpl
>
>
> Key: FLINK-31506
> URL: https://issues.apache.org/jira/browse/FLINK-31506
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Reporter: Jark Wu
>Assignee: Sergey Nuyanzin
>Priority: Major
> Labels: pull-request-available
>
> This should implement {{ExecutableOperation}} for all the {{AlterOperation}}s
> to move the execution logic out from
> {{TableEnvironmentImpl#executeInternal()}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] snuyanzin opened a new pull request, #22210: [FLINK-31506][table] Move execution logic of AlterOperation out from TableEnvironmentImpl
snuyanzin opened a new pull request, #22210: URL: https://github.com/apache/flink/pull/22210 The PR continues the work started at https://github.com/apache/flink/pull/22175 This PR does this for `AlterOperation`. ## What is the purpose of the change Migrate `AlterOperation`s to extend ExecutableOperation and move it's execution logic out from TableEnvironmentImpl. ## Brief change log Migrate `AlterOperation`s to extend ExecutableOperation and move it's execution logic out from TableEnvironmentImpl. ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): ( no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): ( no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: ( no ) - The S3 file system connector: ( no ) ## Documentation - Does this pull request introduce a new feature? ( no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-31506) Move execution logic of AlterOperation out from TableEnvironmentImpl
[
https://issues.apache.org/jira/browse/FLINK-31506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-31506:
---
Labels: pull-request-available (was: )
> Move execution logic of AlterOperation out from TableEnvironmentImpl
>
>
> Key: FLINK-31506
> URL: https://issues.apache.org/jira/browse/FLINK-31506
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Reporter: Jark Wu
>Priority: Major
> Labels: pull-request-available
>
> This should implement {{ExecutableOperation}} for all the {{AlterOperation}}s
> to move the execution logic out from
> {{TableEnvironmentImpl#executeInternal()}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #22209: [FLINK-31505][table] Move execution logic of DropOperation out from TableEnvironmentImpl
flinkbot commented on PR #22209: URL: https://github.com/apache/flink/pull/22209#issuecomment-1475016874 ## CI report: * f5e1772b1a1385099a0f761d89bff4ff47451440 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-31505) Move execution logic of DropOperation out from TableEnvironmentImpl
[
https://issues.apache.org/jira/browse/FLINK-31505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-31505:
---
Labels: pull-request-available (was: )
> Move execution logic of DropOperation out from TableEnvironmentImpl
> ---
>
> Key: FLINK-31505
> URL: https://issues.apache.org/jira/browse/FLINK-31505
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Reporter: Jark Wu
>Assignee: Sergey Nuyanzin
>Priority: Major
> Labels: pull-request-available
>
> This should implement {{ExecutableOperation}} for all the {{DropOperation}}s
> to move the execution logic out from
> {{TableEnvironmentImpl#executeInternal()}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] snuyanzin opened a new pull request, #22209: [FLINK-31505][table] Move execution logic of DropOperation out from TableEnvironmentImpl
snuyanzin opened a new pull request, #22209: URL: https://github.com/apache/flink/pull/22209 The PR continues the work started at https://github.com/apache/flink/pull/22175 This PR does this for `DropOperation`. It also removes methods became unused after merging of https://github.com/apache/flink/pull/22175 ## What is the purpose of the change Migrate `DropOperation`s to extend ExecutableOperation and move it's execution logic out from TableEnvironmentImpl. ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change This change is already covered by existing tests, such as *(please describe tests)*. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): ( no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): ( no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: ( no ) - The S3 file system connector: ( no ) ## Documentation - Does this pull request introduce a new feature? ( no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] JingGe opened a new pull request, #624: remove Jekyll toc
JingGe opened a new pull request, #624:
URL: https://github.com/apache/flink-web/pull/624
Remove all unused Jekyll {% toc %}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Assigned] (FLINK-31501) Move SqlCreateView conversion logic to SqlCreateViewConverter
[
https://issues.apache.org/jira/browse/FLINK-31501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu reassigned FLINK-31501:
---
Assignee: Jark Wu
> Move SqlCreateView conversion logic to SqlCreateViewConverter
> -
>
> Key: FLINK-31501
> URL: https://issues.apache.org/jira/browse/FLINK-31501
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Reporter: Jark Wu
>Assignee: Jark Wu
>Priority: Major
>
> Introduce {{SqlCreateViewConverter}} and move the conversion logic of
> SqlCreateView -> CreateViewOperation to it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] wuchong commented on a diff in pull request #22208: [FLINK-31499] [table-planner] Move SqlCreateTable conversion logic to SqlCreateTableConverter
wuchong commented on code in PR #22208:
URL: https://github.com/apache/flink/pull/22208#discussion_r1141037935
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/operations/converters/SqlCreateTableLikeConverter.java:
##
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.operations.converters;
+
+import org.apache.flink.sql.parser.ddl.SqlCreateTableLike;
+import org.apache.flink.table.operations.Operation;
+
+import static
org.apache.flink.table.planner.operations.CreateTableConverterUtils.convertCreateTable;
+
+/** A converter for {@link SqlCreateTableLike}. */
+public class SqlCreateTableLikeConverter implements
SqlNodeConverter {
+
+@Override
+public Operation convertSqlNode(SqlCreateTableLike node, ConvertContext
context) {
+return convertCreateTable(context, node);
Review Comment:
You can simply call `return new
SqlCreateTableConverter().convertSqlNode(node, context);` to reuse the
conversion logic if we would move the util there.
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/operations/converters/SqlCreateTableConverter.java:
##
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.operations.converters;
+
+import org.apache.flink.sql.parser.ddl.SqlCreateTable;
+import org.apache.flink.table.operations.Operation;
+
+import static
org.apache.flink.table.planner.operations.CreateTableConverterUtils.convertCreateTable;
+
+/** A converter for {@link SqlCreateTable}. */
+public class SqlCreateTableConverter implements
SqlNodeConverter {
+
+@Override
+public Operation convertSqlNode(SqlCreateTable node, ConvertContext
context) {
+return convertCreateTable(context, node);
Review Comment:
This makes the design of `SqlNodeConverter` less useful because the core
logic is out of it. Please move the implementation of `convertCreateTable` here.
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/operations/converters/SqlCreateTableAsConverter.java:
##
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.operations.converters;
+
+import org.apache.flink.sql.parser.ddl.SqlCreateTableAs;
+import org.apache.flink.table.operations.Operation;
+
+import static
org.apache.flink.table.planner.operations.CreateTableConverterUtils.convertCreateTableAs;
+
+/** A converter for {@link SqlCreateTableAs}. */
+public class SqlCreateTableAsConverter implements
SqlNodeConverter {
+@Override
[jira] [Assigned] (FLINK-31511) Translate documentation sql_functions_zh.yml to the latest version
[ https://issues.apache.org/jira/browse/FLINK-31511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu reassigned FLINK-31511: --- Assignee: Hang Ruan > Translate documentation sql_functions_zh.yml to the latest version > --- > > Key: FLINK-31511 > URL: https://issues.apache.org/jira/browse/FLINK-31511 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Hang Ruan >Assignee: Hang Ruan >Priority: Minor > > Some content of these functions in sql_functions_zh.yml is outdated. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] lincoln-lil commented on a diff in pull request #22197: [FLINK-31487][table-planner] Add targetColumns to DynamicTableSink#Context
lincoln-lil commented on code in PR #22197:
URL: https://github.com/apache/flink/pull/22197#discussion_r1141026773
##
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/plan/common/PartialInsertTest.scala:
##
@@ -163,6 +192,21 @@ class PartialInsertTest(isBatch: Boolean) extends
TableTestBase {
"SELECT a,b,c,d,e,123 FROM MyTable"
)
}
+
+ @Test
+ def testPartialInsert(): Unit = {
Review Comment:
Ok
##
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/SinkModifyOperation.java:
##
@@ -121,6 +135,11 @@ public QueryOperation getChild() {
return child;
}
+/** return an empty array when no column list specified. */
Review Comment:
Good catch! comments should be updated here since the unnecessarily
'int[0][]' was replaced by null.
##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/calcite/PreValidateReWriter.scala:
##
@@ -401,6 +404,15 @@ object PreValidateReWriter {
SqlUtil.newContextException(pos, e)
}
+ private def validateUnsupportedCompositeColumn(id: SqlIdentifier): Unit = {
+assert(id != null)
+if (!id.isSimple) {
+ val pos = id.getParserPosition
+ // TODO add accurate msg s"column name must be a simple identifier,
composite column name '${id.toString}' is not supported yet"
Review Comment:
The length check seems ok when running style check, I didn't find a suitable
resource for the error message when do poc, and will see if we can get rid of
the TODO.
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/operations/SqlToOperationConverter.java:
##
@@ -1558,8 +1572,16 @@ private Operation convertUpdate(SqlUpdate sqlUpdate) {
catalogManager.qualifyIdentifier(unresolvedTableIdentifier));
// get query
PlannerQueryOperation queryOperation = new
PlannerQueryOperation(tableModify);
+// TODO
+List updateColumnList =
+sqlUpdate.getTargetColumnList().stream()
+.map(c -> ((SqlIdentifier) c).getSimple())
+.collect(Collectors.toList());
return new SinkModifyOperation(
-contextResolvedTable, queryOperation,
SinkModifyOperation.ModifyType.UPDATE);
+contextResolvedTable,
+queryOperation,
+null, // targetColumns
Review Comment:
yes, here should be updateColumnList, seems some local changes get lost..
I'll update it.
##
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/batch/sql/RowLevelUpdateTest.xml:
##
Review Comment:
I think manually reorder the test xml file will add more cost for
maintaining since the auto generated mechanism doesn't care about the method
order, and once we add some new tests in the future then just re-generate it is
ok, WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a diff in pull request #219: [FLINK-31422] Add Servable for Logistic Regression Model
lindong28 commented on code in PR #219:
URL: https://github.com/apache/flink-ml/pull/219#discussion_r1141014031
##
flink-ml-lib/src/test/java/org/apache/flink/ml/classification/LogisticRegressionTest.java:
##
@@ -282,6 +287,50 @@ public void testSetModelData() throws Exception {
logisticRegression.getRawPredictionCol());
}
+@Test
+public void testSaveLoadServableAndPredict() throws Exception {
+LogisticRegression logisticRegression = new
LogisticRegression().setWeightCol("weight");
+LogisticRegressionModel model =
logisticRegression.fit(binomialDataTable);
+
+LogisticRegressionModelServable servable =
+saveAndLoadServable(
+tEnv,
+model,
+tempFolder.newFolder().getAbsolutePath(),
+LogisticRegressionModel::loadServable);
+
+assertEquals("features", servable.getFeaturesCol());
+assertEquals("prediction", servable.getPredictionCol());
+assertEquals("rawPrediction", servable.getRawPredictionCol());
+
+DataFrame output =
servable.transform(LogisticRegressionModelServableTest.PREDICT_DATA);
+LogisticRegressionModelServableTest.verifyPredictionResult(
+output,
+servable.getFeaturesCol(),
+servable.getPredictionCol(),
+servable.getRawPredictionCol());
+}
+
+@Test
+public void testSetModelDataToServable() throws Exception {
+LogisticRegression logisticRegression = new
LogisticRegression().setWeightCol("weight");
+LogisticRegressionModel model =
logisticRegression.fit(binomialDataTable);
+byte[] serializedModelData =
+
LogisticRegressionModelDataUtil.getModelDataByteStream(model.getModelData()[0])
+.executeAndCollect()
+.next();
+
+LogisticRegressionModelServable servable = new
LogisticRegressionModelServable();
+servable.setModelData(new ByteArrayInputStream(serializedModelData));
Review Comment:
Would it be useful to forward parameters from model to this servable using
`ParamUtils.updateExistingParams` so that later tests can follow the same
practice consistently?
##
flink-ml-servable-lib/src/test/java/org/apache/flink/ml/classification/LogisticRegressionModelServableTest.java:
##
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification;
+
+import
org.apache.flink.ml.classification.logisticregression.LogisticRegressionModelData;
+import
org.apache.flink.ml.classification.logisticregression.LogisticRegressionModelServable;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.servable.api.DataFrame;
+import org.apache.flink.ml.servable.api.Row;
+import org.apache.flink.ml.servable.types.BasicType;
+import org.apache.flink.ml.servable.types.DataTypes;
+
+import org.junit.Test;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/** Tests the {@link LogisticRegressionModelServable}. */
+public class LogisticRegressionModelServableTest {
+
+protected static final DataFrame PREDICT_DATA =
+new DataFrame(
+new ArrayList<>(Arrays.asList("features", "label",
"weight")),
+new ArrayList<>(
+Arrays.asList(
+DataTypes.VECTOR(BasicType.DOUBLE),
+DataTypes.DOUBLE,
+DataTypes.DOUBLE)),
+Arrays.asList(
+new Row(
+new ArrayList<>(
+Arrays.asList(Vectors.dense(1, 2,
3, 4), 0., 1.))),
+new Row(
+
[jira] [Created] (FLINK-31511) Translate documentation sql_functions_zh.yml to the latest version
Hang Ruan created FLINK-31511: - Summary: Translate documentation sql_functions_zh.yml to the latest version Key: FLINK-31511 URL: https://issues.apache.org/jira/browse/FLINK-31511 Project: Flink Issue Type: Improvement Components: Documentation Reporter: Hang Ruan Some content of these functions in sql_functions_zh.yml is outdated. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31511) Translate documentation sql_functions_zh.yml to the latest version
[ https://issues.apache.org/jira/browse/FLINK-31511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17702109#comment-17702109 ] Hang Ruan commented on FLINK-31511: --- Hi, all, I would like to help update this document as some content is out dated. > Translate documentation sql_functions_zh.yml to the latest version > --- > > Key: FLINK-31511 > URL: https://issues.apache.org/jira/browse/FLINK-31511 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Hang Ruan >Priority: Minor > > Some content of these functions in sql_functions_zh.yml is outdated. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] chucheng92 commented on pull request #22179: [FLINK-31380][table] FLIP-297: Support enhanced show catalogs syntax
chucheng92 commented on PR #22179: URL: https://github.com/apache/flink/pull/22179#issuecomment-1474829928 @Aitozi hi, Considering using a SupportLike parent class may make confused, i have changed and rebased the code. thanks for reviewing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-31319) Kafka new source partitionDiscoveryIntervalMs=0 cause bounded source can not quit
[ https://issues.apache.org/jira/browse/FLINK-31319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu resolved FLINK-31319. Fix Version/s: 1.16.2 Resolution: Fixed Fixed by: * flink-connector-kafka main : 58f35374b6aec63491623321f4de69a0affa629a * flink 1.17: 09fb503f0f3bba2fa0bbab9452baebe07af288cc * flink 1.16: a5f085e042be70f45485165c5755171ac5ed8cb4 > Kafka new source partitionDiscoveryIntervalMs=0 cause bounded source can not > quit > - > > Key: FLINK-31319 > URL: https://issues.apache.org/jira/browse/FLINK-31319 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.17.0, 1.16.1 >Reporter: Ran Tao >Assignee: Ran Tao >Priority: Critical > Labels: pull-request-available > Fix For: 1.17.0, 1.16.2 > > Attachments: image-2023-03-04-01-37-29-360.png, > image-2023-03-04-01-39-20-352.png, image-2023-03-04-01-40-44-124.png, > image-2023-03-04-01-41-55-664.png > > > As kafka option description, partitionDiscoveryIntervalMs <=0 means disabled. > !image-2023-03-04-01-37-29-360.png|width=781,height=147! > just like start kafka enumerator: > !image-2023-03-04-01-39-20-352.png|width=465,height=311! > but inner > handlePartitionSplitChanges use error if condition( < 0): > !image-2023-03-04-01-40-44-124.png|width=576,height=237! > > it will cause noMoreNewPartitionSplits can not be set to true. > !image-2023-03-04-01-41-55-664.png|width=522,height=610! > Finally cause bounded source can not signalNoMoreSplits, so it will not quit. > Besides，Both ends of the if condition should be mutually exclusive. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] leonardBang merged pull request #22192: [FLINK-31319][BP-1.16][connectors/kafka] Fix kafka new source partitionDiscoveryIntervalMs=0 cause bounded source can not quit
leonardBang merged PR #22192: URL: https://github.com/apache/flink/pull/22192 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-connector-kafka] boring-cyborg[bot] commented on pull request #8: [FLINK-31319][connectors/kafka] Fix kafka new source partitionDiscoveryIntervalMs=0 cause bounded source can not quit
boring-cyborg[bot] commented on PR #8: URL: https://github.com/apache/flink-connector-kafka/pull/8#issuecomment-1474791944 Awesome work, congrats on your first merged pull request! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-connector-kafka] leonardBang merged pull request #8: [FLINK-31319][connectors/kafka] Fix kafka new source partitionDiscoveryIntervalMs=0 cause bounded source can not quit
leonardBang merged PR #8: URL: https://github.com/apache/flink-connector-kafka/pull/8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-31505) Move execution logic of DropOperation out from TableEnvironmentImpl
[
https://issues.apache.org/jira/browse/FLINK-31505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Nuyanzin reassigned FLINK-31505:
---
Assignee: Sergey Nuyanzin
> Move execution logic of DropOperation out from TableEnvironmentImpl
> ---
>
> Key: FLINK-31505
> URL: https://issues.apache.org/jira/browse/FLINK-31505
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Reporter: Jark Wu
>Assignee: Sergey Nuyanzin
>Priority: Major
>
> This should implement {{ExecutableOperation}} for all the {{DropOperation}}s
> to move the execution logic out from
> {{TableEnvironmentImpl#executeInternal()}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-ml] jiangxin369 commented on pull request #219: [FLINK-31422] Add Servable for Logistic Regression Model
jiangxin369 commented on PR #219: URL: https://github.com/apache/flink-ml/pull/219#issuecomment-1474774669 @lindong28 Thanks for the review, could you help review again? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-30229) SQL IT case for DynamoDB Sink
[
https://issues.apache.org/jira/browse/FLINK-30229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17702083#comment-17702083
]
Danny Cranmer commented on FLINK-30229:
---
Merged commit
[{{930d196}}|https://github.com/apache/flink-connector-aws/commit/930d196a005c52f23ada6acb6ff47c17a2f6d7d5]
into apache:main
> SQL IT case for DynamoDB Sink
> -
>
> Key: FLINK-30229
> URL: https://issues.apache.org/jira/browse/FLINK-30229
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / DynamoDB
>Reporter: Danny Cranmer
>Assignee: Daren Wong
>Priority: Major
> Labels: pull-request-available
> Fix For: aws-connector-4.2.0
>
>
> Add SQL IT case(s) for DynamoDB sink
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-30229) SQL IT case for DynamoDB Sink
[ https://issues.apache.org/jira/browse/FLINK-30229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Cranmer resolved FLINK-30229. --- Resolution: Fixed > SQL IT case for DynamoDB Sink > - > > Key: FLINK-30229 > URL: https://issues.apache.org/jira/browse/FLINK-30229 > Project: Flink > Issue Type: Improvement > Components: Connectors / DynamoDB >Reporter: Danny Cranmer >Assignee: Daren Wong >Priority: Major > Labels: pull-request-available > Fix For: aws-connector-4.2.0 > > > Add SQL IT case(s) for DynamoDB sink -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-connector-aws] dannycranmer merged pull request #42: [FLINK-30229] Add SQL IT case for DynamoDB Sink
dannycranmer merged PR #42: URL: https://github.com/apache/flink-connector-aws/pull/42 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
