[jira] [Created] (FLINK-18105) Test submitting Java SQL job with Python UDF
sunjincheng created FLINK-18105: --- Summary: Test submitting Java SQL job with Python UDF Key: FLINK-18105 URL: https://issues.apache.org/jira/browse/FLINK-18105 Project: Flink Issue Type: Test Components: API / Python Reporter: sunjincheng Fix For: 1.11.0 Test submitting Java SQL job with Python UDF, Python job with UDF via flink run (include yarn perjob, yarn session, standalone session) with Java dependency management (include pipeline.jar pipeline.classpaths) with Python dependency management (include pyfs, pyreq, pyarch, pyexec, PYFLINK_CLIENT_EXECUTABLE) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18106) Tests Python UDTF support
sunjincheng created FLINK-18106: --- Summary: Tests Python UDTF support Key: FLINK-18106 URL: https://issues.apache.org/jira/browse/FLINK-18106 Project: Flink Issue Type: Test Components: API / Python Reporter: sunjincheng Assignee: Huang Xingbo Fix For: 1.11.0 Check items as follows: * test Python UDTF (inner join/left join) in Blink Planner in batch mode * test Python UDTF (inner join/left join) in Blink Planner in streaming mode * test Python UDTF (inner join/left join) in Flink Planner in batch mode * test Python UDTF (inner join/left join) in Flink Planner in streaming mode -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18107) Performance tests for PyFlink UDFs
sunjincheng created FLINK-18107: --- Summary: Performance tests for PyFlink UDFs Key: FLINK-18107 URL: https://issues.apache.org/jira/browse/FLINK-18107 Project: Flink Issue Type: Test Components: API / Python Reporter: sunjincheng Assignee: Huang Xingbo Fix For: 1.11.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18108) Wheel package consistency checks
sunjincheng created FLINK-18108: --- Summary: Wheel package consistency checks Key: FLINK-18108 URL: https://issues.apache.org/jira/browse/FLINK-18108 Project: Flink Issue Type: Test Components: API / Python Reporter: sunjincheng Assignee: Huang Xingbo Fix For: 1.11.0 Checks wheel packages consistent with built from source code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18775) Rework PyFlink Documentation
sunjincheng created FLINK-18775: --- Summary: Rework PyFlink Documentation Key: FLINK-18775 URL: https://issues.apache.org/jira/browse/FLINK-18775 Project: Flink Issue Type: Improvement Components: API / Python, Documentation Affects Versions: 1.11.1, 1.11.0 Reporter: sunjincheng Fix For: 1.12.0, 1.11.2, 1.11.1, 1.11.0 Since the release of Flink 1.11, users of PyFlink have continued to grow. According to the feedback we received, current Flink documentation is not very friendly to PyFlink users. There are two shortcomings: # Python related content is mixed in the Java/Scala documentation, which makes it difficult for users who only focus on PyFlink to read. # There is already a "Python Table API" section in the Table API document to store PyFlink documents, but the number of articles is small and the content is fragmented. It is difficult for beginners to learn from it. In addition, [FLIP-130|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866298] introduced the Python DataStream API. Many documents will be added for those new APIs. In order to increase the readability and maintainability of the PyFlink document, we would like to rework it via this umbrella JIRA. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18816) Correct API usage in Pyflink Dependency Management page
sunjincheng created FLINK-18816: --- Summary: Correct API usage in Pyflink Dependency Management page Key: FLINK-18816 URL: https://issues.apache.org/jira/browse/FLINK-18816 Project: Flink Issue Type: Improvement Components: API / Python, Documentation Affects Versions: 1.11.1, 1.11.0 Reporter: sunjincheng Correct the API usage in the doc of page[1]. Changes: >From `table_env.get_config().set_configuration` to >`table_env.get_config().get_configuration().set_string`. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/python/dependency_management.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-7020) Upgrade calcite to calcite 1.13
sunjincheng created FLINK-7020: -- Summary: Upgrade calcite to calcite 1.13 Key: FLINK-7020 URL: https://issues.apache.org/jira/browse/FLINK-7020 Project: Flink Issue Type: Improvement Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng Assignee: sunjincheng Calcite 1.13 has released, So I'll try to upgrade the dependence. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7024) Add supported for selecting window proctime/rowtime on row-based Tumble/Slide window
sunjincheng created FLINK-7024: -- Summary: Add supported for selecting window proctime/rowtime on row-based Tumble/Slide window Key: FLINK-7024 URL: https://issues.apache.org/jira/browse/FLINK-7024 Project: Flink Issue Type: Bug Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng Assignee: sunjincheng We get validate exception,when selecting window.proctime/rowtime on row-based group window. {code} table .window(Tumble over 2.rows on 'proctime as 'w) .groupBy('w, 'string) .select('string, countFun('string) as 'cnt, 'w.rowtime as 'proctime) .window(Over partitionBy 'string orderBy 'proctime preceding UNBOUNDED_RANGE following CURRENT_RANGE as 'w2) .select('string, 'cnt.sum over 'w2 as 'cnt) {code} Exception: {code} org.apache.flink.table.api.ValidationException: Window start and Window end cannot be selected for a row-count Tumbling window. at org.apache.flink.table.plan.logical.LogicalNode.failValidation(LogicalNode.scala:143) at org.apache.flink.table.plan.logical.WindowAggregate.validate(operators.scala:660) {code} We should add window.proctime/rowtime check in `validate ` method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7025) Using NullByteKeySelector for Unbounded ProcTime NonPartitioned Over
sunjincheng created FLINK-7025: -- Summary: Using NullByteKeySelector for Unbounded ProcTime NonPartitioned Over Key: FLINK-7025 URL: https://issues.apache.org/jira/browse/FLINK-7025 Project: Flink Issue Type: Bug Reporter: sunjincheng Assignee: sunjincheng Currently we added `Cleanup State` feature. But It not work well if we enabled the stateCleaning on Unbounded ProcTime NonPartitioned Over window, Because in `ProcessFunctionWithCleanupState` we has using the keyed state. So, In this JIRA. I'll change the `Unbounded ProcTime NonPartitioned Over` to `partitioned Over` by using NullByteKeySelector. OR created a `NonKeyedProcessFunctionWithCleanupState`. But I think the first way is simpler. What do you think? [~fhueske] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7101) Fix Non-windowed group-aggregate error when using `minIdleStateRetentionTime` config and retract agg
sunjincheng created FLINK-7101: -- Summary: Fix Non-windowed group-aggregate error when using `minIdleStateRetentionTime` config and retract agg Key: FLINK-7101 URL: https://issues.apache.org/jira/browse/FLINK-7101 Project: Flink Issue Type: Bug Components: Table API & SQL Affects Versions: 1.3.1, 1.3.0 Reporter: sunjincheng Assignee: sunjincheng Fix For: 1.4.0 When Non-windowed group-aggregate using {{minIdleStateRetentionTime}} config and retract AGG, Will emit "NULL" agg value which we do not expect. For example: ({{IntSumWithRetractAggFunction}}) 1. Receive: CRow(Row.of(6L: JLong, 5: JInt, "aaa"), true) 2. Cleanup state 3. Receive: CRow(Row.of(6L: JLong, 5: JInt, "aaa"), false) // acc.f1 = -1, getValue= null So, we must change the logic of {{GroupAggProcessFunction}} as follows: {code} if (inputCnt != 0) { ... } else { ... } {code} TO {code} if (inputCnt > 0) { ... } else { if( null != prevRow.row){ ... } } {code} What do you think? [~fhueske] [~hequn8128] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7173) Fix the illustration of tumbling window.
sunjincheng created FLINK-7173: -- Summary: Fix the illustration of tumbling window. Key: FLINK-7173 URL: https://issues.apache.org/jira/browse/FLINK-7173 Project: Flink Issue Type: Bug Components: DataStream API, Documentation Reporter: sunjincheng Assignee: sunjincheng -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7205) Add UUID supported in TableAPI/SQL
sunjincheng created FLINK-7205: -- Summary: Add UUID supported in TableAPI/SQL Key: FLINK-7205 URL: https://issues.apache.org/jira/browse/FLINK-7205 Project: Flink Issue Type: Sub-task Affects Versions: 1.4.0 Reporter: sunjincheng UUID() returns a value that conforms to UUID version 1 as described in RFC 4122. The value is a 128-bit number represented as a utf8 string of five hexadecimal numbers in ---- format: The first three numbers are generated from the low, middle, and high parts of a timestamp. The high part also includes the UUID version number. The fourth number preserves temporal uniqueness in case the timestamp value loses monotonicity (for example, due to daylight saving time). The fifth number is an IEEE 802 node number that provides spatial uniqueness. A random number is substituted if the latter is not available (for example, because the host device has no Ethernet card, or it is unknown how to find the hardware address of an interface on the host operating system). In this case, spatial uniqueness cannot be guaranteed. Nevertheless, a collision should have very low probability. See: [RFC 4122: http://www.ietf.org/rfc/rfc4122.txt|http://www.ietf.org/rfc/rfc4122.txt] See detailed semantics: MySql: [https://dev.mysql.com/doc/refman/5.7/en/miscellaneous-functions.html#function_uuid|https://dev.mysql.com/doc/refman/5.7/en/miscellaneous-functions.html#function_uuid] Welcome anybody feedback -:). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7342) Add support for Regular Expression Operators (RLIKE)
sunjincheng created FLINK-7342: -- Summary: Add support for Regular Expression Operators (RLIKE) Key: FLINK-7342 URL: https://issues.apache.org/jira/browse/FLINK-7342 Project: Flink Issue Type: Bug Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng Currently Regular Expression Operators (RLIKE) are not supported. e.g.: {code} select * from emps name rlike '^(K|G|W)' {code} we'll get Error: {code} Error: Error while executing SQL "select * from emps name rlike '^(K|G|W)'": parse failed: Encountered "rlike" at line 1, column 25. {code} So, after https://issues.apache.org/jira/browse/CALCITE-1918 fixed. I want add RLIKE supported in flink (TableAPI and SQL). Reference: MySql: https://dev.mysql.com/doc/refman/5.7/en/regexp.html#operator_regexp -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7358) Add implicitly converts support for User-defined function
sunjincheng created FLINK-7358: -- Summary: Add implicitly converts support for User-defined function Key: FLINK-7358 URL: https://issues.apache.org/jira/browse/FLINK-7358 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng Currently if user defined a UDF as follows: {code} object Func extends ScalarFunction { def eval(a: Int, b: Long): String = { ... } } {code} And if the table schema is (a: Int, b: int, c: String), then we can not call the UDF `Func('a, 'b)`. So I want add implicitly converts when we call UDF. *Note: In this JIRA. only for TableAPI, And SQL will be fixed in https://issues.apache.org/jira/browse/CALCITE-1908.* What do you think? [~fhueske] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
dev@flink.apache.org
sunjincheng created FLINK-7464: -- Summary: Add useful build-in Aggregate function into TabalAPI&SQL Key: FLINK-7464 URL: https://issues.apache.org/jira/browse/FLINK-7464 Project: Flink Issue Type: New Feature Components: Table API & SQL Reporter: sunjincheng In this JIRA, will create some sub-task for add specific build-in aggregate function, such as FIRST_VALUE, LAST_VALUE, BloomFilterCount etc. Welcome anybody to add the sub-task. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
dev@flink.apache.org
sunjincheng created FLINK-7465: -- Summary: Add build-in BloomFilterCount on TableAPI&SQL Key: FLINK-7465 URL: https://issues.apache.org/jira/browse/FLINK-7465 Project: Flink Issue Type: Sub-task Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng In this JIRA. use BloomFilter to implement counting functions. BloomFilter Algorithm description: An empty Bloom filter is a bit array of m bits, all set to 0. There must also be k different hash functions defined, each of which maps or hashes some set element to one of the m array positions, generating a uniform random distribution. Typically, k is a constant, much smaller than m, which is proportional to the number of elements to be added; the precise choice of k and the constant of proportionality of m are determined by the intended false positive rate of the filter. To add an element, feed it to each of the k hash functions to get k array positions. Set the bits at all these positions to 1. To query for an element (test whether it is in the set), feed it to each of the k hash functions to get k array positions. If any of the bits at these positions is 0, the element is definitely not in the set – if it were, then all the bits would have been set to 1 when it was inserted. If all are 1, then either the element is in the set, or the bits have by chance been set to 1 during the insertion of other elements, resulting in a false positive. An example of a Bloom filter, representing the set {x, y, z}. The colored arrows show the positions in the bit array that each set element is mapped to. The element w is not in the set {x, y, z}, because it hashes to one bit-array position containing 0. For this figure, m = 18 and k = 3. The sketch as follows: !https://en.wikipedia.org/wiki/Bloom_filter#/media/File:Bloom_filter.svg! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7471) Improve bounded OVER support non-retract method AGG
sunjincheng created FLINK-7471: -- Summary: Improve bounded OVER support non-retract method AGG Key: FLINK-7471 URL: https://issues.apache.org/jira/browse/FLINK-7471 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng Currently BOUNDED OVER WINDOW only support have {{retract}} method AGG. In this JIRA. will add non-retract method support. What do you think? [~fhueske] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7485) Using DataView interface to improve (MIN/MAX)WithRetractAggFunction.
sunjincheng created FLINK-7485: -- Summary: Using DataView interface to improve (MIN/MAX)WithRetractAggFunction. Key: FLINK-7485 URL: https://issues.apache.org/jira/browse/FLINK-7485 Project: Flink Issue Type: Improvement Reporter: sunjincheng Currently MIN/MAX using memory structure {{HashMap}} to store all values, after FLINK-7206 we can improve them by using {{DataView}}. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7558) Improve SQL ValidationException message.
sunjincheng created FLINK-7558: -- Summary: Improve SQL ValidationException message. Key: FLINK-7558 URL: https://issues.apache.org/jira/browse/FLINK-7558 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng org.apache.flink.table.api.ValidationException: SQL validation failed. Operand types of could not be inferred. at org.apache.flink.table.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:91) at org.apache.flink.table.api.TableEnvironment.sql(TableEnvironment.scala:513) at com.alibaba.blink.scala.tool.util.SqlJobAdapter.dealInserts(SqlJobAdapter.java:292) at com.alibaba.blink.scala.tool.util.JobBuildHelper.buildSqlJob(JobBuildHelper.java:80) at com.alibaba.blink.scala.tool.JobLauncher.main(JobLauncher.java:138) Caused by: org.apache.flink.table.api.ValidationException: Operand types of could not be inferred. at org.apache.flink.table.functions.utils.ScalarSqlFunction$$anon$2$$anonfun$2.apply(ScalarSqlFunction.scala:110) at org.apache.flink.table.functions.utils.ScalarSqlFunction$$anon$2$$anonfun$2.apply(ScalarSqlFunction.scala:110) at scala.Option.getOrElse(Option.scala:121) at org.apache.flink.table.functions.utils.ScalarSqlFunction$$anon$2.inferOperandTypes(ScalarSqlFunction.scala:110) at org.apache.calcite.sql.validate.SqlValidatorImpl.inferUnknownTypes(SqlValidatorImpl.java:1769) at -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7572) Improve TableSchema ValidationException message.
sunjincheng created FLINK-7572: -- Summary: Improve TableSchema ValidationException message. Key: FLINK-7572 URL: https://issues.apache.org/jira/browse/FLINK-7572 Project: Flink Issue Type: Improvement Reporter: sunjincheng Assignee: sunjincheng When sql is very complicated(e.g.: more than 100+ tables), the following exception information is very confusing: {code} org.apache.flink.table.api.TableException: Table column names must be unique. at org.apache.flink.table.api.TableSchema.(TableSchema.scala:36) {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7618) Add BINARY supported in FlinkTypeFactory
sunjincheng created FLINK-7618: -- Summary: Add BINARY supported in FlinkTypeFactory Key: FLINK-7618 URL: https://issues.apache.org/jira/browse/FLINK-7618 Project: Flink Issue Type: Bug Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng We will get the following exception when we deal with the BINARY type. {code} org.apache.flink.table.api.TableException: Type is not supported: BINARY at org.apache.flink.table.api.TableException$.apply(exceptions.scala:53) at org.apache.flink.table.calcite.FlinkTypeFactory$.toTypeInfo(FlinkTypeFactory.scala:377) at org.apache.flink.table.codegen.CodeGenerator.visitCall(CodeGenerator.scala:741) at org.apache.flink.table.codegen.CodeGenerator.visitCall(CodeGenerator.scala:67) at org.apache.calcite.rex.RexCall.accept(RexCall.java:104) at org.apache.flink.table.codegen.CodeGenerator$$anonfun$16.apply(CodeGenerator.scala:754) at org.apache.flink.table.codegen.CodeGenerator$$anonfun$16.apply(CodeGenerator.scala:744) {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-8323) Fix Mod scala function bug
sunjincheng created FLINK-8323: -- Summary: Fix Mod scala function bug Key: FLINK-8323 URL: https://issues.apache.org/jira/browse/FLINK-8323 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.5.0 Reporter: sunjincheng Assignee: sunjincheng As we know mod(151435632,6)=0, but currently we get `-15488` when call `MOD(151435632,6)`. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-8325) Add support for constant parameters in GROUP WINDOW aggregate
sunjincheng created FLINK-8325: -- Summary: Add support for constant parameters in GROUP WINDOW aggregate Key: FLINK-8325 URL: https://issues.apache.org/jira/browse/FLINK-8325 Project: Flink Issue Type: Bug Components: Table API & SQL Affects Versions: 1.5.0 Reporter: sunjincheng Assignee: sunjincheng COUNT(1) with Group Window, always output 0. e.g. DATA: {code} val data = List( (1L, 1, "Hi"), (2L, 2, "Hello"), (4L, 2, "Hello"), (8L, 3, "Hello world"), (16L, 3, "Hello world")) {code} SQL: {code} SELECT b, COUNT(1) FROM MyTable GROUP BY Hop(proctime, interval '0.001' SECOND, interval '0.002' SECOND),b {code} OUTPUT: {code} 1,0,1, 1,0,1, 2,0,1, 2,0,1, 2,0,2, 3,0,1, 3,0,1 {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-8355) DataSet Should not union a NULL row for AGG without GROUP BY clause.
sunjincheng created FLINK-8355: -- Summary: DataSet Should not union a NULL row for AGG without GROUP BY clause. Key: FLINK-8355 URL: https://issues.apache.org/jira/browse/FLINK-8355 Project: Flink Issue Type: Bug Components: Table API & SQL Affects Versions: 1.5.0 Reporter: sunjincheng Currently {{DataSetAggregateWithNullValuesRule}} will UINON a NULL row for non grouped aggregate query. when {{CountAggFunction}} support {{COUNT(*)}}(FLINK-8325). the result will incorrect. for example, if Tabble {{T1}} has 3 records. when we run the follow SQL in DataSet: {code} SELECT COUNT(*) as cnt from Tab // cnt = 4(incorrect). {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-5105) Improve ReduceState: value put into ReducingState should always be a copy
sunjincheng created FLINK-5105: -- Summary: Improve ReduceState: value put into ReducingState should always be a copy Key: FLINK-5105 URL: https://issues.apache.org/jira/browse/FLINK-5105 Project: Flink Issue Type: Bug Components: State Backends, Checkpointing Reporter: sunjincheng In case of overlapping sliding windows, multiple references are hold on the same object. If we modify value1 or value2 the results are incorrect. The value that is put into a ReducingState is always copied. That would allow to modify and emit one of the two input values (the one which is comes from the state). The FoldingState has the same problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-5106) improving IncrementalAggregateReduceFunction
sunjincheng created FLINK-5106: -- Summary: improving IncrementalAggregateReduceFunction Key: FLINK-5106 URL: https://issues.apache.org/jira/browse/FLINK-5106 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng Please refer to FLINK-4937. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-5219) Add non-grouped session windows for batch tables
sunjincheng created FLINK-5219: -- Summary: Add non-grouped session windows for batch tables Key: FLINK-5219 URL: https://issues.apache.org/jira/browse/FLINK-5219 Project: Flink Issue Type: Sub-task Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng Add non-grouped session windows for batch tables as described in [FLIP-11|https://cwiki.apache.org/confluence/display/FLINK/FLIP-11%3A+Table+API+Stream+Aggregations]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-5386) Refacturing Window Clause
sunjincheng created FLINK-5386: -- Summary: Refacturing Window Clause Key: FLINK-5386 URL: https://issues.apache.org/jira/browse/FLINK-5386 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-5695) Optimize table type systems based on database semantics
sunjincheng created FLINK-5695: -- Summary: Optimize table type systems based on database semantics Key: FLINK-5695 URL: https://issues.apache.org/jira/browse/FLINK-5695 Project: Flink Issue Type: Sub-task Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng Optimize table type systems based on database semantics.As follows: groupBy > Table GroupedTable ∧ < ∧ | select&agg| | | | where| | select| groupBy | agg| | ... | | window | ∨ -> TableWindowedTable <- select&over So, I want remove table Type "WindowGroupedTable". -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-5776) Improve XXMapRunner support create instance by carrying constructor parameters
sunjincheng created FLINK-5776: -- Summary: Improve XXMapRunner support create instance by carrying constructor parameters Key: FLINK-5776 URL: https://issues.apache.org/jira/browse/FLINK-5776 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng At present, MapRunner FlatMapRunner only supports create non-parameter instance, but sometimes we need to carry constructor parameters to instantiate, so I would like to improve XXMapRunner support create instance by carrying constructor parameters. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-5792) Improve “UDF/UDTF" to support with parameter constructor
sunjincheng created FLINK-5792: -- Summary: Improve “UDF/UDTF" to support with parameter constructor Key: FLINK-5792 URL: https://issues.apache.org/jira/browse/FLINK-5792 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng Improved “UDF/UDTF" to support with parameter constructor. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-5794) Improve “UDF" to support with parameter constructor
sunjincheng created FLINK-5794: -- Summary: Improve “UDF" to support with parameter constructor Key: FLINK-5794 URL: https://issues.apache.org/jira/browse/FLINK-5794 Project: Flink Issue Type: Improvement Reporter: sunjincheng Assignee: sunjincheng Depends on FLINK-5792 . -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-5795) Improve “UDTF" to support with parameter constructor
sunjincheng created FLINK-5795: -- Summary: Improve “UDTF" to support with parameter constructor Key: FLINK-5795 URL: https://issues.apache.org/jira/browse/FLINK-5795 Project: Flink Issue Type: Sub-task Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-5803) Add [partitioned] processing time OVER RANGE BETWEEN UNBOUNDED PRECEDING aggregation to SQL
sunjincheng created FLINK-5803: -- Summary: Add [partitioned] processing time OVER RANGE BETWEEN UNBOUNDED PRECEDING aggregation to SQL Key: FLINK-5803 URL: https://issues.apache.org/jira/browse/FLINK-5803 Project: Flink Issue Type: Sub-task Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-5804) Add [non-partitioned] processing time OVER RANGE BETWEEN UNBOUNDED PRECEDING aggregation to SQL
sunjincheng created FLINK-5804: -- Summary: Add [non-partitioned] processing time OVER RANGE BETWEEN UNBOUNDED PRECEDING aggregation to SQL Key: FLINK-5804 URL: https://issues.apache.org/jira/browse/FLINK-5804 Project: Flink Issue Type: Sub-task Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-5957) Remove `getAccumulatorType` method from `AggregateFunction`
sunjincheng created FLINK-5957: -- Summary: Remove `getAccumulatorType` method from `AggregateFunction` Key: FLINK-5957 URL: https://issues.apache.org/jira/browse/FLINK-5957 Project: Flink Issue Type: Sub-task Reporter: sunjincheng Assignee: sunjincheng Build-in aggregateFunction need not implement the `getAccumulatorType` method. We can get TypeInformation by `TypeInformation.of() ` or `TypeInformation.of(new TypeHint[AGG.type](){})`. What do you think? [~fhueske] [~shaoxuan] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-5990) Add event time OVER ROWS BETWEEN x PRECEDING aggregation to SQL
sunjincheng created FLINK-5990: -- Summary: Add event time OVER ROWS BETWEEN x PRECEDING aggregation to SQL Key: FLINK-5990 URL: https://issues.apache.org/jira/browse/FLINK-5990 Project: Flink Issue Type: Sub-task Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-5995) Get a Exception when creating the ListStateDescriptor with a TypeInformation
sunjincheng created FLINK-5995: -- Summary: Get a Exception when creating the ListStateDescriptor with a TypeInformation Key: FLINK-5995 URL: https://issues.apache.org/jira/browse/FLINK-5995 Project: Flink Issue Type: Bug Components: Core Reporter: sunjincheng When use OperatorState and creating the ListStateDescriptor with a TypeInformation,I got a exception. The Exception info is: {code} org.apache.flink.runtime.client.JobExecutionException: Job execution failed. at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$mcV$sp(JobManager.scala:915) at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:858) at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:858) at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Caused by: java.lang.IllegalStateException: Serializer not yet initialized. at org.apache.flink.api.common.state.StateDescriptor.getSerializer(StateDescriptor.java:169) at org.apache.flink.api.common.state.ListStateDescriptor.getElementSerializer(ListStateDescriptor.java:93) at org.apache.flink.runtime.state.DefaultOperatorStateBackend.getOperatorState(DefaultOperatorStateBackend.java:110) at org.apache.flink.runtime.state.DefaultOperatorStateBackend.getOperatorState(DefaultOperatorStateBackend.java:91) at org.apache.flink.table.runtime.aggregate.UnboundedNonPartitionedProcessingOverProcessFunction.initializeState(UnboundedNonPartitionedProcessingOverProcessFunction.scala:104) at org.apache.flink.streaming.api.functions.util.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178) at org.apache.flink.streaming.api.functions.util.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160) at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:106) at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:242) at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:681) at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:669) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:670) at java.lang.Thread.run(Thread.java:745) {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6018) Minor improvements about `AbstractKeyedStateBackend#getPartitionedState` method
sunjincheng created FLINK-6018: -- Summary: Minor improvements about `AbstractKeyedStateBackend#getPartitionedState` method Key: FLINK-6018 URL: https://issues.apache.org/jira/browse/FLINK-6018 Project: Flink Issue Type: Improvement Components: DataStream API, State Backends, Checkpointing Reporter: sunjincheng Assignee: sunjincheng The code snippet currently in the `AbstractKeyedStateBackend # getPartitionedState` method, as follows: {code} // TODO: This is wrong, it should throw an exception that the initialization has not properly happened if (!stateDescriptor.isSerializerInitialized()) { stateDescriptor.initializeSerializerUnlessSet(new ExecutionConfig()); } {code} I hope this code has a little improvement to the following: approach 1: According to the `TODO` information we throw an exception {code} if (!stateDescriptor.isSerializerInitialized()) { throw new IllegalStateException("The serializer of the descriptor has not been initialized!"); } {code} approach 2: Try to initialize and remove `if (!stateDescriptor.isSerializerInitialized()) {` logic. {code} stateDescriptor.initializeSerializerUnlessSet(new ExecutionConfig()); {code} Meanwhile, If we use the approach 2, I suggest that `AbstractKeyedStateBackend` add a `private final ExecutionConfig executionConfig` property. then we can change the code like this: {code} stateDescriptor.initializeSerializerUnlessSet(executionConfig); {code} Are the above suggestions reasonable for you? Welcome anybody's feedback and corrections. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6055) Supported setting timers on a Non-Keyed Stream
sunjincheng created FLINK-6055: -- Summary: Supported setting timers on a Non-Keyed Stream Key: FLINK-6055 URL: https://issues.apache.org/jira/browse/FLINK-6055 Project: Flink Issue Type: New Feature Reporter: sunjincheng Assignee: sunjincheng After [FLINK-4460] Allow ProcessFunction on non-keyed streams, I want supported setting timers on a Non-Keyed Stream. What do you think? [~aljoscha] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6074) Fix
sunjincheng created FLINK-6074: -- Summary: Fix Key: FLINK-6074 URL: https://issues.apache.org/jira/browse/FLINK-6074 Project: Flink Issue Type: Bug Reporter: sunjincheng -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6097) Guaranteed the order of the extracted field references
sunjincheng created FLINK-6097: -- Summary: Guaranteed the order of the extracted field references Key: FLINK-6097 URL: https://issues.apache.org/jira/browse/FLINK-6097 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng The current `ProjectionTranslator # identifyFieldReferences` method uses the` Set` temporary save field, causing the field to be out of order, although the out of order is also working, but I think the order is better than out of order. So I want to improve it,Orderly extraction field. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6138) Improve UnboundedNonPartitionedProcessingOverProcessFunction
sunjincheng created FLINK-6138: -- Summary: Improve UnboundedNonPartitionedProcessingOverProcessFunction Key: FLINK-6138 URL: https://issues.apache.org/jira/browse/FLINK-6138 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng Hi,[~fhueske] Just like you suggested in FLINK-5804, I completed the FLINK-5995 bug fix, now I want Create the ListStateDescriptor with the aggregationStateType instead of a serializer. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6145) Add [non-partitioned] event time OVER ROWS BETWEEN x PRECEDING aggregation to SQL
sunjincheng created FLINK-6145: -- Summary: Add [non-partitioned] event time OVER ROWS BETWEEN x PRECEDING aggregation to SQL Key: FLINK-6145 URL: https://issues.apache.org/jira/browse/FLINK-6145 Project: Flink Issue Type: Sub-task Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng The goal of this issue is to add support for OVER ROWS aggregations on event time streams to the SQL interface. Queries similar to the following should be supported: {code} SELECT a, SUM(b) OVER (ORDER BY rowTime() ROWS BETWEEN 2 PRECEDING AND CURRENT ROW) AS sumB, MIN(b) OVER (ORDER BY rowTime() ROWS BETWEEN 2 PRECEDING AND CURRENT ROW) AS minB FROM myStream {code} The following restrictions should initially apply: - All OVER clauses in the same SELECT clause must be exactly the same. - The ORDER BY clause may only have rowTime() as parameter. rowTime() is a parameterless scalar function that just indicates event time mode. - UNBOUNDED PRECEDING is not supported (see FLINK-5804) - FOLLOWING is not supported. The restrictions will be resolved in follow up issues. If we find that some of the restrictions are trivial to address, we can add the functionality in this issue as well. This issue includes: - Design of the DataStream operator to compute OVER ROW aggregates - Translation from Calcite's RelNode representation (LogicalProject with RexOver expression). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6200) Add event time OVER RANGE BETWEEN UNBOUNDED PRECEDING aggregation to SQL
sunjincheng created FLINK-6200: -- Summary: Add event time OVER RANGE BETWEEN UNBOUNDED PRECEDING aggregation to SQL Key: FLINK-6200 URL: https://issues.apache.org/jira/browse/FLINK-6200 Project: Flink Issue Type: New Feature Components: Table API & SQL Reporter: sunjincheng The goal of this issue is to add support for OVER RANGE aggregations on event time streams to the SQL interface. Queries similar to the following should be supported: SELECT a, SUM(b) OVER (PARTITION BY c ORDER BY rowTime() RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS sumB, MIN(b) OVER (PARTITION BY c ORDER BY rowTime() RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS minB FROM myStream The following restrictions should initially apply: All OVER clauses in the same SELECT clause must be exactly the same. The PARTITION BY clause is optional (no partitioning results in single threaded execution). The ORDER BY clause may only have rowTime() as parameter. rowTime() is a parameterless scalar function that just indicates processing time mode. bounded PRECEDING is not supported (see FLINK-5655) FOLLOWING is not supported. The restrictions will be resolved in follow up issues. If we find that some of the restrictions are trivial to address, we can add the functionality in this issue as well. This issue includes: Design of the DataStream operator to compute OVER ROW aggregates Translation from Calcite's RelNode representation (LogicalProject with RexOver expression). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6204) Improve Event-Time OVER ROWS BETWEEN UNBOUNDED PRECEDING aggregation to SQL
sunjincheng created FLINK-6204: -- Summary: Improve Event-Time OVER ROWS BETWEEN UNBOUNDED PRECEDING aggregation to SQL Key: FLINK-6204 URL: https://issues.apache.org/jira/browse/FLINK-6204 Project: Flink Issue Type: Bug Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng Currently `event time OVER ROWS BETWEEN UNBOUNDED PRECEDING aggregation to SQL` implementation class: ` UnboundedEventTimeOverProcessFunction` use data size uncontrollable memory data structures`sortedTimestamps: util.LinkedList [Long] cache data timestamps and sort timestamps. IMO,It's not a good way, because in the production environment there are millions of window data pre millisecond in our application scenario.So, I want to improve it. Welcome anyone to give me feedback. What do you think? [~fhueske] and [~Yuhong_kyo] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6219) Add a state backend which supports sorting
sunjincheng created FLINK-6219: -- Summary: Add a state backend which supports sorting Key: FLINK-6219 URL: https://issues.apache.org/jira/browse/FLINK-6219 Project: Flink Issue Type: New Feature Components: State Backends, Checkpointing, Table API & SQL Reporter: sunjincheng When we implement the OVER window of [FLIP11|https://cwiki.apache.org/confluence/display/FLINK/FLIP-11%3A+Table+API+Stream+Aggregations] We notice that we need a state backend which supports sorting, allows for efficient insertion, traversal in order, and removal from the head. For example: In event-time OVER window, we need to sort by time,If the datas as follow: {code} (1L, 1, Hello) (2L, 2, Hello) (5L, 5, Hello) (4L, 4, Hello) {code} We randomly insert the datas, just like: {code} put((2L, 2, Hello)),put((1L, 1, Hello)),put((5L, 5, Hello)),put((4L, 4, Hello)), {code} We deal with elements in time order: {code} process((1L, 1, Hello)),process((2L, 2, Hello)),process((4L, 4, Hello)),process((5L, 5, Hello)) {code} Welcome anyone to give feedback,And what do you think? [~xiaogang.shi] [~aljoscha] [~fhueske] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6228) Integrating the OVER windows in the Table API
sunjincheng created FLINK-6228: -- Summary: Integrating the OVER windows in the Table API Key: FLINK-6228 URL: https://issues.apache.org/jira/browse/FLINK-6228 Project: Flink Issue Type: Sub-task Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng Syntax: {code} table .overWindows( (Rows|Range [ partitionBy value_expression , ... [ n ]] (preceding UNBOUNDED|value_specification.(rows|milli|second|minute|hour|day|month|year)|CURRENTROW) [following UNBOUNDED|value_specification.(rows|milli|second|minute|hour|day|month|year)|CURRENTROW] as alias,...[n]) ) .select( [col1,...[n]], (agg(col1) OVER overWindowAlias, … [n]) {code} Implement restrictions: ** All OVER clauses in the same SELECT clause must be exactly the same. ** The PARTITION BY clause is optional (no partitioning results in single threaded execution). ** The ORDER BY clause is hidden in tableApi, According to time characteristic automatic identification. ** FOLLOWING is not supported. I will soon add a user interface design document. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6259) Fix a small spelling error
sunjincheng created FLINK-6259: -- Summary: Fix a small spelling error Key: FLINK-6259 URL: https://issues.apache.org/jira/browse/FLINK-6259 Project: Flink Issue Type: Bug Components: Gelly Reporter: sunjincheng Assignee: sunjincheng flink-gelly-scala/pom.xml {{har-with-dependencies}} -> {{ jar-with-dependencies}} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6266) Remove the useless import in FlinkRelBuilder
sunjincheng created FLINK-6266: -- Summary: Remove the useless import in FlinkRelBuilder Key: FLINK-6266 URL: https://issues.apache.org/jira/browse/FLINK-6266 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng Remove FLINK-6037 legacy useless import. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6267) Remove the useless import in FlinkRelBuilder
sunjincheng created FLINK-6267: -- Summary: Remove the useless import in FlinkRelBuilder Key: FLINK-6267 URL: https://issues.apache.org/jira/browse/FLINK-6267 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng Remove FLINK-6037 legacy useless import. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6304) Clear a lot of useless import
sunjincheng created FLINK-6304: -- Summary: Clear a lot of useless import Key: FLINK-6304 URL: https://issues.apache.org/jira/browse/FLINK-6304 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng There are some classes(as follow) have useless import, I want clear them before release 1.3. {code} DataSetSlideTimeWindowAggFlatMapFunction CommonScan FlinkRel StreamTableSourceScanRule DataStreamOverAggregateRule DataStreamAggregateRule {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6392) Change the alias of Window from optional to essential.
sunjincheng created FLINK-6392: -- Summary: Change the alias of Window from optional to essential. Key: FLINK-6392 URL: https://issues.apache.org/jira/browse/FLINK-6392 Project: Flink Issue Type: Improvement Components: Table API & SQL Affects Versions: 1.3.0 Reporter: sunjincheng Assignee: sunjincheng Fix For: 1.3.0 Currently, The window clause use case looks like: {code} tab //Table('a,'b,'c) .window( Slide over 10.milli every 5.milli as 'w) .groupBy('w,'a,'b) // WindowGroupedTable .select('a, 'b, 'c.sum, 'w.start, 'w.end) {code} As we see the alias of window is essential. But the current implementation of the TableAPI does not have the constraint for the alias,So we must refactoring the API definition using TYPE SYSTEM lead to constraint for the alias. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6406) Cleanup useless import
sunjincheng created FLINK-6406: -- Summary: Cleanup useless import Key: FLINK-6406 URL: https://issues.apache.org/jira/browse/FLINK-6406 Project: Flink Issue Type: Improvement Components: Table API & SQL Affects Versions: 1.3.0 Reporter: sunjincheng Assignee: sunjincheng When browsing the code, it is found that there are some useless reference in the following file which need cleanup. *packages.scala *ExternalCatalogTable *arithmetic.scala *array.scala *ColumnStats -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6426) Correct the method parameter type error in the group-window table API document
sunjincheng created FLINK-6426: -- Summary: Correct the method parameter type error in the group-window table API document Key: FLINK-6426 URL: https://issues.apache.org/jira/browse/FLINK-6426 Project: Flink Issue Type: Sub-task Components: Documentation, Table API & SQL Affects Versions: 1.3.0 Reporter: sunjincheng Assignee: sunjincheng Correct the document from ` .window([w: Window] as 'w)` to ` .window([w: WindowWithoutAlias] as 'w)` -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6428) Add supported DISTINCT in dataStream SQL
sunjincheng created FLINK-6428: -- Summary: Add supported DISTINCT in dataStream SQL Key: FLINK-6428 URL: https://issues.apache.org/jira/browse/FLINK-6428 Project: Flink Issue Type: New Feature Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng Add supported DISTINCT in dataStream SQL as follow: DATA: {code} (name, age) (kevin, 28), (sunny, 6), (jack, 6) {code} SQL: {code} SELECT DISTINCT age FROM MyTable" {code} RESULTS: {code} 28, 6 {code} [~fhueske] do you we need this feature? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6462) Add requires over indicators for UDAF
sunjincheng created FLINK-6462: -- Summary: Add requires over indicators for UDAF Key: FLINK-6462 URL: https://issues.apache.org/jira/browse/FLINK-6462 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng The window function in the standard database only supports the "over" window, such as `LAG`,` LEAD`, `FIRST_VALUE`,` LAST_VALUE`. These window functions do not apply to `Slide`,` Tumble`, `Session` window. So when the user-defined AGG need to be clearly defined. In calcite `SqlAggFunction` will using `requiresOver` to distinguish. In this JIRA. will deal with this feature. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6479) Hot fix FLINK-5906 bug
sunjincheng created FLINK-6479: -- Summary: Hot fix FLINK-5906 bug Key: FLINK-6479 URL: https://issues.apache.org/jira/browse/FLINK-6479 Project: Flink Issue Type: Bug Components: Table API & SQL Affects Versions: 1.3.0 Reporter: sunjincheng Assignee: sunjincheng {code} val windowedTable = table .window(Slide over 2.rows every 1.rows on 'proctime as 'w) .groupBy('w, 'string) .select(weightAvgFun('long, 'int)) {code} will got exception: {code} java.lang.IndexOutOfBoundsException: index (2) must be less than size (2) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:310) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:292) at com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:65) at org.apache.calcite.util.Pair$6.get(Pair.java:347) at scala.collection.convert.Wrappers$JListWrapper.apply(Wrappers.scala:88) at org.apache.flink.table.plan.nodes.CommonAggregate$$anonfun$groupingToString$1.apply(CommonAggregate.scala:34) at org.apache.flink.table.plan.nodes.CommonAggregate$$anonfun$groupingToString$1.apply(CommonAggregate.scala:34) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofInt.foreach(ArrayOps.scala:156) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.mutable.ArrayOps$ofInt.map(ArrayOps.scala:156) at org.apache.flink.table.plan.nodes.CommonAggregate$class.groupingToString(CommonAggregate.scala:34) at org.apache.flink.table.plan.nodes.datastream.DataStreamGroupWindowAggregate.groupingToString(DataStreamGroupWindowAggregate.scala:44) {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6583) Enable QueryConfig in count base GroupWindow
sunjincheng created FLINK-6583: -- Summary: Enable QueryConfig in count base GroupWindow Key: FLINK-6583 URL: https://issues.apache.org/jira/browse/FLINK-6583 Project: Flink Issue Type: New Feature Components: Table API & SQL Affects Versions: 1.3.0, 1.4.0 Reporter: sunjincheng Assignee: sunjincheng Enable QueryConfig in count base GroupWindow by Add a custom Trigger `CountTriggerWithCleanupState`. See more in FLINK-6491. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6597) Cleanup WindowOperatorTest useless imports
sunjincheng created FLINK-6597: -- Summary: Cleanup WindowOperatorTest useless imports Key: FLINK-6597 URL: https://issues.apache.org/jira/browse/FLINK-6597 Project: Flink Issue Type: Test Reporter: sunjincheng Assignee: sunjincheng Fix For: 1.3.0 Don't miss doing any good thing no matter how insignificant it looks. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6598) Fix `DataStreamGroupAggregateRule` matches error.
sunjincheng created FLINK-6598: -- Summary: Fix `DataStreamGroupAggregateRule` matches error. Key: FLINK-6598 URL: https://issues.apache.org/jira/browse/FLINK-6598 Project: Flink Issue Type: Bug Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng Fix For: 1.3.0 Change retrun from `!distinctAggs && !groupSets && !agg.indicator` to `!distinctAggs && !groupSets && !agg.indicator && !agg.getAggCallList.isEmpty`. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6617) Improve JAVA and SCALA logical plans consistent test
sunjincheng created FLINK-6617: -- Summary: Improve JAVA and SCALA logical plans consistent test Key: FLINK-6617 URL: https://issues.apache.org/jira/browse/FLINK-6617 Project: Flink Issue Type: Test Components: Table API & SQL Affects Versions: 1.3.0 Reporter: sunjincheng Assignee: sunjincheng Currently,we need some `StringExpression` test,for all JAVA and SCALA API. Such as:`GroupAggregations`,`GroupWindowAggregaton`(Session,Tumble),`Calc` etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6618) Fix `GroupWindow` JAVA logical plans not consistent with SCALA logical plans.
sunjincheng created FLINK-6618: -- Summary: Fix `GroupWindow` JAVA logical plans not consistent with SCALA logical plans. Key: FLINK-6618 URL: https://issues.apache.org/jira/browse/FLINK-6618 Project: Flink Issue Type: Bug Components: Table API & SQL Affects Versions: 1.3.0 Reporter: sunjincheng Assignee: sunjincheng I find 2 bugs as follows: 1. `GroupWindowStringExpressionTest` testcase bug, `Assert.assertEquals("Logical Plans do not match", resJava.logicalPlan, resJava.logicalPlan)` -> `Assert.assertEquals("Logical Plans do not match", resJava.logicalPlan, resScala.logicalPlan)` 2. When i fix the bug above, we got anther bug: {code} java.lang.AssertionError: Logical Plans do not match Expected :Project(ListBuffer('string, 'TMP_4, 'TMP_5, 'TMP_6, ('TMP_7 * 2) as '_c4),WindowAggregate(List('string),SlidingGroupWindow('w, 'rowtime, 1440.millis, 720.millis),List(),List(CountAggFunction(List('string)) as 'TMP_4, sum('int) as 'TMP_5, WeightedAvg(List('long, 'int)) as 'TMP_6, WeightedAvg(List('int, 'int)) as 'TMP_7),Project(ArrayBuffer('string, 'int, 'long, 'rowtime),CatalogNode(WrappedArray(_DataStreamTable_0),RecordType(INTEGER int, BIGINT long, VARCHAR(2147483647) string, TIMESTAMP(3) rowtime) Actual :Project(ListBuffer('string, 'TMP_0, 'TMP_1, 'TMP_2, ('TMP_3 * 2) as '_c4),WindowAggregate(ArrayBuffer('string),SlidingGroupWindow('w, 'rowtime, 1440.millis, 720.millis),List(),List(CountAggFunction(ArrayBuffer('string)) as 'TMP_0, sum('int) as 'TMP_1, WeightedAvg(ArrayBuffer('long, 'int)) as 'TMP_2, WeightedAvg(ArrayBuffer('int, 'int)) as 'TMP_3),Project(ArrayBuffer('string, 'int, 'long, 'rowtime),CatalogNode(WrappedArray(_DataStreamTable_0),RecordType(INTEGER int, BIGINT long, VARCHAR(2147483647) string, TIMESTAMP(3) rowtime) {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6619) Check Table API & SQL support for 1.3.0 RC01 Release
sunjincheng created FLINK-6619: -- Summary: Check Table API & SQL support for 1.3.0 RC01 Release Key: FLINK-6619 URL: https://issues.apache.org/jira/browse/FLINK-6619 Project: Flink Issue Type: Test Components: Table API & SQL Affects Versions: 1.3.0 Reporter: sunjincheng Assignee: sunjincheng In this JIRA. I will do the following tasks for Flink 1.3.0 RC01 Release. * Check that the JAVA and SCALA logical plans are consistent. * Check that the SQL and Table API logical plans are consistent. * Check that UDF, UDTF, and UDAF are working properly in group-windows and over-windows. * Check that all built-in Agg on Batch and Stream are working properly. When I do the task above, I'll created some sub-task. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6632) Fix parameter case sensitive error for test passing/rejecting filter API
sunjincheng created FLINK-6632: -- Summary: Fix parameter case sensitive error for test passing/rejecting filter API Key: FLINK-6632 URL: https://issues.apache.org/jira/browse/FLINK-6632 Project: Flink Issue Type: Sub-task Reporter: sunjincheng Assignee: sunjincheng TableAPI {{testAllPassingFilter}}: {code} val t = util.addTable[(Int, Long, String)]('int, 'long, 'string) val resScala = t.filter(Literal(true)).select('int as 'myInt, 'string) val resJava = t.filter("TrUe").select("int as myInt, string") {code} We got error: {code} org.apache.flink.table.api.ValidationException: Cannot resolve [TrUe] given input [int, long, string]. {code} The error is caused by : {code} lazy val boolLiteral: PackratParser[Expression] = ("true" | "false") ^^ { str => Literal(str.toBoolean) } {code} I want import the method as follow: {code} lazy val boolLiteral: PackratParser[Expression] = ("(t|T)(r|R)(u|U)(e|E)".r | "(f|F)(a|A)(l|L)(s|S)(e|E)".r) ^^ { str => Literal(str.toBoolean)} {code} Is there any drawback to this improvement? Welcome anyone feedback ? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6649) Improve Non-window group aggregate with configurable `earlyFire`.
sunjincheng created FLINK-6649: -- Summary: Improve Non-window group aggregate with configurable `earlyFire`. Key: FLINK-6649 URL: https://issues.apache.org/jira/browse/FLINK-6649 Project: Flink Issue Type: Improvement Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng Assignee: sunjincheng Currently, Non-windowed group aggregate is earlyFiring at count(1), that is every row will emit a aggregate result. But some times user want config count number (`early firing with count[N]`) , to reduce the downstream pressure. This JIRA. will enable the config of e`arlyFiring` for Non-windowed group aggregate. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6650) Fix Non-windowed group-aggregate error when using append-table mode.
sunjincheng created FLINK-6650: -- Summary: Fix Non-windowed group-aggregate error when using append-table mode. Key: FLINK-6650 URL: https://issues.apache.org/jira/browse/FLINK-6650 Project: Flink Issue Type: Sub-task Reporter: sunjincheng Assignee: sunjincheng When I test Non-windowed group-aggregate with {{stream.toTable(tEnv, 'a, 'b, 'c).select('a.sum, weightAvgFun('a, 'b)).toAppendStream[Row].addSink(new StreamITCase.StringSink)}}, I got the error as follows: {code} org.apache.flink.table.api.TableException: Table is not an append-only table. Output needs to handle update and delete changes. at org.apache.flink.table.api.StreamTableEnvironment.translate(StreamTableEnvironment.scala:631) at org.apache.flink.table.api.StreamTableEnvironment.translate(StreamTableEnvironment.scala:607) at org.apache.flink.table.api.scala.StreamTableEnvironment.toAppendStream(StreamTableEnvironment.scala:219) at org.apache.flink.table.api.scala.StreamTableEnvironment.toAppendStream(StreamTableEnvironment.scala:195) at org.apache.flink.table.api.scala.TableConversions.toAppendStream(TableConversions.scala:121) {code} The reason is {{DataStreamGroupAggregate#producesUpdates}} as follows: {code} override def producesUpdates = true {code} I think in the view of the user, what user want are(for example): Data: {code} val data = List( (1L, 1, "Hello"), (2L, 2, "Hello"), (3L, 3, "Hello"), (4L, 4, "Hello"), (5L, 5, "Hello"), (6L, 6, "Hello"), (7L, 7, "Hello World"), (8L, 8, "Hello World"), (20L, 20, "Hello World")) {code} *Case1: TableAPI {code} stream.toTable(tEnv, 'a, 'b, 'c).select('a.sum).toRetractStream[Row] .addSink(new StreamITCase.RetractingSink) {code} Result {code} 1 3 6 10 15 21 28 36 56 {code} * Case 2: TableAPI {code} stream.toTable(tEnv, 'a, 'b, 'c).select('a.sum).toRetractStream[Row] .addSink(new StreamITCase.RetractingSink) {code} Result: {code} 56 {code} In fact about #Case 1,we can using unbounded OVER windows, as follows: TableAPI {code} stream.toTable(tEnv, 'a, 'b, 'c, 'proctime.proctime) .window(Over orderBy 'proctime preceding UNBOUNDED_ROW as 'w) .select('a.sum over 'w) .toAppendStream[Row].addSink(new StreamITCase.StringSink) {code} Result {code} Same as #Case1 {code} But after the [FLINK-6649 | https://issues.apache.org/jira/browse/FLINK-6649] OVER can not express the #Case1 with earlyFiring. So I still think Non-windowed group-aggregate not always update-table, user can decide which mode to use. Is there any drawback to this improvement? Welcome anyone feedback? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6686) Improve UDXF(UDF,UDTF,UDAF) test case
sunjincheng created FLINK-6686: -- Summary: Improve UDXF(UDF,UDTF,UDAF) test case Key: FLINK-6686 URL: https://issues.apache.org/jira/browse/FLINK-6686 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.3.0 Reporter: sunjincheng Assignee: sunjincheng 1. Add Check that UDF, UDTF, and UDAF are working properly in group-windows and over-windows. 2. Add Check that all built-in Agg on Batch and Stream are working properly. Let types such as Timestamp, BigDecimal or Pojo flow through UDF. UDTF, UDAF (input and output types) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6736) Fix UDTF codegen bug when window follow by join( UDTF)
sunjincheng created FLINK-6736: -- Summary: Fix UDTF codegen bug when window follow by join( UDTF) Key: FLINK-6736 URL: https://issues.apache.org/jira/browse/FLINK-6736 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.3.0 Reporter: sunjincheng Assignee: sunjincheng When we run the tableAPI as follows: {code} val table = stream.toTable(tEnv, 'long.rowtime, 'int, 'double, 'float, 'bigdec, 'date,'pojo, 'string) val windowedTable = table .join(udtf2('string) as ('a, 'b)) .window(Slide over 5.milli every 2.milli on 'long as 'w) .groupBy('w) .select('int.count, agg1('pojo, 'bigdec, 'date, 'int), 'w.start, 'w.end) {code} We will get the error message: {code} org.apache.flink.runtime.client.JobExecutionException: Job execution failed. at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$mcV$sp(JobManager.scala:933) at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:876) at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:876) at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Caused by: org.apache.flink.api.common.InvalidProgramException: Table program cannot be compiled. This is a bug. Please file an issue. at org.apache.flink.table.codegen.Compiler$class.compile(Compiler.scala:36) at org.apache.flink.table.runtime.CRowCorrelateProcessRunner.compile(CRowCorrelateProcessRunner.scala:35) at org.apache.flink.table.runtime.CRowCorrelateProcessRunner.open(CRowCorrelateProcessRunner.scala:59) at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36) at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:111) at org.apache.flink.streaming.api.operators.ProcessOperator.open(ProcessOperator.java:56) at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:377) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:254) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702) at java.lang.Thread.run(Thread.java:745) Caused by: org.codehaus.commons.compiler.CompileException: Line 77, Column 62: Unknown variable or type "in2" at org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:11523) at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:6292) at org.codehaus.janino.UnitCompiler.access$12900(UnitCompiler.java:209) at org.codehaus.janino.UnitCompiler$18.visitPackage(UnitCompiler.java:5904) at org.codehaus.janino.UnitCompiler$18.visitPackage(UnitCompiler.java:5901) at org.codehaus.janino.Java$Package.accept(Java.java:4074) at org.codehaus.janino.UnitCompiler.getType(UnitCompiler.java:5901) at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:6287) at org.codehaus.janino.UnitCompiler.access$13500(UnitCompiler.java:209) {code} The reason is {{val generator = new CodeGenerator(config, false, inputSchema.physicalTypeInfo)}} `physicalTypeInfo` will remove the TimeIndicator. I think we should fix this. What do you think [~fhueske] [~twalthr] , And hope your suggestions. :) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6737) Fix over expression parse String error.
sunjincheng created FLINK-6737: -- Summary: Fix over expression parse String error. Key: FLINK-6737 URL: https://issues.apache.org/jira/browse/FLINK-6737 Project: Flink Issue Type: Sub-task Reporter: sunjincheng Assignee: sunjincheng When we run the TableAPI as follows: {code} val windowedTable = table .window(Over partitionBy 'c orderBy 'proctime preceding UNBOUNDED_ROW as 'w) .select('c, "countFun(b)" over 'w as 'mycount, weightAvgFun('a, 'b) over 'w as 'wAvg) {code} We get the error: {code} org.apache.flink.table.api.TableException: The over method can only using with aggregation expression. at org.apache.flink.table.api.scala.ImplicitExpressionOperations$class.over(expressionDsl.scala:469) at org.apache.flink.table.api.scala.ImplicitExpressionConversions$LiteralStringExpression.over(expressionDsl.scala:756) {code} The reason is, the `over` method of `expressionDsl` not parse the String case. I think we should fix this before 1.3 release. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6739) Fix all string reference variable error.
sunjincheng created FLINK-6739: -- Summary: Fix all string reference variable error. Key: FLINK-6739 URL: https://issues.apache.org/jira/browse/FLINK-6739 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.3.0 Reporter: sunjincheng Assignee: sunjincheng I find a lot of string reference variable error such as {{in '${methodName}' methods }}. We should change it from {{in '${methodName}' methods }} to {{in \'${methodName}\' methods }} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6740) Fix "parameterTypeEquals" method error.
sunjincheng created FLINK-6740: -- Summary: Fix "parameterTypeEquals" method error. Key: FLINK-6740 URL: https://issues.apache.org/jira/browse/FLINK-6740 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.3.0 Reporter: sunjincheng Assignee: sunjincheng When we define UDTF as follows: {code} class TableFuncPojo extends TableFunction[TPojo] { def eval(age: Int, name:String): Unit = { collect(new TPojo(age.toLong,name)) } def eval(age: Date, name:String): Unit = { collect(new TPojo(age.getTime,name)) } } {code} TableAPI: {code} val table = stream.toTable(tEnv, 'long2, 'int, 'double, 'float, 'bigdec, 'ts, 'date,'pojo, 'string, 'long.rowtime) val windowedTable = table .join(udtf('date, 'string) as 'pojo2).select('pojo2) {code} We will get the error as following: {code} org.apache.flink.table.api.ValidationException: Found multiple 'eval' methods which match the signature. at org.apache.flink.table.functions.utils.UserDefinedFunctionUtils$.getUserDefinedMethod(UserDefinedFunctionUtils.scala:180) at org.apache.flink.table.plan.logical.LogicalTableFunctionCall.validate(operators.scala:700) at org.apache.flink.table.api.Table.join(table.scala:539) at org.apache.flink.table.api.Table.join(table.scala:328) at org.apache.flink.table.runtime.datastream.DataStreamAggregateITCase.test1(DataStreamAggregateITCase.scala:84) {code} The reason is in {{ parameterTypeEquals }} method, logical as follows: {code} candidate == classOf[Date] && (expected == classOf[Int] || expected == classOf[JInt]) {code} But when we only define one `eval` method, we got different result, as follows: {code} // for def eval(age: Int, name:String) Pojo{id=0, name='Hello'} Pojo{id=1, name='Hallo'} Pojo{id=1, name='Hello world'} Pojo{id=1, name='Hello world'} Pojo{id=1, name='Hello'} Pojo{id=1, name='Hi'} Pojo{id=8, name='Hello'} // for def eval(age: Date, name:String) Pojo{id=-2880, name='Hello'} Pojo{id=5760, name='Hallo'} Pojo{id=5760, name='Hello world'} Pojo{id=5760, name='Hello world'} Pojo{id=5760, name='Hello'} Pojo{id=5760, name='Hi'} Pojo{id=66240, name='Hello'} {code} So, We should modify the logical of {{ parameterTypeEquals }} method. What do you think? Welcome anybody feedback... -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6760) Fix OverWindowTest alias test error
sunjincheng created FLINK-6760: -- Summary: Fix OverWindowTest alias test error Key: FLINK-6760 URL: https://issues.apache.org/jira/browse/FLINK-6760 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.3.0 Reporter: sunjincheng Assignee: sunjincheng For Sql: {code} val sql = "SELECT c, count(a) OVER (ORDER BY proctime ROWS BETWEEN 2 preceding AND CURRENT ROW) as cnt1 CURRENT ROW from MyTable" {code} The alias `cnt1` The alias did not take effect when we generated the plan string. But we can using the alias in outer layer query, for example: {code} val sql = "SELECT cnt1 from (SELECT c, count(a) OVER (ORDER BY proctime ROWS BETWEEN 2 preceding AND CURRENT ROW) as cnt1 CURRENT ROW from MyTable)" {code} So in this JIRA. we just fix the test case for 1.3 release. In another JIRA. will improve the alias. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6810) Add Some build-in Scalar Function
sunjincheng created FLINK-6810: -- Summary: Add Some build-in Scalar Function Key: FLINK-6810 URL: https://issues.apache.org/jira/browse/FLINK-6810 Project: Flink Issue Type: New Feature Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng Assignee: sunjincheng In this JIRA, will create some sub-task for add specific scalar function, such as mathematical-function {{LOG}}, date-functions {{ DATEADD}},string-functions {{LPAD}}, etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6811) Add DATEADD/DATESUB/DATEDIFF as build-in scalar function
sunjincheng created FLINK-6811: -- Summary: Add DATEADD/DATESUB/DATEDIFF as build-in scalar function Key: FLINK-6811 URL: https://issues.apache.org/jira/browse/FLINK-6811 Project: Flink Issue Type: Sub-task Reporter: sunjincheng -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6813) Add DATEDIFF as build-in scalar function
sunjincheng created FLINK-6813: -- Summary: Add DATEDIFF as build-in scalar function Key: FLINK-6813 URL: https://issues.apache.org/jira/browse/FLINK-6813 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng Assignee: sunjincheng * Syntax DATEDIFF ( datepart , startdate , enddate ) -datepart Is the part of startdate and enddate that specifies the type of boundary crossed. -startdate Is an expression that can be resolved to a time, date. -enddate Same with startdate. * Example SELECT DATEDIFF(year, '2015-12-31 23:59:59.999', '2017-01-01 00:00:00.000') from tab; --> 2 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6837) Fix a small error message bug, And improve some message info.
sunjincheng created FLINK-6837: -- Summary: Fix a small error message bug, And improve some message info. Key: FLINK-6837 URL: https://issues.apache.org/jira/browse/FLINK-6837 Project: Flink Issue Type: Bug Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng Assignee: sunjincheng Fix a variable reference error, and improve some error message info. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6839) Improve SQL OVER alias When only one OVER window agg in selection.
sunjincheng created FLINK-6839: -- Summary: Improve SQL OVER alias When only one OVER window agg in selection. Key: FLINK-6839 URL: https://issues.apache.org/jira/browse/FLINK-6839 Project: Flink Issue Type: Improvement Reporter: sunjincheng For OVER SQL: {code} SELECT a COUNT(c) OVER (ORDER BY proctime RANGE BETWEEN INTERVAL '10' SECOND PRECEDING AND CURRENT ROW) as cnt1 FROM MyTable {code} We expect plan {{DataStreamCalc(select=[a, w0$o0 AS cnt1]) But we get {{DataStreamCalc(select=[a, w0$o0 AS $1]) }}. this improve only for plan check. the functional is work well in nested queries,e.g.: {code} SELECT cnt1 from (SELECT a COUNT(c) OVER (ORDER BY proctime RANGE BETWEEN INTERVAL '10' SECOND PRECEDING AND CURRENT ROW) as cnt1 FROM MyTable) {code} The SQL above is work well. which mentioned in [FLINK-6760|https://issues.apache.org/jira/browse/FLINK-6760]. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6841) using TableSourceTable for both Stream and Batch OR remove useless import
sunjincheng created FLINK-6841: -- Summary: using TableSourceTable for both Stream and Batch OR remove useless import Key: FLINK-6841 URL: https://issues.apache.org/jira/browse/FLINK-6841 Project: Flink Issue Type: Improvement Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng Assignee: sunjincheng 1. {{StreamTableSourceTable}} exist useless import of {{TableException}} 2. {{StreamTableSourceTable}} only override {{getRowType}} of {{FlinkTable}}, I think we can override the method in {{TableSourceTable}}, If so we can using {{TableSourceTable}} for both {{Stream}} and {{Batch}}. What do you think? [~fhueske] [~twalthr] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6845) Cleanup "println(StreamITCase.testResults)" call in test case
sunjincheng created FLINK-6845: -- Summary: Cleanup "println(StreamITCase.testResults)" call in test case Key: FLINK-6845 URL: https://issues.apache.org/jira/browse/FLINK-6845 Project: Flink Issue Type: Test Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng Assignee: sunjincheng Cleanup "println(StreamITCase.testResults)" call in test case. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6846) Add TESTTIMESTAMPADD supported in TableAPI
sunjincheng created FLINK-6846: -- Summary: Add TESTTIMESTAMPADD supported in TableAPI Key: FLINK-6846 URL: https://issues.apache.org/jira/browse/FLINK-6846 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng Assignee: sunjincheng See FLINK-6811 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6847) Add TIMESTAMPDIFF supported in TableAPI
sunjincheng created FLINK-6847: -- Summary: Add TIMESTAMPDIFF supported in TableAPI Key: FLINK-6847 URL: https://issues.apache.org/jira/browse/FLINK-6847 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng Assignee: sunjincheng see FLINK-6813 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6851) Improve built-in scalar function when upgrade calcite version to 1.13
sunjincheng created FLINK-6851: -- Summary: Improve built-in scalar function when upgrade calcite version to 1.13 Key: FLINK-6851 URL: https://issues.apache.org/jira/browse/FLINK-6851 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng I found that the difference between calcite 1.12 and calcite master affects the semantics of the build-in scalar function, for example: The SQL: {code} select timestampadd(MONTH, 1, date '2016-05-31'), timestampadd(MONTH, 5, date '2016-01-31'), timestampadd(MONTH, -1, date '2016-03-31') from depts; {code} Calcite 1.12 Result: {code} ++++ | EXPR$0 | EXPR$1 | EXPR$2 | ++++ | 2016-07-01 | 2016-07-01 | 2016-03-01 | ++++ 3 rows selected (1.06 seconds) {code} Calcite Master result: {code} ++++ | EXPR$0 | EXPR$1 | EXPR$2 | ++++ | 2016-06-30 | 2016-06-30 | 2016-02-29 | ++++ 3 rows selected (1.047 seconds) {code} The MSSQL result: {code} select dateadd(MONTH, 1, '2016-05-31'), dateadd(MONTH, 5, '2016-01-31'), dateadd(MONTH, -1,'2016-03-31') from pvt; 2016-06-30 00:00:00 2016-06-30 00:00:00 2016-02-29 00:00:00 {code} So, when we flink upgrade the calcite version to 1.13. we can open some useful feature. e.g: {{TIMESTAMPADD(SqlTypeFamily.ANY, SqlTypeFamily.INTEGER, SqlTypeFamily.DATE)}}. So, In this JIRA. will create some subtask to improve flink according to calcite 1.13. I appreciated if you can give me some feedback? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6886) Fix Timestamp field can not be selected in event time case when toDataStream[T], `T` not a `Row` Type.
sunjincheng created FLINK-6886: -- Summary: Fix Timestamp field can not be selected in event time case when toDataStream[T], `T` not a `Row` Type. Key: FLINK-6886 URL: https://issues.apache.org/jira/browse/FLINK-6886 Project: Flink Issue Type: Bug Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng Assignee: sunjincheng Currently for event-time window(group/over), When contain `Timestamp` type field in `SELECT Clause`, And toDataStream[T], `T` not a `Row` Type, Such `PojoType`, will throw a exception. In this JIRA. will fix this bug. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6891) Add LOG supported in SQL
sunjincheng created FLINK-6891: -- Summary: Add LOG supported in SQL Key: FLINK-6891 URL: https://issues.apache.org/jira/browse/FLINK-6891 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng LONG (N) A single parameter version of the function returns the natural logarithm of N, and if two arguments are called, it returns any radix of the logarithm of N. * Syntax: LOG ( float_expression [, base ] ) * Arguments **float_expression: Is an expression of type float or of a type that can be implicitly converted to float. **base: Optional integer argument that sets the base for the logarithm. * Return Types float * Example: LOG(10) -> 2.30 *See more: **[MSQL|https://docs.microsoft.com/en-us/sql/t-sql/functions/log-transact-sql] **[MySQL| https://dev.mysql.com/doc/refman/5.7/en/mathematical-functions.html#function_log] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6892) Add LPAD supported in SQL
sunjincheng created FLINK-6892: -- Summary: Add LPAD supported in SQL Key: FLINK-6892 URL: https://issues.apache.org/jira/browse/FLINK-6892 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng LPAD(str,len,padstr) Returns the string str, left-padded with the string padstr to a length of len characters. If str is longer than len, the return value is shortened to len characters. * Syntax: LPAD(str,len,padstr) * Arguments **str: - **len: - **padstr: - * Return Types String * Example: LPAD('hi',4,'??') -> '??hi' LPAD('hi',1,'??') -> 'h' * See more: ** [MySQL| https://dev.mysql.com/doc/refman/5.7/en/mathematical-functions.html#function_lpad] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6893) Add BIN supported in SQL
sunjincheng created FLINK-6893: -- Summary: Add BIN supported in SQL Key: FLINK-6893 URL: https://issues.apache.org/jira/browse/FLINK-6893 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng BIN(N) Returns a string representation of the binary value of N, where N is a longlong (BIGINT) number. This is equivalent to CONV(N,10,2). Returns NULL if N is NULL. * Syntax: BIN(num) * Arguments **num: a long/bigint value * Return Types String * Example: BIN(12) -> '1100' * See more: ** [MySQL| https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_bin] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6894) Add DATE_FORMAT supported in SQL
sunjincheng created FLINK-6894: -- Summary: Add DATE_FORMAT supported in SQL Key: FLINK-6894 URL: https://issues.apache.org/jira/browse/FLINK-6894 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng DATE_FORMAT(date,format) Formats the date value according to the format string. * Syntax: DATE_FORMAT(date,format) * Arguments **date: - **format: - * Return Types String * Example: DATE_FORMAT('2009-10-04 22:23:00', '%W %M %Y') -> 'Sunday October 2009' DATE_FORMAT('1900-10-04 22:23:00','%D %y %a %d %m %b %j') -> '4th 00 Thu 04 10 Oct 277' * See more: ** [MySQL| https://dev.mysql.com/doc/refman/5.7/en/date-and-time-functions.html#function_date-format] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6895) Add STR_TO_DATE supported in SQL
sunjincheng created FLINK-6895: -- Summary: Add STR_TO_DATE supported in SQL Key: FLINK-6895 URL: https://issues.apache.org/jira/browse/FLINK-6895 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng STR_TO_DATE(str,format) This is the inverse of the DATE_FORMAT() function. It takes a string str and a format string format. STR_TO_DATE() returns a DATETIME value if the format string contains both date and time parts, or a DATE or TIME value if the string contains only date or time parts. If the date, time, or datetime value extracted from str is illegal, STR_TO_DATE() returns NULL and produces a warning. * Syntax: STR_TO_DATE(str,format) * Arguments **str: - **format: - * Return Types DATAETIME/DATE/TIME * Example: STR_TO_DATE('01,5,2013','%d,%m,%Y') -> '2013-05-01' SELECT STR_TO_DATE('a09:30:17','a%h:%i:%s') -> '09:30:17' * See more: ** [MySQL| https://dev.mysql.com/doc/refman/5.7/en/date-and-time-functions.html#function_str-to-date] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6924) ADD LOG/LPAD/RPAD/BIN supported in TableAPI
sunjincheng created FLINK-6924: -- Summary: ADD LOG/LPAD/RPAD/BIN supported in TableAPI Key: FLINK-6924 URL: https://issues.apache.org/jira/browse/FLINK-6924 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng See FLINK-6891/ FLINK-6892/ FLINK-6893 for detail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6925) Add CONCAT/CONCAT_WS supported in SQL
sunjincheng created FLINK-6925: -- Summary: Add CONCAT/CONCAT_WS supported in SQL Key: FLINK-6925 URL: https://issues.apache.org/jira/browse/FLINK-6925 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng CONCAT(str1,str2,...)Returns the string that results from concatenating the arguments. May have one or more arguments. If all arguments are nonbinary strings, the result is a nonbinary string. If the arguments include any binary strings, the result is a binary string. A numeric argument is converted to its equivalent nonbinary string form. CONCAT() returns NULL if any argument is NULL. * Syntax: CONCAT(str1,str2,...) * Arguments ** str1,str2,... - * Return Types string * Example: CONCAT('F', 'lin', 'k') -> 'Flink' CONCAT('M', NULL, 'L') -> NULL CONCAT(14.3) -> '14.3' * See more: ** [MySQL| https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_concat] CONCAT_WS() stands for Concatenate With Separator and is a special form of CONCAT(). The first argument is the separator for the rest of the arguments. The separator is added between the strings to be concatenated. The separator can be a string, as can the rest of the arguments. If the separator is NULL, the result is NULL. * Syntax: CONCAT_WS(separator,str1,str2,...) * Arguments ** separator - ** str1,str2,... - * Return Types string * Example: CONCAT_WS(',','First name','Second name','Last Name') -> 'First name,Second name,Last Name' * See more: ** [MySQL| https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_concat-ws] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6926) Add MD5/SHA1/SHA2 supported in SQL
sunjincheng created FLINK-6926: -- Summary: Add MD5/SHA1/SHA2 supported in SQL Key: FLINK-6926 URL: https://issues.apache.org/jira/browse/FLINK-6926 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng MD5(str)Calculates an MD5 128-bit checksum for the string. The value is returned as a string of 32 hexadecimal digits, or NULL if the argument was NULL. The return value can, for example, be used as a hash key. See the notes at the beginning of this section about storing hash values efficiently. The return value is a nonbinary string in the connection character set. * Example: MD5('testing') - 'ae2b1fca515949e5d54fb22b8ed95575' * See more: ** [MySQL| https://dev.mysql.com/doc/refman/5.7/en/encryption-functions.html#function_sha1] SHA1(str), SHA(str)Calculates an SHA-1 160-bit checksum for the string, as described in RFC 3174 (Secure Hash Algorithm). The value is returned as a string of 40 hexadecimal digits, or NULL if the argument was NULL. One of the possible uses for this function is as a hash key. See the notes at the beginning of this section about storing hash values efficiently. You can also use SHA1() as a cryptographic function for storing passwords. SHA() is synonymous with SHA1(). The return value is a nonbinary string in the connection character set. * Example: SHA1('abc') -> 'a9993e364706816aba3e25717850c26c9cd0d89d' SHA2(str, hash_length)Calculates the SHA-2 family of hash functions (SHA-224, SHA-256, SHA-384, and SHA-512). The first argument is the cleartext string to be hashed. The second argument indicates the desired bit length of the result, which must have a value of 224, 256, 384, 512, or 0 (which is equivalent to 256). If either argument is NULL or the hash length is not one of the permitted values, the return value is NULL. Otherwise, the function result is a hash value containing the desired number of bits. See the notes at the beginning of this section about storing hash values efficiently. The return value is a nonbinary string in the connection character set. * Example: SHA2('abc', 224) -> '23097d223405d8228642a477bda255b32aadbce4bda0b3f7e36c9da7' * See more: ** [MySQL| https://dev.mysql.com/doc/refman/5.7/en/encryption-functions.html#function_sha2] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6941) Selecting window start / end on over window causes field not resolve exception
sunjincheng created FLINK-6941: -- Summary: Selecting window start / end on over window causes field not resolve exception Key: FLINK-6941 URL: https://issues.apache.org/jira/browse/FLINK-6941 Project: Flink Issue Type: Bug Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng Assignee: sunjincheng Selecting window start / end on over window causes field not resolve exception. The following program: {code} table .window( Over partitionBy 'c orderBy 'proctime preceding UNBOUNDED_ROW as 'w) .select('c, countFun('b) over 'w, 'w.start, 'w.end) {code} causes {code} org.apache.flink.table.api.ValidationException: Cannot resolve [w] given input [a, b, c, proctime]. at org.apache.flink.table.plan.logical.LogicalNode.failValidation(LogicalNode.scala:143) at org.apache.flink.table.plan.logical.LogicalNode$$anonfun$validate$1.applyOrElse(LogicalNode.scala:86) at org.apache.flink.table.plan.logical.LogicalNode$$anonfun$validate$1.applyOrElse(LogicalNode.scala:83) at org.apache.flink.table.plan.TreeNode.postOrderTransform(TreeNode.scala:72) at org.apache.flink.table.plan.TreeNode$$anonfun$1.apply(TreeNode.scala:46) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48) {code} We should validate that the start and end window properties are not accessed on over windows. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6942) Add a document for the FLINK-6810 related scalar functions
sunjincheng created FLINK-6942: -- Summary: Add a document for the FLINK-6810 related scalar functions Key: FLINK-6942 URL: https://issues.apache.org/jira/browse/FLINK-6942 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng Add a document for the FLINK-6810 related scalar functions -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6959) Add documentation for SQL OVER windows
sunjincheng created FLINK-6959: -- Summary: Add documentation for SQL OVER windows Key: FLINK-6959 URL: https://issues.apache.org/jira/browse/FLINK-6959 Project: Flink Issue Type: Improvement Components: Documentation, Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng Assignee: sunjincheng Add documentation for SQL OVER windows. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6960) Add E(2.7182818284590452354),PI(3.14159265358979323846) supported in SQL
sunjincheng created FLINK-6960: -- Summary: Add E(2.7182818284590452354),PI(3.14159265358979323846) supported in SQL Key: FLINK-6960 URL: https://issues.apache.org/jira/browse/FLINK-6960 Project: Flink Issue Type: Sub-task Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng E=Math.E PI=Math.PI -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6973) Add L/RPAD supported in TableAPI
sunjincheng created FLINK-6973: -- Summary: Add L/RPAD supported in TableAPI Key: FLINK-6973 URL: https://issues.apache.org/jira/browse/FLINK-6973 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng See FLINK-6892 for detail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6974) Add BIN supported in TableAPI
sunjincheng created FLINK-6974: -- Summary: Add BIN supported in TableAPI Key: FLINK-6974 URL: https://issues.apache.org/jira/browse/FLINK-6974 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng See FLINK-6893 for detail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6975) Add CONCAT/CONCAT_WS supported in TableAPI
sunjincheng created FLINK-6975: -- Summary: Add CONCAT/CONCAT_WS supported in TableAPI Key: FLINK-6975 URL: https://issues.apache.org/jira/browse/FLINK-6975 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng See FLINK-6925 for detail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6977) Add MD5/SHA1/SHA2 supported in TableAPI
sunjincheng created FLINK-6977: -- Summary: Add MD5/SHA1/SHA2 supported in TableAPI Key: FLINK-6977 URL: https://issues.apache.org/jira/browse/FLINK-6977 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng See FLINK-6895 for detail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)