[GitHub] flink pull request #3051: Flink 5399
GitHub user shijinkui opened a pull request: https://github.com/apache/flink/pull/3051 Flink 5399 Add checkpointId and triggerTime to TriggerSavepointSuccess We can record the history of trigger checkpoint out of Flink System. - [X] General - The pull request references the related JIRA issue ("[FLINK-5399] Add more information to checkpoint result of TriggerSavepointSuccess") - The pull request addresses only one issue - Each commit in the PR has a meaningful commit message (including the JIRA id) - [X] Documentation - Documentation has been added for new functionality - Old documentation affected by the pull request has been updated - JavaDoc for public methods has been added - [X] Tests & Build - Functionality added by the pull request is covered by tests - `mvn clean verify` has been executed successfully locally or a Travis build has passed You can merge this pull request into a Git repository by running: $ git pull https://github.com/shijinkui/flink FLINK-5399 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3051.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3051 commit 3b41c0942ef7ddd5921a32afbee2133392a594b7 Author: shijinkui Date: 2016-12-27T15:51:10Z [FLINK-5395] [Build System] support locally build distribution by script create_release_files.sh commit b5d0bec57586498b99e7d5fd0f02adfd0b46f46a Author: shijinkui Date: 2016-12-29T06:46:57Z [FLINK-5399] Add more information to checkpoint result of TriggerSavepointSuccess --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-5399) Add more information to checkpoint result of TriggerSavepointSuccess
[ https://issues.apache.org/jira/browse/FLINK-5399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15784687#comment-15784687 ] ASF GitHub Bot commented on FLINK-5399: --- GitHub user shijinkui opened a pull request: https://github.com/apache/flink/pull/3051 Flink 5399 Add checkpointId and triggerTime to TriggerSavepointSuccess We can record the history of trigger checkpoint out of Flink System. - [X] General - The pull request references the related JIRA issue ("[FLINK-5399] Add more information to checkpoint result of TriggerSavepointSuccess") - The pull request addresses only one issue - Each commit in the PR has a meaningful commit message (including the JIRA id) - [X] Documentation - Documentation has been added for new functionality - Old documentation affected by the pull request has been updated - JavaDoc for public methods has been added - [X] Tests & Build - Functionality added by the pull request is covered by tests - `mvn clean verify` has been executed successfully locally or a Travis build has passed You can merge this pull request into a Git repository by running: $ git pull https://github.com/shijinkui/flink FLINK-5399 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3051.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3051 commit 3b41c0942ef7ddd5921a32afbee2133392a594b7 Author: shijinkui Date: 2016-12-27T15:51:10Z [FLINK-5395] [Build System] support locally build distribution by script create_release_files.sh commit b5d0bec57586498b99e7d5fd0f02adfd0b46f46a Author: shijinkui Date: 2016-12-29T06:46:57Z [FLINK-5399] Add more information to checkpoint result of TriggerSavepointSuccess > Add more information to checkpoint result of TriggerSavepointSuccess > > > Key: FLINK-5399 > URL: https://issues.apache.org/jira/browse/FLINK-5399 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing >Reporter: shijinkui > > Add checkpointId and triggerTime to TriggerSavepointSuccess > We can record the history of trigger checkpoint out of Flink System. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5303) Add CUBE/ROLLUP/GROUPING SETS operator in SQL
[ https://issues.apache.org/jira/browse/FLINK-5303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15784659#comment-15784659 ] ASF GitHub Bot commented on FLINK-5303: --- Github user KurtYoung commented on a diff in the pull request: https://github.com/apache/flink/pull/2976#discussion_r94104843 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/runtime/aggregate/AggregateUtil.scala --- @@ -98,7 +98,9 @@ object AggregateUtil { namedAggregates: Seq[CalcitePair[AggregateCall, String]], inputType: RelDataType, outputType: RelDataType, -groupings: Array[Int]): RichGroupReduceFunction[Row, Row] = { +groupings: Array[Int], +indicator: Boolean --- End diff -- This variable is not used > Add CUBE/ROLLUP/GROUPING SETS operator in SQL > - > > Key: FLINK-5303 > URL: https://issues.apache.org/jira/browse/FLINK-5303 > Project: Flink > Issue Type: New Feature > Components: Documentation, Table API & SQL >Reporter: Alexander Chermenin >Assignee: Alexander Chermenin > > Add support for such operators as CUBE, ROLLUP and GROUPING SETS in SQL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5303) Add CUBE/ROLLUP/GROUPING SETS operator in SQL
[ https://issues.apache.org/jira/browse/FLINK-5303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15784656#comment-15784656 ] ASF GitHub Bot commented on FLINK-5303: --- Github user KurtYoung commented on a diff in the pull request: https://github.com/apache/flink/pull/2976#discussion_r94104766 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/runtime/aggregate/AggregateReduceCombineFunction.scala --- @@ -41,12 +41,14 @@ class AggregateReduceCombineFunction( private val aggregates: Array[Aggregate[_ <: Any]], private val groupKeysMapping: Array[(Int, Int)], private val aggregateMapping: Array[(Int, Int)], +private val additionalMapping: Array[(Int, Int)], --- End diff -- Can we have a more explicit name about "what is the additional mapping?". And we should update the class document. > Add CUBE/ROLLUP/GROUPING SETS operator in SQL > - > > Key: FLINK-5303 > URL: https://issues.apache.org/jira/browse/FLINK-5303 > Project: Flink > Issue Type: New Feature > Components: Documentation, Table API & SQL >Reporter: Alexander Chermenin >Assignee: Alexander Chermenin > > Add support for such operators as CUBE, ROLLUP and GROUPING SETS in SQL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #2976: [FLINK-5303] [table] Support for SQL GROUPING SETS...
Github user KurtYoung commented on a diff in the pull request: https://github.com/apache/flink/pull/2976#discussion_r94104843 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/runtime/aggregate/AggregateUtil.scala --- @@ -98,7 +98,9 @@ object AggregateUtil { namedAggregates: Seq[CalcitePair[AggregateCall, String]], inputType: RelDataType, outputType: RelDataType, -groupings: Array[Int]): RichGroupReduceFunction[Row, Row] = { +groupings: Array[Int], +indicator: Boolean --- End diff -- This variable is not used --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2976: [FLINK-5303] [table] Support for SQL GROUPING SETS...
Github user KurtYoung commented on a diff in the pull request: https://github.com/apache/flink/pull/2976#discussion_r94104787 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/runtime/aggregate/AggregateReduceGroupFunction.scala --- @@ -40,6 +40,7 @@ class AggregateReduceGroupFunction( private val aggregates: Array[Aggregate[_ <: Any]], private val groupKeysMapping: Array[(Int, Int)], private val aggregateMapping: Array[(Int, Int)], +private val additionalMapping: Array[(Int, Int)], --- End diff -- Please update the class document --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-5303) Add CUBE/ROLLUP/GROUPING SETS operator in SQL
[ https://issues.apache.org/jira/browse/FLINK-5303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15784657#comment-15784657 ] ASF GitHub Bot commented on FLINK-5303: --- Github user KurtYoung commented on a diff in the pull request: https://github.com/apache/flink/pull/2976#discussion_r94104787 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/runtime/aggregate/AggregateReduceGroupFunction.scala --- @@ -40,6 +40,7 @@ class AggregateReduceGroupFunction( private val aggregates: Array[Aggregate[_ <: Any]], private val groupKeysMapping: Array[(Int, Int)], private val aggregateMapping: Array[(Int, Int)], +private val additionalMapping: Array[(Int, Int)], --- End diff -- Please update the class document > Add CUBE/ROLLUP/GROUPING SETS operator in SQL > - > > Key: FLINK-5303 > URL: https://issues.apache.org/jira/browse/FLINK-5303 > Project: Flink > Issue Type: New Feature > Components: Documentation, Table API & SQL >Reporter: Alexander Chermenin >Assignee: Alexander Chermenin > > Add support for such operators as CUBE, ROLLUP and GROUPING SETS in SQL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #2976: [FLINK-5303] [table] Support for SQL GROUPING SETS...
Github user KurtYoung commented on a diff in the pull request: https://github.com/apache/flink/pull/2976#discussion_r94104766 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/runtime/aggregate/AggregateReduceCombineFunction.scala --- @@ -41,12 +41,14 @@ class AggregateReduceCombineFunction( private val aggregates: Array[Aggregate[_ <: Any]], private val groupKeysMapping: Array[(Int, Int)], private val aggregateMapping: Array[(Int, Int)], +private val additionalMapping: Array[(Int, Int)], --- End diff -- Can we have a more explicit name about "what is the additional mapping?". And we should update the class document. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-5303) Add CUBE/ROLLUP/GROUPING SETS operator in SQL
[ https://issues.apache.org/jira/browse/FLINK-5303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15784651#comment-15784651 ] ASF GitHub Bot commented on FLINK-5303: --- Github user KurtYoung commented on a diff in the pull request: https://github.com/apache/flink/pull/2976#discussion_r94104637 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/plan/nodes/datastream/DataStreamAggregate.scala --- @@ -70,7 +71,9 @@ class DataStreamAggregate( namedAggregates, getRowType, inputType, - grouping) + grouping, + indicator +) --- End diff -- No need to have new line here. > Add CUBE/ROLLUP/GROUPING SETS operator in SQL > - > > Key: FLINK-5303 > URL: https://issues.apache.org/jira/browse/FLINK-5303 > Project: Flink > Issue Type: New Feature > Components: Documentation, Table API & SQL >Reporter: Alexander Chermenin >Assignee: Alexander Chermenin > > Add support for such operators as CUBE, ROLLUP and GROUPING SETS in SQL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5303) Add CUBE/ROLLUP/GROUPING SETS operator in SQL
[ https://issues.apache.org/jira/browse/FLINK-5303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15784650#comment-15784650 ] ASF GitHub Bot commented on FLINK-5303: --- Github user KurtYoung commented on a diff in the pull request: https://github.com/apache/flink/pull/2976#discussion_r94104624 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/plan/nodes/dataset/DataSetAggregate.scala --- @@ -104,7 +107,9 @@ class DataSetAggregate( namedAggregates, inputType, rowRelDataType, - grouping) + grouping, + indicator +) --- End diff -- No need to have new line here. > Add CUBE/ROLLUP/GROUPING SETS operator in SQL > - > > Key: FLINK-5303 > URL: https://issues.apache.org/jira/browse/FLINK-5303 > Project: Flink > Issue Type: New Feature > Components: Documentation, Table API & SQL >Reporter: Alexander Chermenin >Assignee: Alexander Chermenin > > Add support for such operators as CUBE, ROLLUP and GROUPING SETS in SQL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5303) Add CUBE/ROLLUP/GROUPING SETS operator in SQL
[ https://issues.apache.org/jira/browse/FLINK-5303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15784652#comment-15784652 ] ASF GitHub Bot commented on FLINK-5303: --- Github user KurtYoung commented on a diff in the pull request: https://github.com/apache/flink/pull/2976#discussion_r94104657 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/plan/nodes/datastream/DataStreamAggregate.scala --- @@ -207,7 +210,9 @@ class DataStreamAggregate( inputType, rowRelDataType, grouping, -namedProperties) +indicator, +namedProperties + ) --- End diff -- Same as here and a couple of other places > Add CUBE/ROLLUP/GROUPING SETS operator in SQL > - > > Key: FLINK-5303 > URL: https://issues.apache.org/jira/browse/FLINK-5303 > Project: Flink > Issue Type: New Feature > Components: Documentation, Table API & SQL >Reporter: Alexander Chermenin >Assignee: Alexander Chermenin > > Add support for such operators as CUBE, ROLLUP and GROUPING SETS in SQL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #2976: [FLINK-5303] [table] Support for SQL GROUPING SETS...
Github user KurtYoung commented on a diff in the pull request: https://github.com/apache/flink/pull/2976#discussion_r94104657 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/plan/nodes/datastream/DataStreamAggregate.scala --- @@ -207,7 +210,9 @@ class DataStreamAggregate( inputType, rowRelDataType, grouping, -namedProperties) +indicator, +namedProperties + ) --- End diff -- Same as here and a couple of other places --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2976: [FLINK-5303] [table] Support for SQL GROUPING SETS...
Github user KurtYoung commented on a diff in the pull request: https://github.com/apache/flink/pull/2976#discussion_r94104637 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/plan/nodes/datastream/DataStreamAggregate.scala --- @@ -70,7 +71,9 @@ class DataStreamAggregate( namedAggregates, getRowType, inputType, - grouping) + grouping, + indicator +) --- End diff -- No need to have new line here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2976: [FLINK-5303] [table] Support for SQL GROUPING SETS...
Github user KurtYoung commented on a diff in the pull request: https://github.com/apache/flink/pull/2976#discussion_r94104624 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/plan/nodes/dataset/DataSetAggregate.scala --- @@ -104,7 +107,9 @@ class DataSetAggregate( namedAggregates, inputType, rowRelDataType, - grouping) + grouping, + indicator +) --- End diff -- No need to have new line here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-5399) Add more information to checkpoint result of TriggerSavepointSuccess
shijinkui created FLINK-5399: Summary: Add more information to checkpoint result of TriggerSavepointSuccess Key: FLINK-5399 URL: https://issues.apache.org/jira/browse/FLINK-5399 Project: Flink Issue Type: Improvement Components: State Backends, Checkpointing Reporter: shijinkui Add checkpointId and triggerTime to TriggerSavepointSuccess We can record the history of trigger checkpoint out of Flink System. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-5398) Exclude generated files in module flink-batch-connectors in license checking
Xiaogang Shi created FLINK-5398: --- Summary: Exclude generated files in module flink-batch-connectors in license checking Key: FLINK-5398 URL: https://issues.apache.org/jira/browse/FLINK-5398 Project: Flink Issue Type: Bug Reporter: Xiaogang Shi Now the master branch fails to execute {{mvn install}} due to unlicensed files in the module flink-batch-connectors. We should exclude these generated files in the pom file. Unapproved licenses: flink-batch-connectors/flink-avro/src/test/java/org/apache/flink/api/io/avro/generated/Address.java flink-batch-connectors/flink-avro/src/test/java/org/apache/flink/api/io/avro/generated/Colors.java flink-batch-connectors/flink-avro/src/test/java/org/apache/flink/api/io/avro/generated/Fixed16.java flink-batch-connectors/flink-avro/src/test/java/org/apache/flink/api/io/avro/generated/User.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5280) Extend TableSource to support nested data
[ https://issues.apache.org/jira/browse/FLINK-5280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15784253#comment-15784253 ] ASF GitHub Bot commented on FLINK-5280: --- Github user wuchong commented on a diff in the pull request: https://github.com/apache/flink/pull/3039#discussion_r94095173 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/functions/utils/UserDefinedFunctionUtils.scala --- @@ -268,23 +268,9 @@ object UserDefinedFunctionUtils { def getFieldInfo(inputType: TypeInformation[_]) --- End diff -- I think we can refactor this. Provide a `getFieldInfo` static method, get the field names from `getFieldInfo`, and then do additional check outside. What do you think ? > Extend TableSource to support nested data > - > > Key: FLINK-5280 > URL: https://issues.apache.org/jira/browse/FLINK-5280 > Project: Flink > Issue Type: Improvement > Components: Table API & SQL >Affects Versions: 1.2.0 >Reporter: Fabian Hueske >Assignee: Ivan Mushketyk > > The {{TableSource}} interface does currently only support the definition of > flat rows. > However, there are several storage formats for nested data that should be > supported such as Avro, Json, Parquet, and Orc. The Table API and SQL can > also natively handle nested rows. > The {{TableSource}} interface and the code to register table sources in > Calcite's schema need to be extended to support nested data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #3039: [FLINK-5280] Update TableSource to support nested ...
Github user wuchong commented on a diff in the pull request: https://github.com/apache/flink/pull/3039#discussion_r94095173 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/functions/utils/UserDefinedFunctionUtils.scala --- @@ -268,23 +268,9 @@ object UserDefinedFunctionUtils { def getFieldInfo(inputType: TypeInformation[_]) --- End diff -- I think we can refactor this. Provide a `getFieldInfo` static method, get the field names from `getFieldInfo`, and then do additional check outside. What do you think ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3039: [FLINK-5280] Update TableSource to support nested ...
Github user mushketyk commented on a diff in the pull request: https://github.com/apache/flink/pull/3039#discussion_r94087339 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/functions/utils/UserDefinedFunctionUtils.scala --- @@ -268,23 +268,9 @@ object UserDefinedFunctionUtils { def getFieldInfo(inputType: TypeInformation[_]) --- End diff -- `getFieldInfo` is overridden in `StreamTableEnvironment`, so I cannot make it a static method. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-5280) Extend TableSource to support nested data
[ https://issues.apache.org/jira/browse/FLINK-5280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15783939#comment-15783939 ] ASF GitHub Bot commented on FLINK-5280: --- Github user mushketyk commented on a diff in the pull request: https://github.com/apache/flink/pull/3039#discussion_r94087339 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/functions/utils/UserDefinedFunctionUtils.scala --- @@ -268,23 +268,9 @@ object UserDefinedFunctionUtils { def getFieldInfo(inputType: TypeInformation[_]) --- End diff -- `getFieldInfo` is overridden in `StreamTableEnvironment`, so I cannot make it a static method. > Extend TableSource to support nested data > - > > Key: FLINK-5280 > URL: https://issues.apache.org/jira/browse/FLINK-5280 > Project: Flink > Issue Type: Improvement > Components: Table API & SQL >Affects Versions: 1.2.0 >Reporter: Fabian Hueske >Assignee: Ivan Mushketyk > > The {{TableSource}} interface does currently only support the definition of > flat rows. > However, there are several storage formats for nested data that should be > supported such as Avro, Json, Parquet, and Orc. The Table API and SQL can > also natively handle nested rows. > The {{TableSource}} interface and the code to register table sources in > Calcite's schema need to be extended to support nested data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5397) Fail to deserialize savepoints in v1.1 when there exist missing fields in class serialization descriptors
[ https://issues.apache.org/jira/browse/FLINK-5397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15782811#comment-15782811 ] Xiaogang Shi commented on FLINK-5397: - The idea does work and is better. Very thanks for the quick fix :) > Fail to deserialize savepoints in v1.1 when there exist missing fields in > class serialization descriptors > - > > Key: FLINK-5397 > URL: https://issues.apache.org/jira/browse/FLINK-5397 > Project: Flink > Issue Type: Bug > Components: State Backends, Checkpointing >Reporter: Xiaogang Shi >Assignee: Stefan Richter > > To restore from the savepoints in previous versions, Flink now keeps all > classes whose serialization is changed and put them in a separated package > ("migration"). > When deserializing the old savepoints, flink will look up correct descriptors > ({{ObjectStreamClass}}) for these classes, without using those ones written > in serialized data. The implementation however is problematic when there > exist missing field descriptors in the serialized data. > When serializing an object, Java will only write the descriptors of those > non-null fields. But when we look up class descriptors with given classes, > all fields will be put into the descriptors. As a result, we will deserialize > the savepoints with incorrect descriptors, leading to serialization > exceptions. > A simple resolution is to update the name of read descriptors using > Reflections, without using different descriptors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5397) Fail to deserialize savepoints in v1.1 when there exist missing fields in class serialization descriptors
[ https://issues.apache.org/jira/browse/FLINK-5397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15782747#comment-15782747 ] Stefan Richter commented on FLINK-5397: --- Thanks for pointing out this problem. I opened a PR #3050 with a fix that does not require overriding strings by reflection. The idea is to simply replace the class only during {{resolveClass(...)}}. > Fail to deserialize savepoints in v1.1 when there exist missing fields in > class serialization descriptors > - > > Key: FLINK-5397 > URL: https://issues.apache.org/jira/browse/FLINK-5397 > Project: Flink > Issue Type: Bug > Components: State Backends, Checkpointing >Reporter: Xiaogang Shi >Assignee: Stefan Richter > > To restore from the savepoints in previous versions, Flink now keeps all > classes whose serialization is changed and put them in a separated package > ("migration"). > When deserializing the old savepoints, flink will look up correct descriptors > ({{ObjectStreamClass}}) for these classes, without using those ones written > in serialized data. The implementation however is problematic when there > exist missing field descriptors in the serialized data. > When serializing an object, Java will only write the descriptors of those > non-null fields. But when we look up class descriptors with given classes, > all fields will be put into the descriptors. As a result, we will deserialize > the savepoints with incorrect descriptors, leading to serialization > exceptions. > A simple resolution is to update the name of read descriptors using > Reflections, without using different descriptors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5397) Fail to deserialize savepoints in v1.1 when there exist missing fields in class serialization descriptors
[ https://issues.apache.org/jira/browse/FLINK-5397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15782743#comment-15782743 ] ASF GitHub Bot commented on FLINK-5397: --- GitHub user StefanRRichter opened a pull request: https://github.com/apache/flink/pull/3050 [FLINK-5397] Do not replace ObjectStreamClass on deserialization of m… This PR fixes [FLINK-5397] by not replacing the ObjectStreamClass in ``readClassDescriptor()`` during deserialization of classes that should be replaced from the migration package. Instead, we replace the class directly in ``resolveClass(...)``. You can merge this pull request into a Git repository by running: $ git pull https://github.com/StefanRRichter/flink migration-keep-objectstreamclass Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3050.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3050 commit 01834068b50a9cf4186ce83fa9c57d5b8f2bfc8c Author: Stefan Richter Date: 2016-12-28T11:50:00Z [FLINK-5397] Do not replace ObjectStreamClass on deserialization of migration package classes, override resolveClass(...) instead > Fail to deserialize savepoints in v1.1 when there exist missing fields in > class serialization descriptors > - > > Key: FLINK-5397 > URL: https://issues.apache.org/jira/browse/FLINK-5397 > Project: Flink > Issue Type: Bug > Components: State Backends, Checkpointing >Reporter: Xiaogang Shi >Assignee: Stefan Richter > > To restore from the savepoints in previous versions, Flink now keeps all > classes whose serialization is changed and put them in a separated package > ("migration"). > When deserializing the old savepoints, flink will look up correct descriptors > ({{ObjectStreamClass}}) for these classes, without using those ones written > in serialized data. The implementation however is problematic when there > exist missing field descriptors in the serialized data. > When serializing an object, Java will only write the descriptors of those > non-null fields. But when we look up class descriptors with given classes, > all fields will be put into the descriptors. As a result, we will deserialize > the savepoints with incorrect descriptors, leading to serialization > exceptions. > A simple resolution is to update the name of read descriptors using > Reflections, without using different descriptors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #3050: [FLINK-5397] Do not replace ObjectStreamClass on d...
GitHub user StefanRRichter opened a pull request: https://github.com/apache/flink/pull/3050 [FLINK-5397] Do not replace ObjectStreamClass on deserialization of m⦠This PR fixes [FLINK-5397] by not replacing the ObjectStreamClass in ``readClassDescriptor()`` during deserialization of classes that should be replaced from the migration package. Instead, we replace the class directly in ``resolveClass(...)``. You can merge this pull request into a Git repository by running: $ git pull https://github.com/StefanRRichter/flink migration-keep-objectstreamclass Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3050.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3050 commit 01834068b50a9cf4186ce83fa9c57d5b8f2bfc8c Author: Stefan Richter Date: 2016-12-28T11:50:00Z [FLINK-5397] Do not replace ObjectStreamClass on deserialization of migration package classes, override resolveClass(...) instead --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Assigned] (FLINK-5397) Fail to deserialize savepoints in v1.1 when there exist missing fields in class serialization descriptors
[ https://issues.apache.org/jira/browse/FLINK-5397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Richter reassigned FLINK-5397: - Assignee: Stefan Richter > Fail to deserialize savepoints in v1.1 when there exist missing fields in > class serialization descriptors > - > > Key: FLINK-5397 > URL: https://issues.apache.org/jira/browse/FLINK-5397 > Project: Flink > Issue Type: Bug > Components: State Backends, Checkpointing >Reporter: Xiaogang Shi >Assignee: Stefan Richter > > To restore from the savepoints in previous versions, Flink now keeps all > classes whose serialization is changed and put them in a separated package > ("migration"). > When deserializing the old savepoints, flink will look up correct descriptors > ({{ObjectStreamClass}}) for these classes, without using those ones written > in serialized data. The implementation however is problematic when there > exist missing field descriptors in the serialized data. > When serializing an object, Java will only write the descriptors of those > non-null fields. But when we look up class descriptors with given classes, > all fields will be put into the descriptors. As a result, we will deserialize > the savepoints with incorrect descriptors, leading to serialization > exceptions. > A simple resolution is to update the name of read descriptors using > Reflections, without using different descriptors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (FLINK-4641) Support branching CEP patterns
[ https://issues.apache.org/jira/browse/FLINK-4641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Chermenin reassigned FLINK-4641: -- Assignee: Alexander Chermenin > Support branching CEP patterns > --- > > Key: FLINK-4641 > URL: https://issues.apache.org/jira/browse/FLINK-4641 > Project: Flink > Issue Type: Improvement > Components: CEP >Reporter: Till Rohrmann >Assignee: Alexander Chermenin > > We should add support for branching CEP patterns to the Pattern API. > {code} > |--> B --| > || > A -- --> D > || > |--> C --| > {code} > This feature will require changes to the {{Pattern}} class and the > {{NFACompiler}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (FLINK-4540) Detached job execution may prevent cluster shutdown
[ https://issues.apache.org/jira/browse/FLINK-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Maier reassigned FLINK-4540: -- Assignee: Roman Maier > Detached job execution may prevent cluster shutdown > --- > > Key: FLINK-4540 > URL: https://issues.apache.org/jira/browse/FLINK-4540 > Project: Flink > Issue Type: Bug > Components: YARN >Affects Versions: 1.2.0, 1.1.2 >Reporter: Maximilian Michels >Assignee: Roman Maier >Priority: Minor > Fix For: 1.2.0 > > > There is a problem with the detached execution of jobs. This can prevent > cluster shutdown 1) when eager jobs are executed, i.e. the job calls > `collect()/count()`, and 2) when the user jar doesn't contain a job. > 1) For example, {{./flink -d -m yarn-cluster -yn 1 > ../examples/batch/WordCount.jar}} will throw an exception and only disconnect > the YarnClusterClient afterwards. In detached mode, the code assumes the > cluster is shutdown through the {{shutdownAfterJob}} method which ensures > that the YarnJobManager shuts down after the job completes. Due to the > exception thrown when executing eager jobs, the jobmanager never receives a > job and thus never shuts down the cluster. > 2) The same problem also occurs in detached execution when the user jar > doesn't contain a job. > A good solution would be to defer cluster startup until the job has been > fully assembled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-5397) Fail to deserialize savepoints in v1.1 when there exist missing fields in class serialization descriptors
[ https://issues.apache.org/jira/browse/FLINK-5397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaogang Shi updated FLINK-5397: Description: To restore from the savepoints in previous versions, Flink now keeps all classes whose serialization is changed and put them in a separated package ("migration"). When deserializing the old savepoints, flink will look up correct descriptors ({{ObjectStreamClass}}) for these classes, without using those ones written in serialized data. The implementation however is problematic when there exist missing field descriptors in the serialized data. When serializing an object, Java will only write the descriptors of those non-null fields. But when we look up class descriptors with given classes, all fields will be put into the descriptors. As a result, we will deserialize the savepoints with incorrect descriptors, leading to serialization exceptions. A simple resolution is to update the name of read descriptors using Reflections, without using different descriptors. was: To restore from the savepoints in previous versions, Flink now keeps all classes whose serialization is changed and put them in a separated package ("migration"). When deserializing the old savepoints, flink will look up correct descriptors ({{ObjectStreamClass}}) for these classes, without using those ones written in serialized data. The implementation however is problematic when there exist missing field descriptors in the serialized data. When deserializing an object, Java will only write the descriptors of those non-null fields. But when we look up class descriptors with given classes, all fields will be put into the descriptors. As a result, we will deserialize the savepoints with incorrect descriptors, leading to serialization exceptions. A simple resolution is to update the name of read descriptors using Reflections, without using a different descriptors. > Fail to deserialize savepoints in v1.1 when there exist missing fields in > class serialization descriptors > - > > Key: FLINK-5397 > URL: https://issues.apache.org/jira/browse/FLINK-5397 > Project: Flink > Issue Type: Bug > Components: State Backends, Checkpointing >Reporter: Xiaogang Shi > > To restore from the savepoints in previous versions, Flink now keeps all > classes whose serialization is changed and put them in a separated package > ("migration"). > When deserializing the old savepoints, flink will look up correct descriptors > ({{ObjectStreamClass}}) for these classes, without using those ones written > in serialized data. The implementation however is problematic when there > exist missing field descriptors in the serialized data. > When serializing an object, Java will only write the descriptors of those > non-null fields. But when we look up class descriptors with given classes, > all fields will be put into the descriptors. As a result, we will deserialize > the savepoints with incorrect descriptors, leading to serialization > exceptions. > A simple resolution is to update the name of read descriptors using > Reflections, without using different descriptors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-5397) Fail to deserialize savepoints in v1.1 when there exist missing fields in class serialization descriptors
Xiaogang Shi created FLINK-5397: --- Summary: Fail to deserialize savepoints in v1.1 when there exist missing fields in class serialization descriptors Key: FLINK-5397 URL: https://issues.apache.org/jira/browse/FLINK-5397 Project: Flink Issue Type: Bug Components: State Backends, Checkpointing Reporter: Xiaogang Shi To restore from the savepoints in previous versions, Flink now keeps all classes whose serialization is changed and put them in a separated package ("migration"). When deserializing the old savepoints, flink will look up correct descriptors ({{ObjectStreamClass}}) for these classes, without using those ones written in serialized data. The implementation however is problematic when there exist missing field descriptors in the serialized data. When deserializing an object, Java will only write the descriptors of those non-null fields. But when we look up class descriptors with given classes, all fields will be put into the descriptors. As a result, we will deserialize the savepoints with incorrect descriptors, leading to serialization exceptions. A simple resolution is to update the name of read descriptors using Reflections, without using a different descriptors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3615) Add support for non-native SQL types
[ https://issues.apache.org/jira/browse/FLINK-3615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15782410#comment-15782410 ] Alexander Chermenin commented on FLINK-3615: Hi all. Is it an actual issue or it has been solved in FLINK-3916? > Add support for non-native SQL types > > > Key: FLINK-3615 > URL: https://issues.apache.org/jira/browse/FLINK-3615 > Project: Flink > Issue Type: Improvement > Components: Table API & SQL >Reporter: Vasia Kalavri > > The TypeConverter of the Table API currently only supports basic types. We > should maybe re-design the way {{sqlTypeToTypeInfo}} works. It is used in the > {{CodeGenerator}} for visiting literals, in {{DataSetAggregate}} to create > the {{RowTypeInfo}} and in {{determineReturnType}}. We could maybe provide a > custom implementation per operator to determine the return type, based on the > input fields. -- This message was sent by Atlassian JIRA (v6.3.4#6332)