[GitHub] flink pull request #3051: Flink 5399
GitHub user shijinkui opened a pull request:
https://github.com/apache/flink/pull/3051
Flink 5399
Add checkpointId and triggerTime to TriggerSavepointSuccess
We can record the history of trigger checkpoint out of Flink System.
- [X] General
- The pull request references the related JIRA issue ("[FLINK-5399] Add
more information to checkpoint result of TriggerSavepointSuccess")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [X] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [X] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shijinkui/flink FLINK-5399
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3051.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3051
commit 3b41c0942ef7ddd5921a32afbee2133392a594b7
Author: shijinkui
Date: 2016-12-27T15:51:10Z
[FLINK-5395] [Build System] support locally build distribution by script
create_release_files.sh
commit b5d0bec57586498b99e7d5fd0f02adfd0b46f46a
Author: shijinkui
Date: 2016-12-29T06:46:57Z
[FLINK-5399] Add more information to checkpoint result of
TriggerSavepointSuccess
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-5399) Add more information to checkpoint result of TriggerSavepointSuccess
[
https://issues.apache.org/jira/browse/FLINK-5399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784687#comment-15784687
]
ASF GitHub Bot commented on FLINK-5399:
---
GitHub user shijinkui opened a pull request:
https://github.com/apache/flink/pull/3051
Flink 5399
Add checkpointId and triggerTime to TriggerSavepointSuccess
We can record the history of trigger checkpoint out of Flink System.
- [X] General
- The pull request references the related JIRA issue ("[FLINK-5399] Add
more information to checkpoint result of TriggerSavepointSuccess")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [X] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [X] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shijinkui/flink FLINK-5399
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3051.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3051
commit 3b41c0942ef7ddd5921a32afbee2133392a594b7
Author: shijinkui
Date: 2016-12-27T15:51:10Z
[FLINK-5395] [Build System] support locally build distribution by script
create_release_files.sh
commit b5d0bec57586498b99e7d5fd0f02adfd0b46f46a
Author: shijinkui
Date: 2016-12-29T06:46:57Z
[FLINK-5399] Add more information to checkpoint result of
TriggerSavepointSuccess
> Add more information to checkpoint result of TriggerSavepointSuccess
>
>
> Key: FLINK-5399
> URL: https://issues.apache.org/jira/browse/FLINK-5399
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
>Reporter: shijinkui
>
> Add checkpointId and triggerTime to TriggerSavepointSuccess
> We can record the history of trigger checkpoint out of Flink System.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-5303) Add CUBE/ROLLUP/GROUPING SETS operator in SQL
[
https://issues.apache.org/jira/browse/FLINK-5303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784659#comment-15784659
]
ASF GitHub Bot commented on FLINK-5303:
---
Github user KurtYoung commented on a diff in the pull request:
https://github.com/apache/flink/pull/2976#discussion_r94104843
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/runtime/aggregate/AggregateUtil.scala
---
@@ -98,7 +98,9 @@ object AggregateUtil {
namedAggregates: Seq[CalcitePair[AggregateCall, String]],
inputType: RelDataType,
outputType: RelDataType,
-groupings: Array[Int]): RichGroupReduceFunction[Row, Row] = {
+groupings: Array[Int],
+indicator: Boolean
--- End diff --
This variable is not used
> Add CUBE/ROLLUP/GROUPING SETS operator in SQL
> -
>
> Key: FLINK-5303
> URL: https://issues.apache.org/jira/browse/FLINK-5303
> Project: Flink
> Issue Type: New Feature
> Components: Documentation, Table API & SQL
>Reporter: Alexander Chermenin
>Assignee: Alexander Chermenin
>
> Add support for such operators as CUBE, ROLLUP and GROUPING SETS in SQL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-5303) Add CUBE/ROLLUP/GROUPING SETS operator in SQL
[ https://issues.apache.org/jira/browse/FLINK-5303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784656#comment-15784656 ] ASF GitHub Bot commented on FLINK-5303: --- Github user KurtYoung commented on a diff in the pull request: https://github.com/apache/flink/pull/2976#discussion_r94104766 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/runtime/aggregate/AggregateReduceCombineFunction.scala --- @@ -41,12 +41,14 @@ class AggregateReduceCombineFunction( private val aggregates: Array[Aggregate[_ <: Any]], private val groupKeysMapping: Array[(Int, Int)], private val aggregateMapping: Array[(Int, Int)], +private val additionalMapping: Array[(Int, Int)], --- End diff -- Can we have a more explicit name about "what is the additional mapping?". And we should update the class document. > Add CUBE/ROLLUP/GROUPING SETS operator in SQL > - > > Key: FLINK-5303 > URL: https://issues.apache.org/jira/browse/FLINK-5303 > Project: Flink > Issue Type: New Feature > Components: Documentation, Table API & SQL >Reporter: Alexander Chermenin >Assignee: Alexander Chermenin > > Add support for such operators as CUBE, ROLLUP and GROUPING SETS in SQL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #2976: [FLINK-5303] [table] Support for SQL GROUPING SETS...
Github user KurtYoung commented on a diff in the pull request:
https://github.com/apache/flink/pull/2976#discussion_r94104843
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/runtime/aggregate/AggregateUtil.scala
---
@@ -98,7 +98,9 @@ object AggregateUtil {
namedAggregates: Seq[CalcitePair[AggregateCall, String]],
inputType: RelDataType,
outputType: RelDataType,
-groupings: Array[Int]): RichGroupReduceFunction[Row, Row] = {
+groupings: Array[Int],
+indicator: Boolean
--- End diff --
This variable is not used
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2976: [FLINK-5303] [table] Support for SQL GROUPING SETS...
Github user KurtYoung commented on a diff in the pull request: https://github.com/apache/flink/pull/2976#discussion_r94104787 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/runtime/aggregate/AggregateReduceGroupFunction.scala --- @@ -40,6 +40,7 @@ class AggregateReduceGroupFunction( private val aggregates: Array[Aggregate[_ <: Any]], private val groupKeysMapping: Array[(Int, Int)], private val aggregateMapping: Array[(Int, Int)], +private val additionalMapping: Array[(Int, Int)], --- End diff -- Please update the class document --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-5303) Add CUBE/ROLLUP/GROUPING SETS operator in SQL
[ https://issues.apache.org/jira/browse/FLINK-5303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784657#comment-15784657 ] ASF GitHub Bot commented on FLINK-5303: --- Github user KurtYoung commented on a diff in the pull request: https://github.com/apache/flink/pull/2976#discussion_r94104787 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/runtime/aggregate/AggregateReduceGroupFunction.scala --- @@ -40,6 +40,7 @@ class AggregateReduceGroupFunction( private val aggregates: Array[Aggregate[_ <: Any]], private val groupKeysMapping: Array[(Int, Int)], private val aggregateMapping: Array[(Int, Int)], +private val additionalMapping: Array[(Int, Int)], --- End diff -- Please update the class document > Add CUBE/ROLLUP/GROUPING SETS operator in SQL > - > > Key: FLINK-5303 > URL: https://issues.apache.org/jira/browse/FLINK-5303 > Project: Flink > Issue Type: New Feature > Components: Documentation, Table API & SQL >Reporter: Alexander Chermenin >Assignee: Alexander Chermenin > > Add support for such operators as CUBE, ROLLUP and GROUPING SETS in SQL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #2976: [FLINK-5303] [table] Support for SQL GROUPING SETS...
Github user KurtYoung commented on a diff in the pull request: https://github.com/apache/flink/pull/2976#discussion_r94104766 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/runtime/aggregate/AggregateReduceCombineFunction.scala --- @@ -41,12 +41,14 @@ class AggregateReduceCombineFunction( private val aggregates: Array[Aggregate[_ <: Any]], private val groupKeysMapping: Array[(Int, Int)], private val aggregateMapping: Array[(Int, Int)], +private val additionalMapping: Array[(Int, Int)], --- End diff -- Can we have a more explicit name about "what is the additional mapping?". And we should update the class document. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-5303) Add CUBE/ROLLUP/GROUPING SETS operator in SQL
[ https://issues.apache.org/jira/browse/FLINK-5303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784651#comment-15784651 ] ASF GitHub Bot commented on FLINK-5303: --- Github user KurtYoung commented on a diff in the pull request: https://github.com/apache/flink/pull/2976#discussion_r94104637 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/plan/nodes/datastream/DataStreamAggregate.scala --- @@ -70,7 +71,9 @@ class DataStreamAggregate( namedAggregates, getRowType, inputType, - grouping) + grouping, + indicator +) --- End diff -- No need to have new line here. > Add CUBE/ROLLUP/GROUPING SETS operator in SQL > - > > Key: FLINK-5303 > URL: https://issues.apache.org/jira/browse/FLINK-5303 > Project: Flink > Issue Type: New Feature > Components: Documentation, Table API & SQL >Reporter: Alexander Chermenin >Assignee: Alexander Chermenin > > Add support for such operators as CUBE, ROLLUP and GROUPING SETS in SQL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5303) Add CUBE/ROLLUP/GROUPING SETS operator in SQL
[ https://issues.apache.org/jira/browse/FLINK-5303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784650#comment-15784650 ] ASF GitHub Bot commented on FLINK-5303: --- Github user KurtYoung commented on a diff in the pull request: https://github.com/apache/flink/pull/2976#discussion_r94104624 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/plan/nodes/dataset/DataSetAggregate.scala --- @@ -104,7 +107,9 @@ class DataSetAggregate( namedAggregates, inputType, rowRelDataType, - grouping) + grouping, + indicator +) --- End diff -- No need to have new line here. > Add CUBE/ROLLUP/GROUPING SETS operator in SQL > - > > Key: FLINK-5303 > URL: https://issues.apache.org/jira/browse/FLINK-5303 > Project: Flink > Issue Type: New Feature > Components: Documentation, Table API & SQL >Reporter: Alexander Chermenin >Assignee: Alexander Chermenin > > Add support for such operators as CUBE, ROLLUP and GROUPING SETS in SQL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5303) Add CUBE/ROLLUP/GROUPING SETS operator in SQL
[ https://issues.apache.org/jira/browse/FLINK-5303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784652#comment-15784652 ] ASF GitHub Bot commented on FLINK-5303: --- Github user KurtYoung commented on a diff in the pull request: https://github.com/apache/flink/pull/2976#discussion_r94104657 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/plan/nodes/datastream/DataStreamAggregate.scala --- @@ -207,7 +210,9 @@ class DataStreamAggregate( inputType, rowRelDataType, grouping, -namedProperties) +indicator, +namedProperties + ) --- End diff -- Same as here and a couple of other places > Add CUBE/ROLLUP/GROUPING SETS operator in SQL > - > > Key: FLINK-5303 > URL: https://issues.apache.org/jira/browse/FLINK-5303 > Project: Flink > Issue Type: New Feature > Components: Documentation, Table API & SQL >Reporter: Alexander Chermenin >Assignee: Alexander Chermenin > > Add support for such operators as CUBE, ROLLUP and GROUPING SETS in SQL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #2976: [FLINK-5303] [table] Support for SQL GROUPING SETS...
Github user KurtYoung commented on a diff in the pull request: https://github.com/apache/flink/pull/2976#discussion_r94104657 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/plan/nodes/datastream/DataStreamAggregate.scala --- @@ -207,7 +210,9 @@ class DataStreamAggregate( inputType, rowRelDataType, grouping, -namedProperties) +indicator, +namedProperties + ) --- End diff -- Same as here and a couple of other places --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2976: [FLINK-5303] [table] Support for SQL GROUPING SETS...
Github user KurtYoung commented on a diff in the pull request: https://github.com/apache/flink/pull/2976#discussion_r94104637 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/plan/nodes/datastream/DataStreamAggregate.scala --- @@ -70,7 +71,9 @@ class DataStreamAggregate( namedAggregates, getRowType, inputType, - grouping) + grouping, + indicator +) --- End diff -- No need to have new line here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2976: [FLINK-5303] [table] Support for SQL GROUPING SETS...
Github user KurtYoung commented on a diff in the pull request: https://github.com/apache/flink/pull/2976#discussion_r94104624 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/table/plan/nodes/dataset/DataSetAggregate.scala --- @@ -104,7 +107,9 @@ class DataSetAggregate( namedAggregates, inputType, rowRelDataType, - grouping) + grouping, + indicator +) --- End diff -- No need to have new line here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-5399) Add more information to checkpoint result of TriggerSavepointSuccess
shijinkui created FLINK-5399: Summary: Add more information to checkpoint result of TriggerSavepointSuccess Key: FLINK-5399 URL: https://issues.apache.org/jira/browse/FLINK-5399 Project: Flink Issue Type: Improvement Components: State Backends, Checkpointing Reporter: shijinkui Add checkpointId and triggerTime to TriggerSavepointSuccess We can record the history of trigger checkpoint out of Flink System. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-5398) Exclude generated files in module flink-batch-connectors in license checking
Xiaogang Shi created FLINK-5398:
---
Summary: Exclude generated files in module flink-batch-connectors
in license checking
Key: FLINK-5398
URL: https://issues.apache.org/jira/browse/FLINK-5398
Project: Flink
Issue Type: Bug
Reporter: Xiaogang Shi
Now the master branch fails to execute {{mvn install}} due to unlicensed files
in the module flink-batch-connectors. We should exclude these generated files
in the pom file.
Unapproved licenses:
flink-batch-connectors/flink-avro/src/test/java/org/apache/flink/api/io/avro/generated/Address.java
flink-batch-connectors/flink-avro/src/test/java/org/apache/flink/api/io/avro/generated/Colors.java
flink-batch-connectors/flink-avro/src/test/java/org/apache/flink/api/io/avro/generated/Fixed16.java
flink-batch-connectors/flink-avro/src/test/java/org/apache/flink/api/io/avro/generated/User.java
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-5280) Extend TableSource to support nested data
[
https://issues.apache.org/jira/browse/FLINK-5280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784253#comment-15784253
]
ASF GitHub Bot commented on FLINK-5280:
---
Github user wuchong commented on a diff in the pull request:
https://github.com/apache/flink/pull/3039#discussion_r94095173
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/functions/utils/UserDefinedFunctionUtils.scala
---
@@ -268,23 +268,9 @@ object UserDefinedFunctionUtils {
def getFieldInfo(inputType: TypeInformation[_])
--- End diff --
I think we can refactor this. Provide a `getFieldInfo` static method, get
the field names from `getFieldInfo`, and then do additional check outside. What
do you think ?
> Extend TableSource to support nested data
> -
>
> Key: FLINK-5280
> URL: https://issues.apache.org/jira/browse/FLINK-5280
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.2.0
>Reporter: Fabian Hueske
>Assignee: Ivan Mushketyk
>
> The {{TableSource}} interface does currently only support the definition of
> flat rows.
> However, there are several storage formats for nested data that should be
> supported such as Avro, Json, Parquet, and Orc. The Table API and SQL can
> also natively handle nested rows.
> The {{TableSource}} interface and the code to register table sources in
> Calcite's schema need to be extended to support nested data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request #3039: [FLINK-5280] Update TableSource to support nested ...
Github user wuchong commented on a diff in the pull request:
https://github.com/apache/flink/pull/3039#discussion_r94095173
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/functions/utils/UserDefinedFunctionUtils.scala
---
@@ -268,23 +268,9 @@ object UserDefinedFunctionUtils {
def getFieldInfo(inputType: TypeInformation[_])
--- End diff --
I think we can refactor this. Provide a `getFieldInfo` static method, get
the field names from `getFieldInfo`, and then do additional check outside. What
do you think ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3039: [FLINK-5280] Update TableSource to support nested ...
Github user mushketyk commented on a diff in the pull request:
https://github.com/apache/flink/pull/3039#discussion_r94087339
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/functions/utils/UserDefinedFunctionUtils.scala
---
@@ -268,23 +268,9 @@ object UserDefinedFunctionUtils {
def getFieldInfo(inputType: TypeInformation[_])
--- End diff --
`getFieldInfo` is overridden in `StreamTableEnvironment`, so I cannot make
it a static method.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-5280) Extend TableSource to support nested data
[
https://issues.apache.org/jira/browse/FLINK-5280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783939#comment-15783939
]
ASF GitHub Bot commented on FLINK-5280:
---
Github user mushketyk commented on a diff in the pull request:
https://github.com/apache/flink/pull/3039#discussion_r94087339
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/functions/utils/UserDefinedFunctionUtils.scala
---
@@ -268,23 +268,9 @@ object UserDefinedFunctionUtils {
def getFieldInfo(inputType: TypeInformation[_])
--- End diff --
`getFieldInfo` is overridden in `StreamTableEnvironment`, so I cannot make
it a static method.
> Extend TableSource to support nested data
> -
>
> Key: FLINK-5280
> URL: https://issues.apache.org/jira/browse/FLINK-5280
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.2.0
>Reporter: Fabian Hueske
>Assignee: Ivan Mushketyk
>
> The {{TableSource}} interface does currently only support the definition of
> flat rows.
> However, there are several storage formats for nested data that should be
> supported such as Avro, Json, Parquet, and Orc. The Table API and SQL can
> also natively handle nested rows.
> The {{TableSource}} interface and the code to register table sources in
> Calcite's schema need to be extended to support nested data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-5397) Fail to deserialize savepoints in v1.1 when there exist missing fields in class serialization descriptors
[
https://issues.apache.org/jira/browse/FLINK-5397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15782811#comment-15782811
]
Xiaogang Shi commented on FLINK-5397:
-
The idea does work and is better. Very thanks for the quick fix :)
> Fail to deserialize savepoints in v1.1 when there exist missing fields in
> class serialization descriptors
> -
>
> Key: FLINK-5397
> URL: https://issues.apache.org/jira/browse/FLINK-5397
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Reporter: Xiaogang Shi
>Assignee: Stefan Richter
>
> To restore from the savepoints in previous versions, Flink now keeps all
> classes whose serialization is changed and put them in a separated package
> ("migration").
> When deserializing the old savepoints, flink will look up correct descriptors
> ({{ObjectStreamClass}}) for these classes, without using those ones written
> in serialized data. The implementation however is problematic when there
> exist missing field descriptors in the serialized data.
> When serializing an object, Java will only write the descriptors of those
> non-null fields. But when we look up class descriptors with given classes,
> all fields will be put into the descriptors. As a result, we will deserialize
> the savepoints with incorrect descriptors, leading to serialization
> exceptions.
> A simple resolution is to update the name of read descriptors using
> Reflections, without using different descriptors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-5397) Fail to deserialize savepoints in v1.1 when there exist missing fields in class serialization descriptors
[
https://issues.apache.org/jira/browse/FLINK-5397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15782747#comment-15782747
]
Stefan Richter commented on FLINK-5397:
---
Thanks for pointing out this problem. I opened a PR #3050 with a fix that does
not require overriding strings by reflection. The idea is to simply replace the
class only during {{resolveClass(...)}}.
> Fail to deserialize savepoints in v1.1 when there exist missing fields in
> class serialization descriptors
> -
>
> Key: FLINK-5397
> URL: https://issues.apache.org/jira/browse/FLINK-5397
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Reporter: Xiaogang Shi
>Assignee: Stefan Richter
>
> To restore from the savepoints in previous versions, Flink now keeps all
> classes whose serialization is changed and put them in a separated package
> ("migration").
> When deserializing the old savepoints, flink will look up correct descriptors
> ({{ObjectStreamClass}}) for these classes, without using those ones written
> in serialized data. The implementation however is problematic when there
> exist missing field descriptors in the serialized data.
> When serializing an object, Java will only write the descriptors of those
> non-null fields. But when we look up class descriptors with given classes,
> all fields will be put into the descriptors. As a result, we will deserialize
> the savepoints with incorrect descriptors, leading to serialization
> exceptions.
> A simple resolution is to update the name of read descriptors using
> Reflections, without using different descriptors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (FLINK-5397) Fail to deserialize savepoints in v1.1 when there exist missing fields in class serialization descriptors
[
https://issues.apache.org/jira/browse/FLINK-5397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Richter reassigned FLINK-5397:
-
Assignee: Stefan Richter
> Fail to deserialize savepoints in v1.1 when there exist missing fields in
> class serialization descriptors
> -
>
> Key: FLINK-5397
> URL: https://issues.apache.org/jira/browse/FLINK-5397
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Reporter: Xiaogang Shi
>Assignee: Stefan Richter
>
> To restore from the savepoints in previous versions, Flink now keeps all
> classes whose serialization is changed and put them in a separated package
> ("migration").
> When deserializing the old savepoints, flink will look up correct descriptors
> ({{ObjectStreamClass}}) for these classes, without using those ones written
> in serialized data. The implementation however is problematic when there
> exist missing field descriptors in the serialized data.
> When serializing an object, Java will only write the descriptors of those
> non-null fields. But when we look up class descriptors with given classes,
> all fields will be put into the descriptors. As a result, we will deserialize
> the savepoints with incorrect descriptors, leading to serialization
> exceptions.
> A simple resolution is to update the name of read descriptors using
> Reflections, without using different descriptors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (FLINK-4641) Support branching CEP patterns
[
https://issues.apache.org/jira/browse/FLINK-4641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Chermenin reassigned FLINK-4641:
--
Assignee: Alexander Chermenin
> Support branching CEP patterns
> ---
>
> Key: FLINK-4641
> URL: https://issues.apache.org/jira/browse/FLINK-4641
> Project: Flink
> Issue Type: Improvement
> Components: CEP
>Reporter: Till Rohrmann
>Assignee: Alexander Chermenin
>
> We should add support for branching CEP patterns to the Pattern API.
> {code}
> |--> B --|
> ||
> A -- --> D
> ||
> |--> C --|
> {code}
> This feature will require changes to the {{Pattern}} class and the
> {{NFACompiler}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (FLINK-4540) Detached job execution may prevent cluster shutdown
[
https://issues.apache.org/jira/browse/FLINK-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Maier reassigned FLINK-4540:
--
Assignee: Roman Maier
> Detached job execution may prevent cluster shutdown
> ---
>
> Key: FLINK-4540
> URL: https://issues.apache.org/jira/browse/FLINK-4540
> Project: Flink
> Issue Type: Bug
> Components: YARN
>Affects Versions: 1.2.0, 1.1.2
>Reporter: Maximilian Michels
>Assignee: Roman Maier
>Priority: Minor
> Fix For: 1.2.0
>
>
> There is a problem with the detached execution of jobs. This can prevent
> cluster shutdown 1) when eager jobs are executed, i.e. the job calls
> `collect()/count()`, and 2) when the user jar doesn't contain a job.
> 1) For example, {{./flink -d -m yarn-cluster -yn 1
> ../examples/batch/WordCount.jar}} will throw an exception and only disconnect
> the YarnClusterClient afterwards. In detached mode, the code assumes the
> cluster is shutdown through the {{shutdownAfterJob}} method which ensures
> that the YarnJobManager shuts down after the job completes. Due to the
> exception thrown when executing eager jobs, the jobmanager never receives a
> job and thus never shuts down the cluster.
> 2) The same problem also occurs in detached execution when the user jar
> doesn't contain a job.
> A good solution would be to defer cluster startup until the job has been
> fully assembled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLINK-5397) Fail to deserialize savepoints in v1.1 when there exist missing fields in class serialization descriptors
[
https://issues.apache.org/jira/browse/FLINK-5397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaogang Shi updated FLINK-5397:
Description:
To restore from the savepoints in previous versions, Flink now keeps all
classes whose serialization is changed and put them in a separated package
("migration").
When deserializing the old savepoints, flink will look up correct descriptors
({{ObjectStreamClass}}) for these classes, without using those ones written in
serialized data. The implementation however is problematic when there exist
missing field descriptors in the serialized data.
When serializing an object, Java will only write the descriptors of those
non-null fields. But when we look up class descriptors with given classes, all
fields will be put into the descriptors. As a result, we will deserialize the
savepoints with incorrect descriptors, leading to serialization exceptions.
A simple resolution is to update the name of read descriptors using
Reflections, without using different descriptors.
was:
To restore from the savepoints in previous versions, Flink now keeps all
classes whose serialization is changed and put them in a separated package
("migration").
When deserializing the old savepoints, flink will look up correct descriptors
({{ObjectStreamClass}}) for these classes, without using those ones written in
serialized data. The implementation however is problematic when there exist
missing field descriptors in the serialized data.
When deserializing an object, Java will only write the descriptors of those
non-null fields. But when we look up class descriptors with given classes, all
fields will be put into the descriptors. As a result, we will deserialize the
savepoints with incorrect descriptors, leading to serialization exceptions.
A simple resolution is to update the name of read descriptors using
Reflections, without using a different descriptors.
> Fail to deserialize savepoints in v1.1 when there exist missing fields in
> class serialization descriptors
> -
>
> Key: FLINK-5397
> URL: https://issues.apache.org/jira/browse/FLINK-5397
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Reporter: Xiaogang Shi
>
> To restore from the savepoints in previous versions, Flink now keeps all
> classes whose serialization is changed and put them in a separated package
> ("migration").
> When deserializing the old savepoints, flink will look up correct descriptors
> ({{ObjectStreamClass}}) for these classes, without using those ones written
> in serialized data. The implementation however is problematic when there
> exist missing field descriptors in the serialized data.
> When serializing an object, Java will only write the descriptors of those
> non-null fields. But when we look up class descriptors with given classes,
> all fields will be put into the descriptors. As a result, we will deserialize
> the savepoints with incorrect descriptors, leading to serialization
> exceptions.
> A simple resolution is to update the name of read descriptors using
> Reflections, without using different descriptors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-5397) Fail to deserialize savepoints in v1.1 when there exist missing fields in class serialization descriptors
Xiaogang Shi created FLINK-5397:
---
Summary: Fail to deserialize savepoints in v1.1 when there exist
missing fields in class serialization descriptors
Key: FLINK-5397
URL: https://issues.apache.org/jira/browse/FLINK-5397
Project: Flink
Issue Type: Bug
Components: State Backends, Checkpointing
Reporter: Xiaogang Shi
To restore from the savepoints in previous versions, Flink now keeps all
classes whose serialization is changed and put them in a separated package
("migration").
When deserializing the old savepoints, flink will look up correct descriptors
({{ObjectStreamClass}}) for these classes, without using those ones written in
serialized data. The implementation however is problematic when there exist
missing field descriptors in the serialized data.
When deserializing an object, Java will only write the descriptors of those
non-null fields. But when we look up class descriptors with given classes, all
fields will be put into the descriptors. As a result, we will deserialize the
savepoints with incorrect descriptors, leading to serialization exceptions.
A simple resolution is to update the name of read descriptors using
Reflections, without using a different descriptors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3615) Add support for non-native SQL types
[
https://issues.apache.org/jira/browse/FLINK-3615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15782410#comment-15782410
]
Alexander Chermenin commented on FLINK-3615:
Hi all. Is it an actual issue or it has been solved in FLINK-3916?
> Add support for non-native SQL types
>
>
> Key: FLINK-3615
> URL: https://issues.apache.org/jira/browse/FLINK-3615
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Reporter: Vasia Kalavri
>
> The TypeConverter of the Table API currently only supports basic types. We
> should maybe re-design the way {{sqlTypeToTypeInfo}} works. It is used in the
> {{CodeGenerator}} for visiting literals, in {{DataSetAggregate}} to create
> the {{RowTypeInfo}} and in {{determineReturnType}}. We could maybe provide a
> custom implementation per operator to determine the return type, based on the
> input fields.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
