[jira] [Commented] (FLINK-8969) Move TimerService into state backend
[ https://issues.apache.org/jira/browse/FLINK-8969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401485#comment-16401485 ] Sihua Zhou commented on FLINK-8969: --- Hi [~phoenixjiangnan], Where is the previous discussion, could you please give a link or paste the content here? > Move TimerService into state backend > > > Key: FLINK-8969 > URL: https://issues.apache.org/jira/browse/FLINK-8969 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Major > > upon discussion with [~aljoscha]. More details need to be added here -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (FLINK-8364) Add iterator() to ListState which returns empty iterator when it has no value
[ https://issues.apache.org/jira/browse/FLINK-8364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li closed FLINK-8364. --- Resolution: Invalid > Add iterator() to ListState which returns empty iterator when it has no value > - > > Key: FLINK-8364 > URL: https://issues.apache.org/jira/browse/FLINK-8364 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing >Affects Versions: 1.4.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Major > Fix For: 1.5.0 > > > Add iterator() to ListState which returns empty iterator when it has no value -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-8771) Upgrade scalastyle to 1.0.0
[ https://issues.apache.org/jira/browse/FLINK-8771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li updated FLINK-8771: Affects Version/s: 1.5.0 > Upgrade scalastyle to 1.0.0 > --- > > Key: FLINK-8771 > URL: https://issues.apache.org/jira/browse/FLINK-8771 > Project: Flink > Issue Type: Improvement > Components: Build System >Affects Versions: 1.5.0 >Reporter: Ted Yu >Assignee: Bowen Li >Priority: Major > Fix For: 1.6.0 > > > scalastyle 1.0.0 fixes issue with import order, explicit type for public > methods, line length limitation and comment validation. > We should upgrade to scalastyle 1.0.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-8771) Upgrade scalastyle to 1.0.0
[ https://issues.apache.org/jira/browse/FLINK-8771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li updated FLINK-8771: Fix Version/s: 1.6.0 > Upgrade scalastyle to 1.0.0 > --- > > Key: FLINK-8771 > URL: https://issues.apache.org/jira/browse/FLINK-8771 > Project: Flink > Issue Type: Improvement > Components: Build System >Affects Versions: 1.5.0 >Reporter: Ted Yu >Assignee: Bowen Li >Priority: Major > Fix For: 1.6.0 > > > scalastyle 1.0.0 fixes issue with import order, explicit type for public > methods, line length limitation and comment validation. > We should upgrade to scalastyle 1.0.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-6968) Store streaming, updating tables with unique key in queryable state
[
https://issues.apache.org/jira/browse/FLINK-6968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401424#comment-16401424
]
ASF GitHub Bot commented on FLINK-6968:
---
Github user liurenjie1024 commented on a diff in the pull request:
https://github.com/apache/flink/pull/5688#discussion_r174990348
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/sinks/QueryableTableSink.scala
---
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.sinks
+
+import java.lang.{Boolean => JBool}
+
+import org.apache.flink.api.common.state.{ValueState, ValueStateDescriptor}
+import org.apache.flink.api.common.typeinfo.TypeInformation
+import org.apache.flink.api.java.functions.KeySelector
+import org.apache.flink.api.java.tuple.{Tuple2 => JTuple2}
+import org.apache.flink.api.java.typeutils.{ResultTypeQueryable,
RowTypeInfo}
+import org.apache.flink.configuration.Configuration
+import org.apache.flink.streaming.api.datastream.DataStream
+import org.apache.flink.streaming.api.functions.ProcessFunction
+import org.apache.flink.table.api.StreamQueryConfig
+import
org.apache.flink.table.runtime.aggregate.ProcessFunctionWithCleanupState
+import org.apache.flink.types.Row
+import org.apache.flink.util.Collector
+import org.slf4j.LoggerFactory
+
+class QueryableTableSink(
+ private val namePrefix: String,
--- End diff --
Fixed.
> Store streaming, updating tables with unique key in queryable state
> ---
>
> Key: FLINK-6968
> URL: https://issues.apache.org/jira/browse/FLINK-6968
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Reporter: Fabian Hueske
>Assignee: Renjie Liu
>Priority: Major
>
> Streaming tables with unique key are continuously updated. For example
> queries with a non-windowed aggregation generate such tables. Commonly, such
> updating tables are emitted via an upsert table sink to an external datastore
> (k-v store, database) to make it accessible to applications.
> This issue is about adding a feature to store and maintain such a table as
> queryable state in Flink. By storing the table in Flnk's queryable state, we
> do not need an external data store to access the results of the query but can
> query the results directly from Flink.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink pull request #5688: [FLINK-6968][Table API & SQL] Add Queryable table ...
Github user liurenjie1024 commented on a diff in the pull request:
https://github.com/apache/flink/pull/5688#discussion_r174990348
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/sinks/QueryableTableSink.scala
---
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.sinks
+
+import java.lang.{Boolean => JBool}
+
+import org.apache.flink.api.common.state.{ValueState, ValueStateDescriptor}
+import org.apache.flink.api.common.typeinfo.TypeInformation
+import org.apache.flink.api.java.functions.KeySelector
+import org.apache.flink.api.java.tuple.{Tuple2 => JTuple2}
+import org.apache.flink.api.java.typeutils.{ResultTypeQueryable,
RowTypeInfo}
+import org.apache.flink.configuration.Configuration
+import org.apache.flink.streaming.api.datastream.DataStream
+import org.apache.flink.streaming.api.functions.ProcessFunction
+import org.apache.flink.table.api.StreamQueryConfig
+import
org.apache.flink.table.runtime.aggregate.ProcessFunctionWithCleanupState
+import org.apache.flink.types.Row
+import org.apache.flink.util.Collector
+import org.slf4j.LoggerFactory
+
+class QueryableTableSink(
+ private val namePrefix: String,
--- End diff --
Fixed.
---
[jira] [Commented] (FLINK-8930) TableApi validation test in ScalarFunctionsValidationTest doesn't work
[ https://issues.apache.org/jira/browse/FLINK-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401420#comment-16401420 ] Wind commented on FLINK-8930: - ok, I've already understood, thanks for the explanations ! > TableApi validation test in ScalarFunctionsValidationTest doesn't work > -- > > Key: FLINK-8930 > URL: https://issues.apache.org/jira/browse/FLINK-8930 > Project: Flink > Issue Type: Bug > Components: Table API & SQL >Reporter: Wind >Priority: Major > > I'm wring a validation test for > [FLINK-6924|https://issues.apache.org/jira/browse/FLINK-6924] in > org.apache.flink.table.expressions.validation.ScalarFunctionsValidationTest. > However, I find that the table api is not truely executed in function > "testTableApi", which is different from "testSqlApi". So we can only test > exceptions which are thrown in "addTableApiTestExpr" like > "ValidationException" because it is thrown during "select" operation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8953) Resolve unresolved field references in FieldComputer expressions
[
https://issues.apache.org/jira/browse/FLINK-8953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401391#comment-16401391
]
Renjie Liu commented on FLINK-8953:
---
Hi, Tim:
I want to look into this, could you assign this to me?
> Resolve unresolved field references in FieldComputer expressions
>
>
> Key: FLINK-8953
> URL: https://issues.apache.org/jira/browse/FLINK-8953
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Reporter: Timo Walther
>Priority: Major
>
> When implementing the {{FieldComputer.getExpression}} method, it is not
> possible to use API classes but only internal expression case classes.
> It would be great to also define timestamp extractors like:
> {code}
> def getExpression(fieldAccesses: Array[ResolvedFieldReference]): Expression
> = {
> // 'x.cast(Types.LONG)
> // ExpressionParser.parseExpression("x.cast(LONG)")
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink issue #5698: [FLINK-8945] [kinesis] Allow customization of KinesisProx...
Github user tweise commented on the issue: https://github.com/apache/flink/pull/5698 +1 @StephanEwen can we get this into 1.5.0 ? ---
[jira] [Commented] (FLINK-8945) Allow customization of the KinesisProxy
[ https://issues.apache.org/jira/browse/FLINK-8945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401380#comment-16401380 ] ASF GitHub Bot commented on FLINK-8945: --- Github user tweise commented on the issue: https://github.com/apache/flink/pull/5698 +1 @StephanEwen can we get this into 1.5.0 ? > Allow customization of the KinesisProxy > --- > > Key: FLINK-8945 > URL: https://issues.apache.org/jira/browse/FLINK-8945 > Project: Flink > Issue Type: Improvement >Reporter: Kailash Hassan Dayanand >Priority: Minor > > Currently the KinesisProxy interface here: > [https://github.com/apache/flink/blob/310f3de62e52f1f977c217d918cc5aac79b87277/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/proxy/KinesisProxy.java#L125] > has a private constructor. This restricts extending the class and prevents > customizations on shard discovery. I am proposing to change this to protected. > While the creating a new implementation of KinesisProxyInterface is possible, > I would like to continue to use implementation of getRecords and > getShardIterator. > This will be a temporary workaround till FLINK-8944 is submitted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8969) Move TimerService into state backend
Bowen Li created FLINK-8969: --- Summary: Move TimerService into state backend Key: FLINK-8969 URL: https://issues.apache.org/jira/browse/FLINK-8969 Project: Flink Issue Type: Improvement Components: State Backends, Checkpointing Reporter: Bowen Li Assignee: Bowen Li upon discussion with [~aljoscha]. More details need to be added here -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8915) CheckpointingStatisticsHandler fails to return PendingCheckpointStats
[
https://issues.apache.org/jira/browse/FLINK-8915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401356#comment-16401356
]
ASF GitHub Bot commented on FLINK-8915:
---
Github user yanghua commented on the issue:
https://github.com/apache/flink/pull/5703
@tillrohrmann please review this PR, thanks!
> CheckpointingStatisticsHandler fails to return PendingCheckpointStats
> --
>
> Key: FLINK-8915
> URL: https://issues.apache.org/jira/browse/FLINK-8915
> Project: Flink
> Issue Type: Bug
> Components: REST
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Gary Yao
>Assignee: vinoyang
>Priority: Blocker
> Labels: flip6
> Fix For: 1.5.0
>
>
> {noformat}
> 2018-03-10 21:47:52,487 ERROR
> org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler
> - Implementation error: Unhandled exception.
> java.lang.IllegalArgumentException: Given checkpoint stats object of type
> org.apache.flink.runtime.checkpoint.PendingCheckpointStats cannot be
> converted.
> at
> org.apache.flink.runtime.rest.messages.checkpoints.CheckpointStatistics.generateCheckpointStatistics(CheckpointStatistics.java:276)
> at
> org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleRequest(CheckpointingStatisticsHandler.java:146)
> at
> org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleRequest(CheckpointingStatisticsHandler.java:54)
> at
> org.apache.flink.runtime.rest.handler.job.AbstractExecutionGraphHandler.lambda$handleRequest$0(AbstractExecutionGraphHandler.java:81)
> at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink issue #5703: [FLINK-8915] CheckpointingStatisticsHandler fails to retu...
Github user yanghua commented on the issue: https://github.com/apache/flink/pull/5703 @tillrohrmann please review this PR, thanks! ---
[jira] [Commented] (FLINK-8903) Built-in agg functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP are broken in Group Windows
[
https://issues.apache.org/jira/browse/FLINK-8903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401281#comment-16401281
]
ASF GitHub Bot commented on FLINK-8903:
---
Github user suez1224 commented on a diff in the pull request:
https://github.com/apache/flink/pull/5706#discussion_r174962566
--- Diff:
flink-libraries/flink-table/src/main/java/org/apache/calcite/rel/rules/AggregateReduceFunctionsRule.java
---
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.RelFactories;
+import org.apache.calcite.rel.logical.LogicalAggregate;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexLiteral;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql.type.SqlTypeUtil;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.tools.RelBuilderFactory;
+import org.apache.calcite.util.CompositeList;
+import org.apache.calcite.util.ImmutableIntList;
+import org.apache.calcite.util.Util;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import java.math.BigDecimal;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+/*
+ * THIS FILE HAS BEEN COPIED FROM THE APACHE CALCITE PROJECT TO MAKE IT
MORE EXTENSIBLE.
--- End diff --
I think we should create a Calcite JIRA to allow
AggregateReduceFunctionsRule in Calcite to support this extension, document the
JIRA ticket here. And remove this overwrite once Calcite is upgraded. What do
you think?
> Built-in agg functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP are broken in
> Group Windows
> ---
>
> Key: FLINK-8903
> URL: https://issues.apache.org/jira/browse/FLINK-8903
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.3.2, 1.5.0, 1.4.2
>Reporter: lilizhao
>Assignee: Fabian Hueske
>Priority: Critical
> Fix For: 1.5.0
>
> Attachments: QQ图片20180312180143.jpg, TableAndSQLTest.java
>
>
> The built-in aggregation functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP
> are translated into regular AVG functions if they are applied in the context
> of a Group Window aggregation (\{{GROUP BY TUMBLE/HOP/SESSION}}).
> The reason is that these functions are internally represented as
> {{SqlAvgAggFunction}} but with different {{SqlKind}}. When translating
> Calcite aggregation functions to Flink Table agg functions, we only look at
> the type of the class, not at the value of the {{kind}} field. We did not
> notice that before, because in all other cases (regular {{GROUP BY}} without
> windows or {{OVER}} windows, we have a translation rule
> {{AggregateReduceFunctionsRule}} that decomposes the more complex functions
> into expressions of {{COUNT}} and {{SUM}} functions such that we never
> execute an {{AVG}} Flink function. That rule can only be applied on
> {{LogicalAggregate}}, however, we represent group windows as
> {{LogicalWindowAggregate}},
[jira] [Commented] (FLINK-8903) Built-in agg functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP are broken in Group Windows
[
https://issues.apache.org/jira/browse/FLINK-8903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401280#comment-16401280
]
ASF GitHub Bot commented on FLINK-8903:
---
Github user suez1224 commented on a diff in the pull request:
https://github.com/apache/flink/pull/5706#discussion_r174965171
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/plan/nodes/logical/FlinkLogicalWindowAggregate.scala
---
@@ -103,6 +106,19 @@ class FlinkLogicalWindowAggregateConverter
FlinkConventions.LOGICAL,
"FlinkLogicalWindowAggregateConverter") {
+ override def matches(call: RelOptRuleCall): Boolean = {
+val agg = call.rel(0).asInstanceOf[LogicalWindowAggregate]
+
+// we do not support these functions natively
+// they have to be converted using the
WindowAggregateReduceFunctionsRule
+val supported =
agg.getAggCallList.asScala.map(_.getAggregation.getKind).forall {
+ case SqlKind.STDDEV_POP | SqlKind.STDDEV_SAMP | SqlKind.VAR_POP |
SqlKind.VAR_SAMP => false
--- End diff --
How about AVG and SUM? they are also in AggregateReduceFunctionsRule. Also,
I think it's better to use SqlKind.AVG_AGG_FUNCTIONS.contains() or
AggregateReduceFunctionsRule.isReducible() (it's private now though) here in
the case statement. So it will keep consistent if calcite changes.
> Built-in agg functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP are broken in
> Group Windows
> ---
>
> Key: FLINK-8903
> URL: https://issues.apache.org/jira/browse/FLINK-8903
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.3.2, 1.5.0, 1.4.2
>Reporter: lilizhao
>Assignee: Fabian Hueske
>Priority: Critical
> Fix For: 1.5.0
>
> Attachments: QQ图片20180312180143.jpg, TableAndSQLTest.java
>
>
> The built-in aggregation functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP
> are translated into regular AVG functions if they are applied in the context
> of a Group Window aggregation (\{{GROUP BY TUMBLE/HOP/SESSION}}).
> The reason is that these functions are internally represented as
> {{SqlAvgAggFunction}} but with different {{SqlKind}}. When translating
> Calcite aggregation functions to Flink Table agg functions, we only look at
> the type of the class, not at the value of the {{kind}} field. We did not
> notice that before, because in all other cases (regular {{GROUP BY}} without
> windows or {{OVER}} windows, we have a translation rule
> {{AggregateReduceFunctionsRule}} that decomposes the more complex functions
> into expressions of {{COUNT}} and {{SUM}} functions such that we never
> execute an {{AVG}} Flink function. That rule can only be applied on
> {{LogicalAggregate}}, however, we represent group windows as
> {{LogicalWindowAggregate}}, so the rule does not match.
> We should fix this by:
> 1. restrict the translation to Flink avg functions in {{AggregateUtil}} to
> {{SqlKind.AVG}}.
> 2. implement a rule (hopefully based on {{AggregateReduceFunctionsRule}})
> that decomposes the complex agg functions into the {{SUM}} and {{COUNT}}.
> Step 1. is easy and a quick fix but we would get an exception "Unsupported
> Function" if {{VAR_POP}} is used in a {{GROUP BY}} window.
> Step 2. might be more involved, depending on how difficult it is to port the
> rule.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink pull request #5706: [FLINK-8903] [table] Fix VAR_SAMP, VAR_POP, STDEV_...
Github user suez1224 commented on a diff in the pull request: https://github.com/apache/flink/pull/5706#discussion_r174962566 --- Diff: flink-libraries/flink-table/src/main/java/org/apache/calcite/rel/rules/AggregateReduceFunctionsRule.java --- @@ -0,0 +1,590 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.calcite.rel.rules; + +import org.apache.calcite.plan.RelOptCluster; +import org.apache.calcite.plan.RelOptRule; +import org.apache.calcite.plan.RelOptRuleCall; +import org.apache.calcite.plan.RelOptRuleOperand; +import org.apache.calcite.rel.RelNode; +import org.apache.calcite.rel.core.Aggregate; +import org.apache.calcite.rel.core.AggregateCall; +import org.apache.calcite.rel.core.RelFactories; +import org.apache.calcite.rel.logical.LogicalAggregate; +import org.apache.calcite.rel.type.RelDataType; +import org.apache.calcite.rel.type.RelDataTypeFactory; +import org.apache.calcite.rel.type.RelDataTypeField; +import org.apache.calcite.rex.RexBuilder; +import org.apache.calcite.rex.RexLiteral; +import org.apache.calcite.rex.RexNode; +import org.apache.calcite.sql.SqlAggFunction; +import org.apache.calcite.sql.SqlKind; +import org.apache.calcite.sql.fun.SqlStdOperatorTable; +import org.apache.calcite.sql.type.SqlTypeUtil; +import org.apache.calcite.tools.RelBuilder; +import org.apache.calcite.tools.RelBuilderFactory; +import org.apache.calcite.util.CompositeList; +import org.apache.calcite.util.ImmutableIntList; +import org.apache.calcite.util.Util; + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.Lists; +import com.google.common.collect.Maps; + +import java.math.BigDecimal; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Map; + +/* + * THIS FILE HAS BEEN COPIED FROM THE APACHE CALCITE PROJECT TO MAKE IT MORE EXTENSIBLE. --- End diff -- I think we should create a Calcite JIRA to allow AggregateReduceFunctionsRule in Calcite to support this extension, document the JIRA ticket here. And remove this overwrite once Calcite is upgraded. What do you think? ---
[GitHub] flink pull request #5706: [FLINK-8903] [table] Fix VAR_SAMP, VAR_POP, STDEV_...
Github user suez1224 commented on a diff in the pull request:
https://github.com/apache/flink/pull/5706#discussion_r174965171
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/plan/nodes/logical/FlinkLogicalWindowAggregate.scala
---
@@ -103,6 +106,19 @@ class FlinkLogicalWindowAggregateConverter
FlinkConventions.LOGICAL,
"FlinkLogicalWindowAggregateConverter") {
+ override def matches(call: RelOptRuleCall): Boolean = {
+val agg = call.rel(0).asInstanceOf[LogicalWindowAggregate]
+
+// we do not support these functions natively
+// they have to be converted using the
WindowAggregateReduceFunctionsRule
+val supported =
agg.getAggCallList.asScala.map(_.getAggregation.getKind).forall {
+ case SqlKind.STDDEV_POP | SqlKind.STDDEV_SAMP | SqlKind.VAR_POP |
SqlKind.VAR_SAMP => false
--- End diff --
How about AVG and SUM? they are also in AggregateReduceFunctionsRule. Also,
I think it's better to use SqlKind.AVG_AGG_FUNCTIONS.contains() or
AggregateReduceFunctionsRule.isReducible() (it's private now though) here in
the case statement. So it will keep consistent if calcite changes.
---
[jira] [Commented] (FLINK-8903) Built-in agg functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP are broken in Group Windows
[
https://issues.apache.org/jira/browse/FLINK-8903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401160#comment-16401160
]
ASF GitHub Bot commented on FLINK-8903:
---
GitHub user fhueske opened a pull request:
https://github.com/apache/flink/pull/5706
[FLINK-8903] [table] Fix VAR_SAMP, VAR_POP, STDEV_SAMP, STDEV_POP functions
on GROUP BY windows.
## What is the purpose of the change
* Fixes the computation of `VAR_SAMP`, `VAR_POP`, `STDDEV_SAMP`,
`STDDEV_POP` aggregations in the context of `GROUP BY` windows (`TUMBLE`,
`HOP`, `SESSION`). Right now, these methods are computed as `AVG`.
## Brief change log
* copy Calcite's `AggregateReduceFunctionsRule` to Flink and improve its
extensibility
* add a `WindowAggregateReduceFunctionsRule` based on the copied
`AggregateReduceFunctionsRule` to decompose the faulty aggregation functions
into `COUNT` and `SUM` functions.
* add restriction to `FlinkLogicalWindowAggregateConverter` to prevent
translation of group window aggregates with failing aggregation functions
* prevent translation of `VAR_SAMP`, `VAR_POP`, `STDDEV_SAMP`, `STDDEV_POP`
in `AggregateUtil`
* add unit tests (plan validation) for batch (SQL, Table API) and stream
(SQL, Table API)
## Verifying this change
* run the added plan tests
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): **no**
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: **no**
- The serializers: **no**
- The runtime per-record code paths (performance sensitive): **no**
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: **no**
- The S3 file system connector: **no**
## Documentation
- Does this pull request introduce a new feature? **no**
- If yes, how is the feature documented? **n/a**
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/fhueske/flink tableVarStddevAggFix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5706.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5706
commit 517567348b0ec0c23ef0c1dcc05c54a91d5c5671
Author: Fabian Hueske
Date: 2018-03-15T20:04:00Z
[FLINK-8903] [table] Fix VAR_SAMP, VAR_POP, STDEV_SAMP, STDEV_POP functions
on GROUP BY windows.
> Built-in agg functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP are broken in
> Group Windows
> ---
>
> Key: FLINK-8903
> URL: https://issues.apache.org/jira/browse/FLINK-8903
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.3.2, 1.5.0, 1.4.2
>Reporter: lilizhao
>Assignee: Fabian Hueske
>Priority: Critical
> Fix For: 1.5.0
>
> Attachments: QQ图片20180312180143.jpg, TableAndSQLTest.java
>
>
> The built-in aggregation functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP
> are translated into regular AVG functions if they are applied in the context
> of a Group Window aggregation (\{{GROUP BY TUMBLE/HOP/SESSION}}).
> The reason is that these functions are internally represented as
> {{SqlAvgAggFunction}} but with different {{SqlKind}}. When translating
> Calcite aggregation functions to Flink Table agg functions, we only look at
> the type of the class, not at the value of the {{kind}} field. We did not
> notice that before, because in all other cases (regular {{GROUP BY}} without
> windows or {{OVER}} windows, we have a translation rule
> {{AggregateReduceFunctionsRule}} that decomposes the more complex functions
> into expressions of {{COUNT}} and {{SUM}} functions such that we never
> execute an {{AVG}} Flink function. That rule can only be applied on
> {{LogicalAggregate}}, however, we represent group windows as
> {{LogicalWindowAggregate}}, so the rule does not match.
> We should fix this by:
> 1. restrict the translation to Flink avg functions in {{AggregateUtil}} to
> {{SqlKind.AVG}}.
> 2. implement a rule (hopefully based on {{AggregateReduceFunctionsRule}})
> that decomposes the complex agg functions into the {{SUM}} and {{COUNT}}.
> Step 1. is easy and a quick fix but we would get an exception "Unsupported
> Function" if {{VAR_POP}} is used in a {{GROUP BY}} window.
> Step 2. might be more involved, depending on how difficult it is to port the
> rule.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink pull request #5706: [FLINK-8903] [table] Fix VAR_SAMP, VAR_POP, STDEV_...
GitHub user fhueske opened a pull request: https://github.com/apache/flink/pull/5706 [FLINK-8903] [table] Fix VAR_SAMP, VAR_POP, STDEV_SAMP, STDEV_POP functions on GROUP BY windows. ## What is the purpose of the change * Fixes the computation of `VAR_SAMP`, `VAR_POP`, `STDDEV_SAMP`, `STDDEV_POP` aggregations in the context of `GROUP BY` windows (`TUMBLE`, `HOP`, `SESSION`). Right now, these methods are computed as `AVG`. ## Brief change log * copy Calcite's `AggregateReduceFunctionsRule` to Flink and improve its extensibility * add a `WindowAggregateReduceFunctionsRule` based on the copied `AggregateReduceFunctionsRule` to decompose the faulty aggregation functions into `COUNT` and `SUM` functions. * add restriction to `FlinkLogicalWindowAggregateConverter` to prevent translation of group window aggregates with failing aggregation functions * prevent translation of `VAR_SAMP`, `VAR_POP`, `STDDEV_SAMP`, `STDDEV_POP` in `AggregateUtil` * add unit tests (plan validation) for batch (SQL, Table API) and stream (SQL, Table API) ## Verifying this change * run the added plan tests ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): **no** - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no** - The serializers: **no** - The runtime per-record code paths (performance sensitive): **no** - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: **no** - The S3 file system connector: **no** ## Documentation - Does this pull request introduce a new feature? **no** - If yes, how is the feature documented? **n/a** You can merge this pull request into a Git repository by running: $ git pull https://github.com/fhueske/flink tableVarStddevAggFix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5706.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5706 commit 517567348b0ec0c23ef0c1dcc05c54a91d5c5671 Author: Fabian Hueske Date: 2018-03-15T20:04:00Z [FLINK-8903] [table] Fix VAR_SAMP, VAR_POP, STDEV_SAMP, STDEV_POP functions on GROUP BY windows. ---
[jira] [Assigned] (FLINK-8903) Built-in agg functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP are broken in Group Windows
[
https://issues.apache.org/jira/browse/FLINK-8903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fabian Hueske reassigned FLINK-8903:
Assignee: Fabian Hueske (was: Shuyi Chen)
> Built-in agg functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP are broken in
> Group Windows
> ---
>
> Key: FLINK-8903
> URL: https://issues.apache.org/jira/browse/FLINK-8903
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.3.2, 1.5.0, 1.4.2
>Reporter: lilizhao
>Assignee: Fabian Hueske
>Priority: Critical
> Fix For: 1.5.0
>
> Attachments: QQ图片20180312180143.jpg, TableAndSQLTest.java
>
>
> The built-in aggregation functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP
> are translated into regular AVG functions if they are applied in the context
> of a Group Window aggregation (\{{GROUP BY TUMBLE/HOP/SESSION}}).
> The reason is that these functions are internally represented as
> {{SqlAvgAggFunction}} but with different {{SqlKind}}. When translating
> Calcite aggregation functions to Flink Table agg functions, we only look at
> the type of the class, not at the value of the {{kind}} field. We did not
> notice that before, because in all other cases (regular {{GROUP BY}} without
> windows or {{OVER}} windows, we have a translation rule
> {{AggregateReduceFunctionsRule}} that decomposes the more complex functions
> into expressions of {{COUNT}} and {{SUM}} functions such that we never
> execute an {{AVG}} Flink function. That rule can only be applied on
> {{LogicalAggregate}}, however, we represent group windows as
> {{LogicalWindowAggregate}}, so the rule does not match.
> We should fix this by:
> 1. restrict the translation to Flink avg functions in {{AggregateUtil}} to
> {{SqlKind.AVG}}.
> 2. implement a rule (hopefully based on {{AggregateReduceFunctionsRule}})
> that decomposes the complex agg functions into the {{SUM}} and {{COUNT}}.
> Step 1. is easy and a quick fix but we would get an exception "Unsupported
> Function" if {{VAR_POP}} is used in a {{GROUP BY}} window.
> Step 2. might be more involved, depending on how difficult it is to port the
> rule.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8888) Upgrade AWS SDK in flink-connector-kinesis
[ https://issues.apache.org/jira/browse/FLINK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401057#comment-16401057 ] ASF GitHub Bot commented on FLINK-: --- Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/5663 Travis CI pending after rebasing this, will push as soon at it gives us a green light... > Upgrade AWS SDK in flink-connector-kinesis > -- > > Key: FLINK- > URL: https://issues.apache.org/jira/browse/FLINK- > Project: Flink > Issue Type: Improvement >Reporter: Kailash Hassan Dayanand >Priority: Minor > > Bump up the java aws sdk version to 1.11.272. Evaluate also the impact of > this version upgrade for KCL and KPL versions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] flink issue #5663: [FLINK-8888] [Kinesis Connectors] Update the AWS SDK for ...
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/5663 Travis CI pending after rebasing this, will push as soon at it gives us a green light... ---
[jira] [Commented] (FLINK-8304) Document Kubernetes and Flink HA setup
[
https://issues.apache.org/jira/browse/FLINK-8304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401026#comment-16401026
]
Ufuk Celebi commented on FLINK-8304:
I think your understanding is correct (although the terminology is slightly
off, not the Kubernetes master will restart a failed Pod, but the corresponding
Deployment). In general, there is no fundamental difference to HA with YARN.
By default, Flink in HA mode will pick a random port for JobManager RPC before
registering the JobManager address (as configured in
{{jobmanager.rpc.address}}) at ZooKeeper. Typically, you have a Kubernetes
Service in front of your JobManager Deployment/Pod (in order to get a fixed
IP/hostname despite potential Pod restarts) which only exposes a fixed set of
ports. Therefore, you need to configure a fixed port for HA as well via the key
I've mentioned above ({{high-availability.jobmanager.port}}).
(I think there are plans to consolidate this config option with the already
existing regular {{jobmanager.rpc.port}}.)
> Document Kubernetes and Flink HA setup
> --
>
> Key: FLINK-8304
> URL: https://issues.apache.org/jira/browse/FLINK-8304
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Ufuk Celebi
>Priority: Major
>
> Currently the Flink on Kubernetes documentation does not mention anything
> about running Flink in HA mode.
> We should add at least the following two things:
> - Currently, there cannot be a standby JobManager pod due to the way Flink HA
> works
> - `high-availability.jobmanager.port` has to be set to a port that is exposed
> via Kubernetes
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (FLINK-7283) PythonPlanBinderTest issues with python paths
[
https://issues.apache.org/jira/browse/FLINK-7283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tzu-Li (Gordon) Tai closed FLINK-7283.
--
Resolution: Fixed
> PythonPlanBinderTest issues with python paths
> -
>
> Key: FLINK-7283
> URL: https://issues.apache.org/jira/browse/FLINK-7283
> Project: Flink
> Issue Type: Bug
> Components: Python API, Tests
>Affects Versions: 1.3.0, 1.3.1, 1.4.0, 1.3.2
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Major
> Fix For: 1.3.3, 1.4.0
>
>
> There are some issues with {{PythonPlanBinderTest}} and the Python2/3 tests:
> - the path is not set correctly (only inside {{config}}, not the
> {{configuration}} that is passed on to the {{PythonPlanBinder}}
> - linux distributions have become quite inventive regarding python binary
> names: some offer {{python}} as Python 2, some as Python 3. Similarly,
> {{python3}} and/or {{python2}} may not be available. If we really want to
> test both, we need to take this into account.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (FLINK-7283) PythonPlanBinderTest issues with python paths
[
https://issues.apache.org/jira/browse/FLINK-7283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tzu-Li (Gordon) Tai reopened FLINK-7283:
> PythonPlanBinderTest issues with python paths
> -
>
> Key: FLINK-7283
> URL: https://issues.apache.org/jira/browse/FLINK-7283
> Project: Flink
> Issue Type: Bug
> Components: Python API, Tests
>Affects Versions: 1.3.0, 1.3.1, 1.4.0, 1.3.2
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Major
> Fix For: 1.4.0, 1.3.3
>
>
> There are some issues with {{PythonPlanBinderTest}} and the Python2/3 tests:
> - the path is not set correctly (only inside {{config}}, not the
> {{configuration}} that is passed on to the {{PythonPlanBinder}}
> - linux distributions have become quite inventive regarding python binary
> names: some offer {{python}} as Python 2, some as Python 3. Similarly,
> {{python3}} and/or {{python2}} may not be available. If we really want to
> test both, we need to take this into account.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-7283) PythonPlanBinderTest issues with python paths
[
https://issues.apache.org/jira/browse/FLINK-7283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tzu-Li (Gordon) Tai updated FLINK-7283:
---
Fix Version/s: (was: 1.3.4)
1.3.3
> PythonPlanBinderTest issues with python paths
> -
>
> Key: FLINK-7283
> URL: https://issues.apache.org/jira/browse/FLINK-7283
> Project: Flink
> Issue Type: Bug
> Components: Python API, Tests
>Affects Versions: 1.3.0, 1.3.1, 1.4.0, 1.3.2
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Major
> Fix For: 1.4.0, 1.3.3
>
>
> There are some issues with {{PythonPlanBinderTest}} and the Python2/3 tests:
> - the path is not set correctly (only inside {{config}}, not the
> {{configuration}} that is passed on to the {{PythonPlanBinder}}
> - linux distributions have become quite inventive regarding python binary
> names: some offer {{python}} as Python 2, some as Python 3. Similarly,
> {{python3}} and/or {{python2}} may not be available. If we really want to
> test both, we need to take this into account.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-7282) Credit-based Network Flow Control
[ https://issues.apache.org/jira/browse/FLINK-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen updated FLINK-7282: External issue URL: https://docs.google.com/document/d/1chTOuOqe0sBsjldA_r-wXYeSIhU2zRGpUaTaik7QZ84 (was: https://docs.google.com/document/d/1chTOuOqe0sBsjldA_r-wXYeSIhU2zRGpUaTaik7QZ84/edit#) > Credit-based Network Flow Control > - > > Key: FLINK-7282 > URL: https://issues.apache.org/jira/browse/FLINK-7282 > Project: Flink > Issue Type: New Feature > Components: Network >Reporter: zhijiang >Assignee: zhijiang >Priority: Major > > This is a part of work for network stack improvements proposed in > [~StephanEwen] 's > [FLIP|https://docs.google.com/document/d/1chTOuOqe0sBsjldA_r-wXYeSIhU2zRGpUaTaik7QZ84/edit#] > Backpressure currently happens very naturally through the TCP network > connections and the bounded buffering capacity. The downsides are : > * All channels multiplexed into the same TCP connection stall together, as > soon as one channel has backpressure. > * Under backpressure, connections can not transport checkpoint barriers. > This flink-managed flow control is similar to the window-based advertisement > mechanism in TCP. The basic approaches are the following: > * Each RemoteInputChannel has fixed exclusive buffers as initial credits, and > SingleInputGate has a fixed buffer pool for managing floating buffers for all > RemoteInputChannels. > * RemoteInputChannel as receiver notifies the current available credits to > the sender side. > * Senders must never send buffers without credit, that means all the buffers > sent must be accepted by receivers so no buffers accumulated on the network > wire. > * Senders also send the current size of backlog that indicates how many > buffers are available on the sender side. The receivers use this information > to decide how to request floating buffers from the fixed buffer pool. > To avoid immediate commits affecting master branch, it will be implemented > into a separate feature branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8903) Built-in agg functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP are broken in Group Windows
[
https://issues.apache.org/jira/browse/FLINK-8903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400990#comment-16400990
]
Shuyi Chen commented on FLINK-8903:
---
Hi [~fhueske], please work on it if you are already in the middle. I am just
interested as well, and thought you might be busy. The way how the
VolcanoPlanner pick the plan might be tricky AFAIR. Let me know if I can help.
> Built-in agg functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP are broken in
> Group Windows
> ---
>
> Key: FLINK-8903
> URL: https://issues.apache.org/jira/browse/FLINK-8903
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.3.2, 1.5.0, 1.4.2
>Reporter: lilizhao
>Assignee: Shuyi Chen
>Priority: Critical
> Fix For: 1.5.0
>
> Attachments: QQ图片20180312180143.jpg, TableAndSQLTest.java
>
>
> The built-in aggregation functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP
> are translated into regular AVG functions if they are applied in the context
> of a Group Window aggregation (\{{GROUP BY TUMBLE/HOP/SESSION}}).
> The reason is that these functions are internally represented as
> {{SqlAvgAggFunction}} but with different {{SqlKind}}. When translating
> Calcite aggregation functions to Flink Table agg functions, we only look at
> the type of the class, not at the value of the {{kind}} field. We did not
> notice that before, because in all other cases (regular {{GROUP BY}} without
> windows or {{OVER}} windows, we have a translation rule
> {{AggregateReduceFunctionsRule}} that decomposes the more complex functions
> into expressions of {{COUNT}} and {{SUM}} functions such that we never
> execute an {{AVG}} Flink function. That rule can only be applied on
> {{LogicalAggregate}}, however, we represent group windows as
> {{LogicalWindowAggregate}}, so the rule does not match.
> We should fix this by:
> 1. restrict the translation to Flink avg functions in {{AggregateUtil}} to
> {{SqlKind.AVG}}.
> 2. implement a rule (hopefully based on {{AggregateReduceFunctionsRule}})
> that decomposes the complex agg functions into the {{SUM}} and {{COUNT}}.
> Step 1. is easy and a quick fix but we would get an exception "Unsupported
> Function" if {{VAR_POP}} is used in a {{GROUP BY}} window.
> Step 2. might be more involved, depending on how difficult it is to port the
> rule.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8903) Built-in agg functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP are broken in Group Windows
[
https://issues.apache.org/jira/browse/FLINK-8903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400963#comment-16400963
]
Fabian Hueske commented on FLINK-8903:
--
Thanks Shuyi!
In fact, I was curious how hard it would be to implement a rule and started
working on this issue. It was quite easy and the rule seems to work, however,
the correct plan is not used. I need to dig deeper into Calcite to figure out
what's going on.
If you'd like to do that, I can also share my branch and you can finish what I
started. Just let me know...
> Built-in agg functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP are broken in
> Group Windows
> ---
>
> Key: FLINK-8903
> URL: https://issues.apache.org/jira/browse/FLINK-8903
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.3.2, 1.5.0, 1.4.2
>Reporter: lilizhao
>Assignee: Shuyi Chen
>Priority: Critical
> Fix For: 1.5.0
>
> Attachments: QQ图片20180312180143.jpg, TableAndSQLTest.java
>
>
> The built-in aggregation functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP
> are translated into regular AVG functions if they are applied in the context
> of a Group Window aggregation (\{{GROUP BY TUMBLE/HOP/SESSION}}).
> The reason is that these functions are internally represented as
> {{SqlAvgAggFunction}} but with different {{SqlKind}}. When translating
> Calcite aggregation functions to Flink Table agg functions, we only look at
> the type of the class, not at the value of the {{kind}} field. We did not
> notice that before, because in all other cases (regular {{GROUP BY}} without
> windows or {{OVER}} windows, we have a translation rule
> {{AggregateReduceFunctionsRule}} that decomposes the more complex functions
> into expressions of {{COUNT}} and {{SUM}} functions such that we never
> execute an {{AVG}} Flink function. That rule can only be applied on
> {{LogicalAggregate}}, however, we represent group windows as
> {{LogicalWindowAggregate}}, so the rule does not match.
> We should fix this by:
> 1. restrict the translation to Flink avg functions in {{AggregateUtil}} to
> {{SqlKind.AVG}}.
> 2. implement a rule (hopefully based on {{AggregateReduceFunctionsRule}})
> that decomposes the complex agg functions into the {{SUM}} and {{COUNT}}.
> Step 1. is easy and a quick fix but we would get an exception "Unsupported
> Function" if {{VAR_POP}} is used in a {{GROUP BY}} window.
> Step 2. might be more involved, depending on how difficult it is to port the
> rule.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8903) Built-in agg functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP are broken in Group Windows
[
https://issues.apache.org/jira/browse/FLINK-8903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400925#comment-16400925
]
Shuyi Chen commented on FLINK-8903:
---
[~fhueske], I can help take a look at this issue.
> Built-in agg functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP are broken in
> Group Windows
> ---
>
> Key: FLINK-8903
> URL: https://issues.apache.org/jira/browse/FLINK-8903
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.3.2, 1.5.0, 1.4.2
>Reporter: lilizhao
>Assignee: Shuyi Chen
>Priority: Critical
> Fix For: 1.5.0
>
> Attachments: QQ图片20180312180143.jpg, TableAndSQLTest.java
>
>
> The built-in aggregation functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP
> are translated into regular AVG functions if they are applied in the context
> of a Group Window aggregation (\{{GROUP BY TUMBLE/HOP/SESSION}}).
> The reason is that these functions are internally represented as
> {{SqlAvgAggFunction}} but with different {{SqlKind}}. When translating
> Calcite aggregation functions to Flink Table agg functions, we only look at
> the type of the class, not at the value of the {{kind}} field. We did not
> notice that before, because in all other cases (regular {{GROUP BY}} without
> windows or {{OVER}} windows, we have a translation rule
> {{AggregateReduceFunctionsRule}} that decomposes the more complex functions
> into expressions of {{COUNT}} and {{SUM}} functions such that we never
> execute an {{AVG}} Flink function. That rule can only be applied on
> {{LogicalAggregate}}, however, we represent group windows as
> {{LogicalWindowAggregate}}, so the rule does not match.
> We should fix this by:
> 1. restrict the translation to Flink avg functions in {{AggregateUtil}} to
> {{SqlKind.AVG}}.
> 2. implement a rule (hopefully based on {{AggregateReduceFunctionsRule}})
> that decomposes the complex agg functions into the {{SUM}} and {{COUNT}}.
> Step 1. is easy and a quick fix but we would get an exception "Unsupported
> Function" if {{VAR_POP}} is used in a {{GROUP BY}} window.
> Step 2. might be more involved, depending on how difficult it is to port the
> rule.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (FLINK-8903) Built-in agg functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP are broken in Group Windows
[
https://issues.apache.org/jira/browse/FLINK-8903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shuyi Chen reassigned FLINK-8903:
-
Assignee: Shuyi Chen
> Built-in agg functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP are broken in
> Group Windows
> ---
>
> Key: FLINK-8903
> URL: https://issues.apache.org/jira/browse/FLINK-8903
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.3.2, 1.5.0, 1.4.2
>Reporter: lilizhao
>Assignee: Shuyi Chen
>Priority: Critical
> Fix For: 1.5.0
>
> Attachments: QQ图片20180312180143.jpg, TableAndSQLTest.java
>
>
> The built-in aggregation functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP
> are translated into regular AVG functions if they are applied in the context
> of a Group Window aggregation (\{{GROUP BY TUMBLE/HOP/SESSION}}).
> The reason is that these functions are internally represented as
> {{SqlAvgAggFunction}} but with different {{SqlKind}}. When translating
> Calcite aggregation functions to Flink Table agg functions, we only look at
> the type of the class, not at the value of the {{kind}} field. We did not
> notice that before, because in all other cases (regular {{GROUP BY}} without
> windows or {{OVER}} windows, we have a translation rule
> {{AggregateReduceFunctionsRule}} that decomposes the more complex functions
> into expressions of {{COUNT}} and {{SUM}} functions such that we never
> execute an {{AVG}} Flink function. That rule can only be applied on
> {{LogicalAggregate}}, however, we represent group windows as
> {{LogicalWindowAggregate}}, so the rule does not match.
> We should fix this by:
> 1. restrict the translation to Flink avg functions in {{AggregateUtil}} to
> {{SqlKind.AVG}}.
> 2. implement a rule (hopefully based on {{AggregateReduceFunctionsRule}})
> that decomposes the complex agg functions into the {{SUM}} and {{COUNT}}.
> Step 1. is easy and a quick fix but we would get an exception "Unsupported
> Function" if {{VAR_POP}} is used in a {{GROUP BY}} window.
> Step 2. might be more involved, depending on how difficult it is to port the
> rule.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-7775) Remove unreferenced method PermanentBlobCache#getNumberOfCachedJobs
[
https://issues.apache.org/jira/browse/FLINK-7775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated FLINK-7775:
--
Description:
{code}
public int getNumberOfCachedJobs() {
return jobRefCounters.size();
}
{code}
The method of PermanentBlobCache is not used.
We should remove it.
was:
{code}
public int getNumberOfCachedJobs() {
return jobRefCounters.size();
}
{code}
The method of PermanentBlobCache is not used.
We should remove it.
> Remove unreferenced method PermanentBlobCache#getNumberOfCachedJobs
> ---
>
> Key: FLINK-7775
> URL: https://issues.apache.org/jira/browse/FLINK-7775
> Project: Flink
> Issue Type: Task
> Components: Local Runtime
>Reporter: Ted Yu
>Priority: Minor
>
> {code}
> public int getNumberOfCachedJobs() {
> return jobRefCounters.size();
> }
> {code}
> The method of PermanentBlobCache is not used.
> We should remove it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-6895) Add STR_TO_DATE supported in SQL
[
https://issues.apache.org/jira/browse/FLINK-6895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400743#comment-16400743
]
Fernando Díaz commented on FLINK-6895:
--
Do you mean the string to format? It can be a column reference, for example:
{code:sql}
UPDATE `table`
SET `column_as_date` = str_to_date( `column_varchar`, '%d-%m-%Y' );
{code}
> Add STR_TO_DATE supported in SQL
>
>
> Key: FLINK-6895
> URL: https://issues.apache.org/jira/browse/FLINK-6895
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Affects Versions: 1.4.0
>Reporter: sunjincheng
>Assignee: Aegeaner
>Priority: Major
>
> STR_TO_DATE(str,format) This is the inverse of the DATE_FORMAT() function. It
> takes a string str and a format string format. STR_TO_DATE() returns a
> DATETIME value if the format string contains both date and time parts, or a
> DATE or TIME value if the string contains only date or time parts. If the
> date, time, or datetime value extracted from str is illegal, STR_TO_DATE()
> returns NULL and produces a warning.
> * Syntax:
> STR_TO_DATE(str,format)
> * Arguments
> **str: -
> **format: -
> * Return Types
> DATAETIME/DATE/TIME
> * Example:
> STR_TO_DATE('01,5,2013','%d,%m,%Y') -> '2013-05-01'
> SELECT STR_TO_DATE('a09:30:17','a%h:%i:%s') -> '09:30:17'
> * See more:
> ** [MySQL|
> https://dev.mysql.com/doc/refman/5.7/en/date-and-time-functions.html#function_str-to-date]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-8903) Built-in agg functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP are broken in Group Windows
[
https://issues.apache.org/jira/browse/FLINK-8903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fabian Hueske updated FLINK-8903:
-
Description:
The built-in aggregation functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP are
translated into regular AVG functions if they are applied in the context of a
Group Window aggregation (\{{GROUP BY TUMBLE/HOP/SESSION}}).
The reason is that these functions are internally represented as
{{SqlAvgAggFunction}} but with different {{SqlKind}}. When translating Calcite
aggregation functions to Flink Table agg functions, we only look at the type of
the class, not at the value of the {{kind}} field. We did not notice that
before, because in all other cases (regular {{GROUP BY}} without windows or
{{OVER}} windows, we have a translation rule {{AggregateReduceFunctionsRule}}
that decomposes the more complex functions into expressions of {{COUNT}} and
{{SUM}} functions such that we never execute an {{AVG}} Flink function. That
rule can only be applied on {{LogicalAggregate}}, however, we represent group
windows as {{LogicalWindowAggregate}}, so the rule does not match.
We should fix this by:
1. restrict the translation to Flink avg functions in {{AggregateUtil}} to
{{SqlKind.AVG}}.
2. implement a rule (hopefully based on {{AggregateReduceFunctionsRule}}) that
decomposes the complex agg functions into the {{SUM}} and {{COUNT}}.
Step 1. is easy and a quick fix but we would get an exception "Unsupported
Function" if {{VAR_POP}} is used in a {{GROUP BY}} window.
Step 2. might be more involved, depending on how difficult it is to port the
rule.
was:
The built-in aggregation functions
> Built-in agg functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP are broken in
> Group Windows
> ---
>
> Key: FLINK-8903
> URL: https://issues.apache.org/jira/browse/FLINK-8903
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.3.2, 1.5.0, 1.4.2
>Reporter: lilizhao
>Priority: Critical
> Fix For: 1.5.0
>
> Attachments: QQ图片20180312180143.jpg, TableAndSQLTest.java
>
>
> The built-in aggregation functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP
> are translated into regular AVG functions if they are applied in the context
> of a Group Window aggregation (\{{GROUP BY TUMBLE/HOP/SESSION}}).
> The reason is that these functions are internally represented as
> {{SqlAvgAggFunction}} but with different {{SqlKind}}. When translating
> Calcite aggregation functions to Flink Table agg functions, we only look at
> the type of the class, not at the value of the {{kind}} field. We did not
> notice that before, because in all other cases (regular {{GROUP BY}} without
> windows or {{OVER}} windows, we have a translation rule
> {{AggregateReduceFunctionsRule}} that decomposes the more complex functions
> into expressions of {{COUNT}} and {{SUM}} functions such that we never
> execute an {{AVG}} Flink function. That rule can only be applied on
> {{LogicalAggregate}}, however, we represent group windows as
> {{LogicalWindowAggregate}}, so the rule does not match.
> We should fix this by:
> 1. restrict the translation to Flink avg functions in {{AggregateUtil}} to
> {{SqlKind.AVG}}.
> 2. implement a rule (hopefully based on {{AggregateReduceFunctionsRule}})
> that decomposes the complex agg functions into the {{SUM}} and {{COUNT}}.
> Step 1. is easy and a quick fix but we would get an exception "Unsupported
> Function" if {{VAR_POP}} is used in a {{GROUP BY}} window.
> Step 2. might be more involved, depending on how difficult it is to port the
> rule.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (FLINK-8685) Code of method "processElement(Ljava/lang/Object;Lorg/apache/flink/streaming/api/functions/ProcessFunction$Context;Lorg/apache/flink/util/Collector;)V" of class "DataStr
[
https://issues.apache.org/jira/browse/FLINK-8685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek reopened FLINK-8685:
-
reopen to change fixVersion
> Code of method
> "processElement(Ljava/lang/Object;Lorg/apache/flink/streaming/api/functions/ProcessFunction$Context;Lorg/apache/flink/util/Collector;)V"
> of class "DataStreamCalcRule$3069" grows beyond 64 KB
> -
>
> Key: FLINK-8685
> URL: https://issues.apache.org/jira/browse/FLINK-8685
> Project: Flink
> Issue Type: Bug
> Components: DataStream API, Table API & SQL
> Environment: Fedora 27
>Reporter: Jahandar Musayev
>Priority: Blocker
>
> I want to use DataStream API and Table API & SQL. I want to read data from
> Apache Kafka and transpose it using SQL. It throws the error below.
> A version of this code for DataSet API works fine.
>
> {noformat}
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue.
> at
> org.apache.flink.table.codegen.Compiler$class.compile(Compiler.scala:36)
> at
> org.apache.flink.table.runtime.CRowProcessRunner.compile(CRowProcessRunner.scala:35)
> at
> org.apache.flink.table.runtime.CRowProcessRunner.open(CRowProcessRunner.scala:49)
> at
> org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
> at
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
> at
> org.apache.flink.streaming.api.operators.ProcessOperator.open(ProcessOperator.java:56)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:393)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:254)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: Compiling "DataStreamCalcRule$3069":
> Code of method
> "processElement(Ljava/lang/Object;Lorg/apache/flink/streaming/api/functions/ProcessFunction$Context;Lorg/apache/flink/util/Collector;)V"
> of class "DataStreamCalcRule$3069" grows beyond 64 KB
> at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:361)
> at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:234)
> at
> org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:446)
> at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:213)
> at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:204)
> at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:80)
> at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:75)
> at
> org.apache.flink.table.codegen.Compiler$class.compile(Compiler.scala:33)
> ... 9 more
> Caused by: org.codehaus.janino.JaninoRuntimeException: Code of method
> "processElement(Ljava/lang/Object;Lorg/apache/flink/streaming/api/functions/ProcessFunction$Context;Lorg/apache/flink/util/Collector;)V"
> of class "DataStreamCalcRule$3069" grows beyond 64 KB
> at org.codehaus.janino.CodeContext.makeSpace(CodeContext.java:974)
> at org.codehaus.janino.CodeContext.write(CodeContext.java:867)
> at org.codehaus.janino.UnitCompiler.writeOpcode(UnitCompiler.java:11753)
> at org.codehaus.janino.UnitCompiler.writeLdc(UnitCompiler.java:10512)
> at org.codehaus.janino.UnitCompiler.pushConstant(UnitCompiler.java:10280)
> at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:5202)
> at org.codehaus.janino.UnitCompiler.access$8400(UnitCompiler.java:212)
> at
> org.codehaus.janino.UnitCompiler$12.visitIntegerLiteral(UnitCompiler.java:4073)
> at
> org.codehaus.janino.UnitCompiler$12.visitIntegerLiteral(UnitCompiler.java:4044)
> at org.codehaus.janino.Java$IntegerLiteral.accept(Java.java:5250)
> at org.codehaus.janino.UnitCompiler.compileGet(UnitCompiler.java:4044)
> at org.codehaus.janino.UnitCompiler.fakeCompile(UnitCompiler.java:3383)
> at
> org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:5218)
> at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:4813)
> at org.codehaus.janino.UnitCompiler.access$8200(UnitCompiler.java:212)
> at
> org.codehaus.janino.UnitCompiler$12.visitMethodInvocation(UnitCompiler.java:4071)
> at
> org.codehaus.janino.UnitCompiler$12.visitMethodInvocation(UnitCompiler.java:4044)
> at org.codehaus.janino.Java$MethodInvocation.accept(Java.java:4874)
> at org.codehaus.janino.UnitCompiler.compileGet(UnitCompiler.java:4044)
> at
> org.codehaus.janino.UnitComp
[jira] [Closed] (FLINK-8685) Code of method "processElement(Ljava/lang/Object;Lorg/apache/flink/streaming/api/functions/ProcessFunction$Context;Lorg/apache/flink/util/Collector;)V" of class "DataStrea
[
https://issues.apache.org/jira/browse/FLINK-8685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek closed FLINK-8685.
---
Resolution: Duplicate
Fix Version/s: (was: 1.5.0)
> Code of method
> "processElement(Ljava/lang/Object;Lorg/apache/flink/streaming/api/functions/ProcessFunction$Context;Lorg/apache/flink/util/Collector;)V"
> of class "DataStreamCalcRule$3069" grows beyond 64 KB
> -
>
> Key: FLINK-8685
> URL: https://issues.apache.org/jira/browse/FLINK-8685
> Project: Flink
> Issue Type: Bug
> Components: DataStream API, Table API & SQL
> Environment: Fedora 27
>Reporter: Jahandar Musayev
>Priority: Blocker
>
> I want to use DataStream API and Table API & SQL. I want to read data from
> Apache Kafka and transpose it using SQL. It throws the error below.
> A version of this code for DataSet API works fine.
>
> {noformat}
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue.
> at
> org.apache.flink.table.codegen.Compiler$class.compile(Compiler.scala:36)
> at
> org.apache.flink.table.runtime.CRowProcessRunner.compile(CRowProcessRunner.scala:35)
> at
> org.apache.flink.table.runtime.CRowProcessRunner.open(CRowProcessRunner.scala:49)
> at
> org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
> at
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
> at
> org.apache.flink.streaming.api.operators.ProcessOperator.open(ProcessOperator.java:56)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:393)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:254)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: Compiling "DataStreamCalcRule$3069":
> Code of method
> "processElement(Ljava/lang/Object;Lorg/apache/flink/streaming/api/functions/ProcessFunction$Context;Lorg/apache/flink/util/Collector;)V"
> of class "DataStreamCalcRule$3069" grows beyond 64 KB
> at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:361)
> at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:234)
> at
> org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:446)
> at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:213)
> at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:204)
> at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:80)
> at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:75)
> at
> org.apache.flink.table.codegen.Compiler$class.compile(Compiler.scala:33)
> ... 9 more
> Caused by: org.codehaus.janino.JaninoRuntimeException: Code of method
> "processElement(Ljava/lang/Object;Lorg/apache/flink/streaming/api/functions/ProcessFunction$Context;Lorg/apache/flink/util/Collector;)V"
> of class "DataStreamCalcRule$3069" grows beyond 64 KB
> at org.codehaus.janino.CodeContext.makeSpace(CodeContext.java:974)
> at org.codehaus.janino.CodeContext.write(CodeContext.java:867)
> at org.codehaus.janino.UnitCompiler.writeOpcode(UnitCompiler.java:11753)
> at org.codehaus.janino.UnitCompiler.writeLdc(UnitCompiler.java:10512)
> at org.codehaus.janino.UnitCompiler.pushConstant(UnitCompiler.java:10280)
> at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:5202)
> at org.codehaus.janino.UnitCompiler.access$8400(UnitCompiler.java:212)
> at
> org.codehaus.janino.UnitCompiler$12.visitIntegerLiteral(UnitCompiler.java:4073)
> at
> org.codehaus.janino.UnitCompiler$12.visitIntegerLiteral(UnitCompiler.java:4044)
> at org.codehaus.janino.Java$IntegerLiteral.accept(Java.java:5250)
> at org.codehaus.janino.UnitCompiler.compileGet(UnitCompiler.java:4044)
> at org.codehaus.janino.UnitCompiler.fakeCompile(UnitCompiler.java:3383)
> at
> org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:5218)
> at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:4813)
> at org.codehaus.janino.UnitCompiler.access$8200(UnitCompiler.java:212)
> at
> org.codehaus.janino.UnitCompiler$12.visitMethodInvocation(UnitCompiler.java:4071)
> at
> org.codehaus.janino.UnitCompiler$12.visitMethodInvocation(UnitCompiler.java:4044)
> at org.codehaus.janino.Java$MethodInvocation.accept(Java.java:4874)
> at org.codehaus.janino.UnitCompiler.compileGet(UnitCompiler.java:4044)
> at
[jira] [Reopened] (FLINK-8806) Failure in UnionInputGate getNextBufferOrEvent()
[
https://issues.apache.org/jira/browse/FLINK-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek reopened FLINK-8806:
-
reopen to fix fixVErsion
> Failure in UnionInputGate getNextBufferOrEvent()
>
>
> Key: FLINK-8806
> URL: https://issues.apache.org/jira/browse/FLINK-8806
> Project: Flink
> Issue Type: Bug
> Components: Network
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Stephan Ewen
>Priority: Blocker
>
> Error occurs in {{SelfConnectionITCase}}:
> Full log: https://api.travis-ci.org/v3/job/346847455/log.txt
> Exception Stack Trace
> {code}
> org.apache.flink.runtime.client.JobExecutionException:
> java.lang.IllegalStateException
> at
> org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:527)
> at
> org.apache.flink.streaming.util.TestStreamEnvironment.execute(TestStreamEnvironment.java:79)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1510)
> at
> org.apache.flink.test.streaming.runtime.SelfConnectionITCase.differentDataStreamDifferentChain(SelfConnectionITCase.java:158)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:108)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:78)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:54)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:144)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Caused by: java.lang.IllegalStateException
> at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:179)
> at
> org.apache.flink.runtime.io.network.partition.consumer.UnionInputGate.getNextBufferOrEvent(UnionInputGate.java:173)
> at
> org.apache.flink.streaming.runtime.io.BarrierTracker.getNextNonBlocked(BarrierTracker.java:94)
> at
> org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.processInput(
[jira] [Closed] (FLINK-8806) Failure in UnionInputGate getNextBufferOrEvent()
[
https://issues.apache.org/jira/browse/FLINK-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek closed FLINK-8806.
---
Resolution: Duplicate
Fix Version/s: (was: 1.6.0)
(was: 1.5.0)
> Failure in UnionInputGate getNextBufferOrEvent()
>
>
> Key: FLINK-8806
> URL: https://issues.apache.org/jira/browse/FLINK-8806
> Project: Flink
> Issue Type: Bug
> Components: Network
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Stephan Ewen
>Priority: Blocker
>
> Error occurs in {{SelfConnectionITCase}}:
> Full log: https://api.travis-ci.org/v3/job/346847455/log.txt
> Exception Stack Trace
> {code}
> org.apache.flink.runtime.client.JobExecutionException:
> java.lang.IllegalStateException
> at
> org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:527)
> at
> org.apache.flink.streaming.util.TestStreamEnvironment.execute(TestStreamEnvironment.java:79)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1510)
> at
> org.apache.flink.test.streaming.runtime.SelfConnectionITCase.differentDataStreamDifferentChain(SelfConnectionITCase.java:158)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:108)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:78)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:54)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:144)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Caused by: java.lang.IllegalStateException
> at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:179)
> at
> org.apache.flink.runtime.io.network.partition.consumer.UnionInputGate.getNextBufferOrEvent(UnionInputGate.java:173)
> at
> org.apache.flink.streaming.runtime.io.BarrierTracker.getNextNonBlocked(BarrierTracker.java:94)
> at
> org.
[jira] [Closed] (FLINK-8885) The DispatcherThreadFactory should register uncaught exception handlers
[
https://issues.apache.org/jira/browse/FLINK-8885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek closed FLINK-8885.
---
Resolution: Duplicate
Fix Version/s: (was: 1.6.0)
(was: 1.5.0)
> The DispatcherThreadFactory should register uncaught exception handlers
> ---
>
> Key: FLINK-8885
> URL: https://issues.apache.org/jira/browse/FLINK-8885
> Project: Flink
> Issue Type: Bug
> Components: TaskManager
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Major
>
> The {{DispatcherThreadFactory}} is responsible for spawning the thread pool
> threads for TaskManager's async dispatcher and for the CheckpointCoordinators
> timed trigger.
> In case of uncaught exceptions in these threads, the system is not healthy
> and more, hence they should register the
> {{FatalExitUcaughtExceptionsHandler}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (FLINK-8885) The DispatcherThreadFactory should register uncaught exception handlers
[
https://issues.apache.org/jira/browse/FLINK-8885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek reopened FLINK-8885:
-
reopen to fix fixVersion
> The DispatcherThreadFactory should register uncaught exception handlers
> ---
>
> Key: FLINK-8885
> URL: https://issues.apache.org/jira/browse/FLINK-8885
> Project: Flink
> Issue Type: Bug
> Components: TaskManager
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Major
>
> The {{DispatcherThreadFactory}} is responsible for spawning the thread pool
> threads for TaskManager's async dispatcher and for the CheckpointCoordinators
> timed trigger.
> In case of uncaught exceptions in these threads, the system is not healthy
> and more, hence they should register the
> {{FatalExitUcaughtExceptionsHandler}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-8903) Built-in agg functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP are broken in Group Windows
[ https://issues.apache.org/jira/browse/FLINK-8903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Hueske updated FLINK-8903: - Affects Version/s: (was: 1.4.1) 1.5.0 1.3.2 1.4.2 > Built-in agg functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP are broken in > Group Windows > --- > > Key: FLINK-8903 > URL: https://issues.apache.org/jira/browse/FLINK-8903 > Project: Flink > Issue Type: Bug > Components: Table API & SQL >Affects Versions: 1.3.2, 1.5.0, 1.4.2 >Reporter: lilizhao >Priority: Critical > Fix For: 1.5.0 > > Attachments: QQ图片20180312180143.jpg, TableAndSQLTest.java > > > Error calculation based on rolling window in table API and SQL API > The variance of the calculation is equal to the average. > 1 The test code is detailed in the appendix > 2 The test data are as follows > 1 li > 100 li > 3 The Table API test result as follows > (true,50.5,101.0,50.5,li,2018-03-09 09:11:00.0) > 4 The SQL API test result as follows > (true,50.5,2,101.0,50.5,li,2018-03-09 09:21:00.0) > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-8903) Built-in agg functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP are broken in Group Windows
[ https://issues.apache.org/jira/browse/FLINK-8903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Hueske updated FLINK-8903: - Description: The built-in aggregation functions was: Error calculation based on rolling window in table API and SQL API The variance of the calculation is equal to the average. 1 The test code is detailed in the appendix 2 The test data are as follows 1 li 100 li 3 The Table API test result as follows (true,50.5,101.0,50.5,li,2018-03-09 09:11:00.0) 4 The SQL API test result as follows (true,50.5,2,101.0,50.5,li,2018-03-09 09:21:00.0) > Built-in agg functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP are broken in > Group Windows > --- > > Key: FLINK-8903 > URL: https://issues.apache.org/jira/browse/FLINK-8903 > Project: Flink > Issue Type: Bug > Components: Table API & SQL >Affects Versions: 1.3.2, 1.5.0, 1.4.2 >Reporter: lilizhao >Priority: Critical > Fix For: 1.5.0 > > Attachments: QQ图片20180312180143.jpg, TableAndSQLTest.java > > > The built-in aggregation functions > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-8903) Built-in agg functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP are broken in Group Windows
[ https://issues.apache.org/jira/browse/FLINK-8903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Hueske updated FLINK-8903: - Summary: Built-in agg functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP are broken in Group Windows (was: Error calculation based on rolling window in table API and SQL API) > Built-in agg functions VAR_POP, VAR_SAMP, STDEV_POP, STDEV_SAMP are broken in > Group Windows > --- > > Key: FLINK-8903 > URL: https://issues.apache.org/jira/browse/FLINK-8903 > Project: Flink > Issue Type: Bug > Components: Table API & SQL >Affects Versions: 1.4.1 >Reporter: lilizhao >Priority: Critical > Fix For: 1.5.0 > > Attachments: QQ图片20180312180143.jpg, TableAndSQLTest.java > > > Error calculation based on rolling window in table API and SQL API > The variance of the calculation is equal to the average. > 1 The test code is detailed in the appendix > 2 The test data are as follows > 1 li > 100 li > 3 The Table API test result as follows > (true,50.5,101.0,50.5,li,2018-03-09 09:11:00.0) > 4 The SQL API test result as follows > (true,50.5,2,101.0,50.5,li,2018-03-09 09:21:00.0) > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8968) Fix native resource leak caused by ReadOptions
[
https://issues.apache.org/jira/browse/FLINK-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400662#comment-16400662
]
ASF GitHub Bot commented on FLINK-8968:
---
Github user StefanRRichter commented on the issue:
https://github.com/apache/flink/pull/5705
LGTM 👍
> Fix native resource leak caused by ReadOptions
> ---
>
> Key: FLINK-8968
> URL: https://issues.apache.org/jira/browse/FLINK-8968
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.5.0
>Reporter: Sihua Zhou
>Assignee: Sihua Zhou
>Priority: Major
> Fix For: 1.5.0
>
>
> We should pull the creation of ReadOptions out of the loop in
> {{RocksDBFullSnapshotOperation.writeKVStateMetaData()}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink issue #5705: [FLINK-8968][state]Fix native resource leak caused by Rea...
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/5705 LGTM ð ---
[jira] [Commented] (FLINK-8903) Error calculation based on rolling window in table API and SQL API
[ https://issues.apache.org/jira/browse/FLINK-8903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400654#comment-16400654 ] Fabian Hueske commented on FLINK-8903: -- Hi [~lilizhao], Sorry for being too eager when closing the issue! You are right, this is a bug. I reproduced the issue and also found the cause of the bug. Let me see, if there's an easy way to fix it. In worst case, we need to disable some of the aggregation functions. Best, Fabian > Error calculation based on rolling window in table API and SQL API > -- > > Key: FLINK-8903 > URL: https://issues.apache.org/jira/browse/FLINK-8903 > Project: Flink > Issue Type: Bug > Components: Table API & SQL >Affects Versions: 1.4.1 >Reporter: lilizhao >Priority: Critical > Fix For: 1.5.0 > > Attachments: QQ图片20180312180143.jpg, TableAndSQLTest.java > > > Error calculation based on rolling window in table API and SQL API > The variance of the calculation is equal to the average. > 1 The test code is detailed in the appendix > 2 The test data are as follows > 1 li > 100 li > 3 The Table API test result as follows > (true,50.5,101.0,50.5,li,2018-03-09 09:11:00.0) > 4 The SQL API test result as follows > (true,50.5,2,101.0,50.5,li,2018-03-09 09:21:00.0) > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8968) Fix native resource leak caused by ReadOptions
[
https://issues.apache.org/jira/browse/FLINK-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400650#comment-16400650
]
ASF GitHub Bot commented on FLINK-8968:
---
Github user sihuazhou commented on the issue:
https://github.com/apache/flink/pull/5705
@StefanRRichter Could you please have a look at this?
> Fix native resource leak caused by ReadOptions
> ---
>
> Key: FLINK-8968
> URL: https://issues.apache.org/jira/browse/FLINK-8968
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.5.0
>Reporter: Sihua Zhou
>Assignee: Sihua Zhou
>Priority: Major
> Fix For: 1.5.0
>
>
> We should pull the creation of ReadOptions out of the loop in
> {{RocksDBFullSnapshotOperation.writeKVStateMetaData()}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink issue #5705: [FLINK-8968][state]Fix native resource leak caused by Rea...
Github user sihuazhou commented on the issue: https://github.com/apache/flink/pull/5705 @StefanRRichter Could you please have a look at this? ---
[jira] [Commented] (FLINK-8968) Fix native resource leak caused by ReadOptions
[
https://issues.apache.org/jira/browse/FLINK-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400646#comment-16400646
]
ASF GitHub Bot commented on FLINK-8968:
---
GitHub user sihuazhou opened a pull request:
https://github.com/apache/flink/pull/5705
[FLINK-8968][state]Fix native resource leak caused by ReadOptions
## What is the purpose of the change
This PR fixes
(FLINK-8968)[https://issues.apache.org/jira/browse/FLINK-8968] Pull the
creation of `ReadOptions` out of loop to avoid native resource leak in Full
checkpoint.
## Brief change log
- pull the creation of `ReadOptions` out of loop in
`RocksDBFullSnapshotOperation.writeKVStateMetaData()`
## Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: ( no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes)
- The S3 file system connector: (no)
## Documentation
no
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sihuazhou/flink fix_readOptions_leak
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5705.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5705
commit f1cd2e7f6962e3307e842712c68200362bec61d7
Author: sihuazhou
Date: 2018-03-15T15:57:11Z
Poll the creation of readOptions out of loop to avoid native resource leak.
> Fix native resource leak caused by ReadOptions
> ---
>
> Key: FLINK-8968
> URL: https://issues.apache.org/jira/browse/FLINK-8968
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.5.0
>Reporter: Sihua Zhou
>Assignee: Sihua Zhou
>Priority: Major
> Fix For: 1.5.0
>
>
> We should pull the creation of ReadOptions out of the loop in
> {{RocksDBFullSnapshotOperation.writeKVStateMetaData()}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink pull request #5705: [FLINK-8968][state]Fix native resource leak caused...
GitHub user sihuazhou opened a pull request: https://github.com/apache/flink/pull/5705 [FLINK-8968][state]Fix native resource leak caused by ReadOptions ## What is the purpose of the change This PR fixes (FLINK-8968)[https://issues.apache.org/jira/browse/FLINK-8968] Pull the creation of `ReadOptions` out of loop to avoid native resource leak in Full checkpoint. ## Brief change log - pull the creation of `ReadOptions` out of loop in `RocksDBFullSnapshotOperation.writeKVStateMetaData()` ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes) - The S3 file system connector: (no) ## Documentation no You can merge this pull request into a Git repository by running: $ git pull https://github.com/sihuazhou/flink fix_readOptions_leak Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5705.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5705 commit f1cd2e7f6962e3307e842712c68200362bec61d7 Author: sihuazhou Date: 2018-03-15T15:57:11Z Poll the creation of readOptions out of loop to avoid native resource leak. ---
[jira] [Created] (FLINK-8968) Fix native resource leak caused by ReadOptions
Sihua Zhou created FLINK-8968:
-
Summary: Fix native resource leak caused by ReadOptions
Key: FLINK-8968
URL: https://issues.apache.org/jira/browse/FLINK-8968
Project: Flink
Issue Type: Bug
Components: State Backends, Checkpointing
Affects Versions: 1.5.0
Reporter: Sihua Zhou
Assignee: Sihua Zhou
Fix For: 1.5.0
We should pull the creation of ReadOptions out of the loop in
{{RocksDBFullSnapshotOperation.writeKVStateMetaData()}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-7804) YarnResourceManager does not execute AMRMClientAsync callbacks in main thread
[
https://issues.apache.org/jira/browse/FLINK-7804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400613#comment-16400613
]
ASF GitHub Bot commented on FLINK-7804:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/5675#discussion_r174834772
--- Diff:
flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManager.java ---
@@ -395,10 +405,13 @@ public void onNodesUpdated(List list) {
@Override
public void onError(Throwable error) {
- onFatalError(error);
+ runAsync(() -> onFatalError(error));
--- End diff --
I think it would be good to let the error propagate directly. In case of an
OOM exception we want to quickly shut down the JVM.
> YarnResourceManager does not execute AMRMClientAsync callbacks in main thread
> -
>
> Key: FLINK-7804
> URL: https://issues.apache.org/jira/browse/FLINK-7804
> Project: Flink
> Issue Type: Bug
> Components: Distributed Coordination, YARN
>Affects Versions: 1.4.0, 1.5.0
>Reporter: Till Rohrmann
>Assignee: Gary Yao
>Priority: Blocker
> Labels: flip-6
> Fix For: 1.5.0
>
>
> The {{YarnResourceManager}} registers callbacks at a {{AMRMClientAsync}}
> which it uses to react to Yarn container allocations. These callbacks (e.g.
> {{onContainersAllocated}} modify the internal state of the
> {{YarnResourceManager}}. This can lead to race conditions with the
> {{requestYarnContainer}} method.
> In order to solve this problem we have to execute the state changing
> operations in the main thread of the {{YarnResourceManager}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-7804) YarnResourceManager does not execute AMRMClientAsync callbacks in main thread
[
https://issues.apache.org/jira/browse/FLINK-7804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400614#comment-16400614
]
ASF GitHub Bot commented on FLINK-7804:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/5675#discussion_r174834515
--- Diff:
flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManager.java ---
@@ -325,67 +328,74 @@ public float getProgress() {
@Override
public void onContainersCompleted(List list) {
- for (ContainerStatus container : list) {
- if (container.getExitStatus() < 0) {
- closeTaskManagerConnection(new ResourceID(
- container.getContainerId().toString()),
new Exception(container.getDiagnostics()));
+ runAsync(() -> {
+ for (ContainerStatus container : list) {
+ if (container.getExitStatus() < 0) {
+ closeTaskManagerConnection(new
ResourceID(
+
container.getContainerId().toString()), new
Exception(container.getDiagnostics()));
+ }
+ workerNodeMap.remove(new
ResourceID(container.getContainerId().toString()));
+ }
}
- workerNodeMap.remove(new
ResourceID(container.getContainerId().toString()));
- }
+ );
}
@Override
public void onContainersAllocated(List containers) {
- for (Container container : containers) {
- log.info(
- "Received new container: {} - Remaining pending
container requests: {}",
- container.getId(),
- numPendingContainerRequests);
-
- if (numPendingContainerRequests > 0) {
- numPendingContainerRequests--;
-
- final String containerIdStr =
container.getId().toString();
-
- workerNodeMap.put(new
ResourceID(containerIdStr), new YarnWorkerNode(container));
-
- try {
- // Context information used to start a
TaskExecutor Java process
- ContainerLaunchContext
taskExecutorLaunchContext = createTaskExecutorLaunchContext(
- container.getResource(),
- containerIdStr,
-
container.getNodeId().getHost());
-
-
nodeManagerClient.startContainer(container, taskExecutorLaunchContext);
- } catch (Throwable t) {
- log.error("Could not start TaskManager
in container {}.", container.getId(), t);
-
- // release the failed container
+ runAsync(() -> {
+ for (Container container : containers) {
+ log.info(
+ "Received new container: {} - Remaining
pending container requests: {}",
+ container.getId(),
+ numPendingContainerRequests);
+
+ if (numPendingContainerRequests > 0) {
+ numPendingContainerRequests--;
+
+ final String containerIdStr =
container.getId().toString();
+
+ workerNodeMap.put(new
ResourceID(containerIdStr), new YarnWorkerNode(container));
+
+ try {
+ // Context information used to
start a TaskExecutor Java process
+ ContainerLaunchContext
taskExecutorLaunchContext = createTaskExecutorLaunchContext(
+ container.getResource(),
+ containerIdStr,
+
container.getNodeId().getHost());
+
+
nodeManagerClient.startContainer(container, taskExecutorLaunchContext);
+ } catch (Throwable t) {
+ log.error("Could not start
TaskManager in container {}.", container.getId(), t);
+
+
[GitHub] flink pull request #5675: [FLINK-7804][flip6] YarnResourceManager does not e...
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/5675#discussion_r174834515
--- Diff:
flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManager.java ---
@@ -325,67 +328,74 @@ public float getProgress() {
@Override
public void onContainersCompleted(List list) {
- for (ContainerStatus container : list) {
- if (container.getExitStatus() < 0) {
- closeTaskManagerConnection(new ResourceID(
- container.getContainerId().toString()),
new Exception(container.getDiagnostics()));
+ runAsync(() -> {
+ for (ContainerStatus container : list) {
+ if (container.getExitStatus() < 0) {
+ closeTaskManagerConnection(new
ResourceID(
+
container.getContainerId().toString()), new
Exception(container.getDiagnostics()));
+ }
+ workerNodeMap.remove(new
ResourceID(container.getContainerId().toString()));
+ }
}
- workerNodeMap.remove(new
ResourceID(container.getContainerId().toString()));
- }
+ );
}
@Override
public void onContainersAllocated(List containers) {
- for (Container container : containers) {
- log.info(
- "Received new container: {} - Remaining pending
container requests: {}",
- container.getId(),
- numPendingContainerRequests);
-
- if (numPendingContainerRequests > 0) {
- numPendingContainerRequests--;
-
- final String containerIdStr =
container.getId().toString();
-
- workerNodeMap.put(new
ResourceID(containerIdStr), new YarnWorkerNode(container));
-
- try {
- // Context information used to start a
TaskExecutor Java process
- ContainerLaunchContext
taskExecutorLaunchContext = createTaskExecutorLaunchContext(
- container.getResource(),
- containerIdStr,
-
container.getNodeId().getHost());
-
-
nodeManagerClient.startContainer(container, taskExecutorLaunchContext);
- } catch (Throwable t) {
- log.error("Could not start TaskManager
in container {}.", container.getId(), t);
-
- // release the failed container
+ runAsync(() -> {
+ for (Container container : containers) {
+ log.info(
+ "Received new container: {} - Remaining
pending container requests: {}",
+ container.getId(),
+ numPendingContainerRequests);
+
+ if (numPendingContainerRequests > 0) {
+ numPendingContainerRequests--;
+
+ final String containerIdStr =
container.getId().toString();
+
+ workerNodeMap.put(new
ResourceID(containerIdStr), new YarnWorkerNode(container));
+
+ try {
+ // Context information used to
start a TaskExecutor Java process
+ ContainerLaunchContext
taskExecutorLaunchContext = createTaskExecutorLaunchContext(
+ container.getResource(),
+ containerIdStr,
+
container.getNodeId().getHost());
+
+
nodeManagerClient.startContainer(container, taskExecutorLaunchContext);
+ } catch (Throwable t) {
+ log.error("Could not start
TaskManager in container {}.", container.getId(), t);
+
+ // release the failed container
+
resourceManagerClient.releaseAssignedContainer(container.getId());
+ // and ask for a new one
+
[GitHub] flink pull request #5675: [FLINK-7804][flip6] YarnResourceManager does not e...
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/5675#discussion_r174834772
--- Diff:
flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManager.java ---
@@ -395,10 +405,13 @@ public void onNodesUpdated(List list) {
@Override
public void onError(Throwable error) {
- onFatalError(error);
+ runAsync(() -> onFatalError(error));
--- End diff --
I think it would be good to let the error propagate directly. In case of an
OOM exception we want to quickly shut down the JVM.
---
[jira] [Commented] (FLINK-8922) Revert FLINK-8859 because it causes segfaults in testing
[ https://issues.apache.org/jira/browse/FLINK-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400595#comment-16400595 ] Sihua Zhou commented on FLINK-8922: --- Hmm... I will try it out, I'm looping the code of `RocksDBKeyedStateBackend.java` again and again ..., and there seems to have some native resource that forgot to be released, but it minor and still not the reason for this issue. Any way, I will try out your suggestion firstly. > Revert FLINK-8859 because it causes segfaults in testing > > > Key: FLINK-8922 > URL: https://issues.apache.org/jira/browse/FLINK-8922 > Project: Flink > Issue Type: Bug > Components: State Backends, Checkpointing >Affects Versions: 1.5.0 >Reporter: Stefan Richter >Assignee: Stefan Richter >Priority: Major > Fix For: 1.5.0 > > > We need to revert FLINK-8859 because it causes problems with RocksDB that > make our automated tests fail on Travis. The change looks actually good and > it is currently unclear why this can introduce such a problem. This might > also be a Rocks in RocksDB. Nevertheless, for the sake of a proper release > testing, we should revert the change for now. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8922) Revert FLINK-8859 because it causes segfaults in testing
[
https://issues.apache.org/jira/browse/FLINK-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400580#comment-16400580
]
Stephan Ewen commented on FLINK-8922:
-
If I see this correctly, we still assume it is related to object release,
because it occurs only on Travis (rarely on developer laptops) related to GCs
happening.
Another think we can try is to have a {{private static final WriteOptions}}
object in the {{RocksDBKeyedStateBackend}} that simply never gets released. We
need to have that object's initialization after the {{System.loadLibrary()}}
call, though. That could help.
If that does not help, it might not be related to GC / object release after
all, and really be a WAL bug in RocksDB.
> Revert FLINK-8859 because it causes segfaults in testing
>
>
> Key: FLINK-8922
> URL: https://issues.apache.org/jira/browse/FLINK-8922
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.5.0
>Reporter: Stefan Richter
>Assignee: Stefan Richter
>Priority: Major
> Fix For: 1.5.0
>
>
> We need to revert FLINK-8859 because it causes problems with RocksDB that
> make our automated tests fail on Travis. The change looks actually good and
> it is currently unclear why this can introduce such a problem. This might
> also be a Rocks in RocksDB. Nevertheless, for the sake of a proper release
> testing, we should revert the change for now.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8297) RocksDBListState stores whole list in single byte[]
[
https://issues.apache.org/jira/browse/FLINK-8297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400570#comment-16400570
]
Jan Lukavský commented on FLINK-8297:
-
Yes, that seems related. I'm not 100% convinced that simply overcoming size
limitation of `Integer.MAX_VALUE` solves the actual problem, because the whole
list would have to still be stored in memory and can therefore result in
various OOM errors, or containers being killed (e.g. on YARN).
> RocksDBListState stores whole list in single byte[]
> ---
>
> Key: FLINK-8297
> URL: https://issues.apache.org/jira/browse/FLINK-8297
> Project: Flink
> Issue Type: Improvement
> Components: Core
>Affects Versions: 1.4.0, 1.3.2
>Reporter: Jan Lukavský
>Priority: Major
>
> RocksDBListState currently keeps whole list of data in single RocksDB
> key-value pair, which implies that the list actually must fit into memory.
> Larger lists are not supported and end up with OOME or other error. The
> RocksDBListState could be modified so that individual items in list are
> stored in separate keys in RocksDB and can then be iterated over. A simple
> implementation could reuse existing RocksDBMapState, with key as index to the
> list and a single RocksDBValueState keeping track of how many items has
> already been added to the list. Because this implementation might be less
> efficient in come cases, it would be good to make it opt-in by a construct
> like
> {{new RocksDBStateBackend().enableLargeListsPerKey()}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8852) SQL Client does not work with new FLIP-6 mode
[ https://issues.apache.org/jira/browse/FLINK-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400565#comment-16400565 ] ASF GitHub Bot commented on FLINK-8852: --- GitHub user twalthr opened a pull request: https://github.com/apache/flink/pull/5704 [FLINK-8852] [sql-client] Add FLIP-6 support to SQL Client ## What is the purpose of the change This PR adds support for the new FLIP-6 mode in the SQL Client. For now, we only test the standalone mode. But in theory the current design should work with any deployment. ## Brief change log - Use new FLIP-6 classes similar to `CliFrontend` - Make some methods visible in `flink-clients` for reducing duplicate code. ## Verifying this change Updated the existing integration tests. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): yes - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: yes - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? JavaDocs You can merge this pull request into a Git repository by running: $ git pull https://github.com/twalthr/flink FLINK-8852 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5704.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5704 commit d179cb885b36f20914f3e94dae602069681166c3 Author: Timo Walther Date: 2018-03-14T14:38:48Z [FLINK-8852] [sql-client] SQL Client does not work with new FLIP-6 mode > SQL Client does not work with new FLIP-6 mode > - > > Key: FLINK-8852 > URL: https://issues.apache.org/jira/browse/FLINK-8852 > Project: Flink > Issue Type: Sub-task > Components: Table API & SQL >Affects Versions: 1.5.0 >Reporter: Fabian Hueske >Assignee: Timo Walther >Priority: Blocker > Fix For: 1.5.0 > > > The SQL client does not submit queries to local Flink cluster that runs in > FLIP-6 mode. It doesn't throw an exception either. > Job submission works if the legacy Flink cluster mode is used (`mode: old`) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] flink pull request #5704: [FLINK-8852] [sql-client] Add FLIP-6 support to SQ...
GitHub user twalthr opened a pull request: https://github.com/apache/flink/pull/5704 [FLINK-8852] [sql-client] Add FLIP-6 support to SQL Client ## What is the purpose of the change This PR adds support for the new FLIP-6 mode in the SQL Client. For now, we only test the standalone mode. But in theory the current design should work with any deployment. ## Brief change log - Use new FLIP-6 classes similar to `CliFrontend` - Make some methods visible in `flink-clients` for reducing duplicate code. ## Verifying this change Updated the existing integration tests. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): yes - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: yes - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? JavaDocs You can merge this pull request into a Git repository by running: $ git pull https://github.com/twalthr/flink FLINK-8852 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5704.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5704 commit d179cb885b36f20914f3e94dae602069681166c3 Author: Timo Walther Date: 2018-03-14T14:38:48Z [FLINK-8852] [sql-client] SQL Client does not work with new FLIP-6 mode ---
[jira] [Closed] (FLINK-8930) TableApi validation test in ScalarFunctionsValidationTest doesn't work
[ https://issues.apache.org/jira/browse/FLINK-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Hueske closed FLINK-8930. Resolution: Not A Problem > TableApi validation test in ScalarFunctionsValidationTest doesn't work > -- > > Key: FLINK-8930 > URL: https://issues.apache.org/jira/browse/FLINK-8930 > Project: Flink > Issue Type: Bug > Components: Table API & SQL >Reporter: Wind >Priority: Major > > I'm wring a validation test for > [FLINK-6924|https://issues.apache.org/jira/browse/FLINK-6924] in > org.apache.flink.table.expressions.validation.ScalarFunctionsValidationTest. > However, I find that the table api is not truely executed in function > "testTableApi", which is different from "testSqlApi". So we can only test > exceptions which are thrown in "addTableApiTestExpr" like > "ValidationException" because it is thrown during "select" operation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8930) TableApi validation test in ScalarFunctionsValidationTest doesn't work
[
https://issues.apache.org/jira/browse/FLINK-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400561#comment-16400561
]
Fabian Hueske commented on FLINK-8930:
--
OK, I see what you mean.
The difference is that the SQL expression is validated by Calcite and the Table
API expression by the Table API. The SQL validation actually succeeds but
Calcite also tries to simplify the expression by evaluating it because all
input parameters are constants. This means that the function is invoked on the
constant parameters. If you change the test SQL call to
{{testSqlApi("rpad(f8,-1,'')", "null")}} ({{f8}} is a String field and hence
not constant), you'll find that it is executed just like the Table API test
method.
In general, this error is not a validation but a runtime error, because it is
not caused by invalid types but by invalid values at runtime.
>From my point of view, this is all expected behavior. I'd suggest to close
>this issue.
> TableApi validation test in ScalarFunctionsValidationTest doesn't work
> --
>
> Key: FLINK-8930
> URL: https://issues.apache.org/jira/browse/FLINK-8930
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Reporter: Wind
>Priority: Major
>
> I'm wring a validation test for
> [FLINK-6924|https://issues.apache.org/jira/browse/FLINK-6924] in
> org.apache.flink.table.expressions.validation.ScalarFunctionsValidationTest.
> However, I find that the table api is not truely executed in function
> "testTableApi", which is different from "testSqlApi". So we can only test
> exceptions which are thrown in "addTableApiTestExpr" like
> "ValidationException" because it is thrown during "select" operation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (FLINK-8930) TableApi validation test in ScalarFunctionsValidationTest doesn't work
[
https://issues.apache.org/jira/browse/FLINK-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400561#comment-16400561
]
Fabian Hueske edited comment on FLINK-8930 at 3/15/18 3:17 PM:
---
OK, I see what you mean.
The difference is that the SQL expression is validated by Calcite and the Table
API expression by the Table API. The SQL validation actually succeeds but
Calcite also tries to simplify the expression by evaluating it because all
input parameters are constants. This means that the function is invoked on the
constant parameters. If you change the test SQL call to
{{testSqlApi("rpad(f8,-1,'')", "null")}} ({{f8}} is a String field and hence
not constant), you'll find that it is executed just like the Table API test
method.
In general, this error is not a validation but a runtime error, because it is
not caused by invalid types but by invalid values at runtime.
>From my point of view, this is all expected behavior. I'd suggest to close
>this issue.
Feel free to reopen the issue if you don't agree.
was (Author: fhueske):
OK, I see what you mean.
The difference is that the SQL expression is validated by Calcite and the Table
API expression by the Table API. The SQL validation actually succeeds but
Calcite also tries to simplify the expression by evaluating it because all
input parameters are constants. This means that the function is invoked on the
constant parameters. If you change the test SQL call to
{{testSqlApi("rpad(f8,-1,'')", "null")}} ({{f8}} is a String field and hence
not constant), you'll find that it is executed just like the Table API test
method.
In general, this error is not a validation but a runtime error, because it is
not caused by invalid types but by invalid values at runtime.
>From my point of view, this is all expected behavior. I'd suggest to close
>this issue.
> TableApi validation test in ScalarFunctionsValidationTest doesn't work
> --
>
> Key: FLINK-8930
> URL: https://issues.apache.org/jira/browse/FLINK-8930
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Reporter: Wind
>Priority: Major
>
> I'm wring a validation test for
> [FLINK-6924|https://issues.apache.org/jira/browse/FLINK-6924] in
> org.apache.flink.table.expressions.validation.ScalarFunctionsValidationTest.
> However, I find that the table api is not truely executed in function
> "testTableApi", which is different from "testSqlApi". So we can only test
> exceptions which are thrown in "addTableApiTestExpr" like
> "ValidationException" because it is thrown during "select" operation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-8967) Port NetworkStackThroughputITCase to flip6
Chesnay Schepler created FLINK-8967: --- Summary: Port NetworkStackThroughputITCase to flip6 Key: FLINK-8967 URL: https://issues.apache.org/jira/browse/FLINK-8967 Project: Flink Issue Type: Improvement Components: Tests Affects Versions: 1.5.0 Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8966) Port AvroExternalJarProgramITCase to flip6
Chesnay Schepler created FLINK-8966: --- Summary: Port AvroExternalJarProgramITCase to flip6 Key: FLINK-8966 URL: https://issues.apache.org/jira/browse/FLINK-8966 Project: Flink Issue Type: Improvement Components: Tests Affects Versions: 1.5.0 Reporter: Chesnay Schepler Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8965) Port TimestampITCase to flip6
Chesnay Schepler created FLINK-8965: --- Summary: Port TimestampITCase to flip6 Key: FLINK-8965 URL: https://issues.apache.org/jira/browse/FLINK-8965 Project: Flink Issue Type: Improvement Components: Tests Affects Versions: 1.5.0 Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-6895) Add STR_TO_DATE supported in SQL
[
https://issues.apache.org/jira/browse/FLINK-6895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400529#comment-16400529
]
Timo Walther commented on FLINK-6895:
-
No, it is the first case that the return type depends on a parameter content.
How does MySQL deal with this? Does the format must be a constant string
literal or does it also allow column references at this position? We could
return a timestamp if the parameter is a non-literal and otherwise return the
correct type.
> Add STR_TO_DATE supported in SQL
>
>
> Key: FLINK-6895
> URL: https://issues.apache.org/jira/browse/FLINK-6895
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Affects Versions: 1.4.0
>Reporter: sunjincheng
>Assignee: Aegeaner
>Priority: Major
>
> STR_TO_DATE(str,format) This is the inverse of the DATE_FORMAT() function. It
> takes a string str and a format string format. STR_TO_DATE() returns a
> DATETIME value if the format string contains both date and time parts, or a
> DATE or TIME value if the string contains only date or time parts. If the
> date, time, or datetime value extracted from str is illegal, STR_TO_DATE()
> returns NULL and produces a warning.
> * Syntax:
> STR_TO_DATE(str,format)
> * Arguments
> **str: -
> **format: -
> * Return Types
> DATAETIME/DATE/TIME
> * Example:
> STR_TO_DATE('01,5,2013','%d,%m,%Y') -> '2013-05-01'
> SELECT STR_TO_DATE('a09:30:17','a%h:%i:%s') -> '09:30:17'
> * See more:
> ** [MySQL|
> https://dev.mysql.com/doc/refman/5.7/en/date-and-time-functions.html#function_str-to-date]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-8963) Port BigUserProgramJobSubmitITCase to flip6
Chesnay Schepler created FLINK-8963: --- Summary: Port BigUserProgramJobSubmitITCase to flip6 Key: FLINK-8963 URL: https://issues.apache.org/jira/browse/FLINK-8963 Project: Flink Issue Type: Improvement Components: Job-Submission, Tests Affects Versions: 1.5.0 Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8964) Port JobSubmissionFailsITCase to flip6
Chesnay Schepler created FLINK-8964: --- Summary: Port JobSubmissionFailsITCase to flip6 Key: FLINK-8964 URL: https://issues.apache.org/jira/browse/FLINK-8964 Project: Flink Issue Type: Improvement Components: Tests Affects Versions: 1.5.0 Reporter: Chesnay Schepler Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8962) Port AccumulatorErrorITCase to flip6
Chesnay Schepler created FLINK-8962: --- Summary: Port AccumulatorErrorITCase to flip6 Key: FLINK-8962 URL: https://issues.apache.org/jira/browse/FLINK-8962 Project: Flink Issue Type: Improvement Components: Tests Affects Versions: 1.5.0 Reporter: Chesnay Schepler Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8961) Port JobRetrievalITCase to flip6
Chesnay Schepler created FLINK-8961: --- Summary: Port JobRetrievalITCase to flip6 Key: FLINK-8961 URL: https://issues.apache.org/jira/browse/FLINK-8961 Project: Flink Issue Type: Improvement Components: Tests Affects Versions: 1.5.0 Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-8702) Migrate tests from FlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler reassigned FLINK-8702: --- Assignee: Chesnay Schepler > Migrate tests from FlinkMiniCluster to MiniClusterResource > -- > > Key: FLINK-8702 > URL: https://issues.apache.org/jira/browse/FLINK-8702 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8959) Port AccumulatorLiveITCase to flip6
Chesnay Schepler created FLINK-8959: --- Summary: Port AccumulatorLiveITCase to flip6 Key: FLINK-8959 URL: https://issues.apache.org/jira/browse/FLINK-8959 Project: Flink Issue Type: Improvement Components: Tests Affects Versions: 1.5.0 Reporter: Chesnay Schepler Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8960) Port SavepointITCase to flip6
Chesnay Schepler created FLINK-8960: --- Summary: Port SavepointITCase to flip6 Key: FLINK-8960 URL: https://issues.apache.org/jira/browse/FLINK-8960 Project: Flink Issue Type: Improvement Components: Tests Affects Versions: 1.5.0 Reporter: Chesnay Schepler Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8958) Port TaskCancelAsyncProducerConsumerITCase to flip6
Chesnay Schepler created FLINK-8958: --- Summary: Port TaskCancelAsyncProducerConsumerITCase to flip6 Key: FLINK-8958 URL: https://issues.apache.org/jira/browse/FLINK-8958 Project: Flink Issue Type: Improvement Components: Tests Affects Versions: 1.5.0 Reporter: Chesnay Schepler Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8957) Port JMXJobManagerMetricTest to flip6
Chesnay Schepler created FLINK-8957: --- Summary: Port JMXJobManagerMetricTest to flip6 Key: FLINK-8957 URL: https://issues.apache.org/jira/browse/FLINK-8957 Project: Flink Issue Type: Improvement Components: Tests Affects Versions: 1.5.0 Reporter: Chesnay Schepler Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8956) Port RescalingITCase to flip6
Chesnay Schepler created FLINK-8956: --- Summary: Port RescalingITCase to flip6 Key: FLINK-8956 URL: https://issues.apache.org/jira/browse/FLINK-8956 Project: Flink Issue Type: Improvement Components: Tests Affects Versions: 1.5.0 Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8955) Port ClassLoaderITCase to flip6
Chesnay Schepler created FLINK-8955: --- Summary: Port ClassLoaderITCase to flip6 Key: FLINK-8955 URL: https://issues.apache.org/jira/browse/FLINK-8955 Project: Flink Issue Type: Improvement Components: Tests Affects Versions: 1.5.0 Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8948) RescalingITCase fails on Travis
[
https://issues.apache.org/jira/browse/FLINK-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400499#comment-16400499
]
Piotr Nowojski commented on FLINK-8948:
---
I will take a look at it
> RescalingITCase fails on Travis
> ---
>
> Key: FLINK-8948
> URL: https://issues.apache.org/jira/browse/FLINK-8948
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing, Tests, Travis
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
> https://travis-ci.org/apache/flink/jobs/353468272
> {code}
> testSavepointRescalingInKeyedStateDerivedMaxParallelism[0](org.apache.flink.test.checkpointing.RescalingITCase)
> Time elapsed: 1.858 sec <<< ERROR!
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply$mcV$sp(JobManager.scala:891)
> at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:834)
> at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:834)
> at
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
> at
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: java.lang.IllegalStateException: null
> at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:179)
> at
> org.apache.flink.runtime.io.network.buffer.BufferBuilder.finish(BufferBuilder.java:105)
> at
> org.apache.flink.runtime.io.network.api.writer.RecordWriter.closeBufferBuilder(RecordWriter.java:218)
> at
> org.apache.flink.runtime.io.network.api.writer.RecordWriter.clearBuffers(RecordWriter.java:178)
> at
> org.apache.flink.streaming.runtime.io.StreamRecordWriter.close(StreamRecordWriter.java:99)
> at
> org.apache.flink.streaming.runtime.io.RecordWriterOutput.close(RecordWriterOutput.java:161)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.releaseOutputs(OperatorChain.java:239)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:402)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:703)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (FLINK-8948) RescalingITCase fails on Travis
[
https://issues.apache.org/jira/browse/FLINK-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski reassigned FLINK-8948:
-
Assignee: Piotr Nowojski
> RescalingITCase fails on Travis
> ---
>
> Key: FLINK-8948
> URL: https://issues.apache.org/jira/browse/FLINK-8948
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing, Tests, Travis
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Assignee: Piotr Nowojski
>Priority: Blocker
> Fix For: 1.5.0
>
>
> https://travis-ci.org/apache/flink/jobs/353468272
> {code}
> testSavepointRescalingInKeyedStateDerivedMaxParallelism[0](org.apache.flink.test.checkpointing.RescalingITCase)
> Time elapsed: 1.858 sec <<< ERROR!
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply$mcV$sp(JobManager.scala:891)
> at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:834)
> at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:834)
> at
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
> at
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: java.lang.IllegalStateException: null
> at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:179)
> at
> org.apache.flink.runtime.io.network.buffer.BufferBuilder.finish(BufferBuilder.java:105)
> at
> org.apache.flink.runtime.io.network.api.writer.RecordWriter.closeBufferBuilder(RecordWriter.java:218)
> at
> org.apache.flink.runtime.io.network.api.writer.RecordWriter.clearBuffers(RecordWriter.java:178)
> at
> org.apache.flink.streaming.runtime.io.StreamRecordWriter.close(StreamRecordWriter.java:99)
> at
> org.apache.flink.streaming.runtime.io.RecordWriterOutput.close(RecordWriterOutput.java:161)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.releaseOutputs(OperatorChain.java:239)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:402)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:703)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-6895) Add STR_TO_DATE supported in SQL
[
https://issues.apache.org/jira/browse/FLINK-6895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400470#comment-16400470
]
Fabian Hueske commented on FLINK-6895:
--
Returning a superclass does not work because Flink generates de/serializers
based on the type. We would lose information that way.
I don't think we had a case yet where the return type of a function depends on
the value of a parameter (in contrast to the type of a parameter).
[~twalthr] any ideas?
> Add STR_TO_DATE supported in SQL
>
>
> Key: FLINK-6895
> URL: https://issues.apache.org/jira/browse/FLINK-6895
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Affects Versions: 1.4.0
>Reporter: sunjincheng
>Assignee: Aegeaner
>Priority: Major
>
> STR_TO_DATE(str,format) This is the inverse of the DATE_FORMAT() function. It
> takes a string str and a format string format. STR_TO_DATE() returns a
> DATETIME value if the format string contains both date and time parts, or a
> DATE or TIME value if the string contains only date or time parts. If the
> date, time, or datetime value extracted from str is illegal, STR_TO_DATE()
> returns NULL and produces a warning.
> * Syntax:
> STR_TO_DATE(str,format)
> * Arguments
> **str: -
> **format: -
> * Return Types
> DATAETIME/DATE/TIME
> * Example:
> STR_TO_DATE('01,5,2013','%d,%m,%Y') -> '2013-05-01'
> SELECT STR_TO_DATE('a09:30:17','a%h:%i:%s') -> '09:30:17'
> * See more:
> ** [MySQL|
> https://dev.mysql.com/doc/refman/5.7/en/date-and-time-functions.html#function_str-to-date]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-8562) Fix YARNSessionFIFOSecuredITCase
[ https://issues.apache.org/jira/browse/FLINK-8562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated FLINK-8562: Fix Version/s: 1.5.0 > Fix YARNSessionFIFOSecuredITCase > > > Key: FLINK-8562 > URL: https://issues.apache.org/jira/browse/FLINK-8562 > Project: Flink > Issue Type: Bug > Components: Security >Affects Versions: 1.5.0, 1.6.0 >Reporter: Shuyi Chen >Assignee: Shuyi Chen >Priority: Blocker > Fix For: 1.5.0 > > > Currently, YARNSessionFIFOSecuredITCase will not fail even if the current > Flink YARN Kerberos integration is failing in production. Please see > FLINK-8275. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8948) RescalingITCase fails on Travis
[
https://issues.apache.org/jira/browse/FLINK-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400467#comment-16400467
]
Aljoscha Krettek commented on FLINK-8948:
-
[~NicoK] & [~pnowojski] Seems like the network stack could be involved.
> RescalingITCase fails on Travis
> ---
>
> Key: FLINK-8948
> URL: https://issues.apache.org/jira/browse/FLINK-8948
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing, Tests, Travis
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
> https://travis-ci.org/apache/flink/jobs/353468272
> {code}
> testSavepointRescalingInKeyedStateDerivedMaxParallelism[0](org.apache.flink.test.checkpointing.RescalingITCase)
> Time elapsed: 1.858 sec <<< ERROR!
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply$mcV$sp(JobManager.scala:891)
> at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:834)
> at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:834)
> at
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
> at
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: java.lang.IllegalStateException: null
> at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:179)
> at
> org.apache.flink.runtime.io.network.buffer.BufferBuilder.finish(BufferBuilder.java:105)
> at
> org.apache.flink.runtime.io.network.api.writer.RecordWriter.closeBufferBuilder(RecordWriter.java:218)
> at
> org.apache.flink.runtime.io.network.api.writer.RecordWriter.clearBuffers(RecordWriter.java:178)
> at
> org.apache.flink.streaming.runtime.io.StreamRecordWriter.close(StreamRecordWriter.java:99)
> at
> org.apache.flink.streaming.runtime.io.RecordWriterOutput.close(RecordWriterOutput.java:161)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.releaseOutputs(OperatorChain.java:239)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:402)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:703)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-8948) RescalingITCase fails on Travis
[
https://issues.apache.org/jira/browse/FLINK-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek updated FLINK-8948:
Fix Version/s: 1.5.0
> RescalingITCase fails on Travis
> ---
>
> Key: FLINK-8948
> URL: https://issues.apache.org/jira/browse/FLINK-8948
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing, Tests, Travis
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
> https://travis-ci.org/apache/flink/jobs/353468272
> {code}
> testSavepointRescalingInKeyedStateDerivedMaxParallelism[0](org.apache.flink.test.checkpointing.RescalingITCase)
> Time elapsed: 1.858 sec <<< ERROR!
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply$mcV$sp(JobManager.scala:891)
> at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:834)
> at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:834)
> at
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
> at
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: java.lang.IllegalStateException: null
> at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:179)
> at
> org.apache.flink.runtime.io.network.buffer.BufferBuilder.finish(BufferBuilder.java:105)
> at
> org.apache.flink.runtime.io.network.api.writer.RecordWriter.closeBufferBuilder(RecordWriter.java:218)
> at
> org.apache.flink.runtime.io.network.api.writer.RecordWriter.clearBuffers(RecordWriter.java:178)
> at
> org.apache.flink.streaming.runtime.io.StreamRecordWriter.close(StreamRecordWriter.java:99)
> at
> org.apache.flink.streaming.runtime.io.RecordWriterOutput.close(RecordWriterOutput.java:161)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.releaseOutputs(OperatorChain.java:239)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:402)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:703)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8952) Support setting the parallelism of individual operators of a query
[
https://issues.apache.org/jira/browse/FLINK-8952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400438#comment-16400438
]
Fabian Hueske commented on FLINK-8952:
--
Yes, they are definitely related. FLINK-8236 is explicitly about the Table API,
while this issue is about all types of queries.
The approach I proposed in this issue is more generic, but wouldn't be as
nicely integrated as a pure Table API solution. At the same time, I think my
approach would be easier to expose to external tools.
> Support setting the parallelism of individual operators of a query
> --
>
> Key: FLINK-8952
> URL: https://issues.apache.org/jira/browse/FLINK-8952
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Reporter: Fabian Hueske
>Priority: Major
>
> Right now it is not possible to set the parallelism of individual operators
> that were generated for a SQL or Table API query.
> We could expose the optimized plan before it is translated to a
> {{DataStream}} or {{DataSet}} program to annotate operators with parallelism.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-8954) Escape control characters when outputting on SQL Client CLI
Timo Walther created FLINK-8954: --- Summary: Escape control characters when outputting on SQL Client CLI Key: FLINK-8954 URL: https://issues.apache.org/jira/browse/FLINK-8954 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: Timo Walther Control characters in the result output of a SQL query influence the behavior of the CLI. We should escape everything that could cause side effects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-8954) Escape control characters when outputting on SQL Client CLI
[ https://issues.apache.org/jira/browse/FLINK-8954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timo Walther updated FLINK-8954: Issue Type: Sub-task (was: Improvement) Parent: FLINK-7594 > Escape control characters when outputting on SQL Client CLI > --- > > Key: FLINK-8954 > URL: https://issues.apache.org/jira/browse/FLINK-8954 > Project: Flink > Issue Type: Sub-task > Components: Table API & SQL >Reporter: Timo Walther >Priority: Major > > Control characters in the result output of a SQL query influence the behavior > of the CLI. We should escape everything that could cause side effects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-8521) FlinkKafkaProducer011ITCase.testRunOutOfProducersInThePool timed out on Travis
[
https://issues.apache.org/jira/browse/FLINK-8521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski updated FLINK-8521:
--
Priority: Critical (was: Blocker)
> FlinkKafkaProducer011ITCase.testRunOutOfProducersInThePool timed out on Travis
> --
>
> Key: FLINK-8521
> URL: https://issues.apache.org/jira/browse/FLINK-8521
> Project: Flink
> Issue Type: Bug
> Components: Kafka Connector, Tests
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Piotr Nowojski
>Priority: Critical
> Labels: test-stability
> Fix For: 1.5.0, 1.4.3
>
>
> The {{FlinkKafkaProducer011ITCase.testRunOutOfProducersInThePool}} timed out
> on Travis with producing no output for longer than 300s.
>
> https://travis-ci.org/tillrohrmann/flink/jobs/334642014
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8521) FlinkKafkaProducer011ITCase.testRunOutOfProducersInThePool timed out on Travis
[
https://issues.apache.org/jira/browse/FLINK-8521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400410#comment-16400410
]
Piotr Nowojski commented on FLINK-8521:
---
This for 99% is another deadlock in Kafka brokers. Symptoms are identical to
previously found dead lock: https://issues.apache.org/jira/browse/KAFKA-6042 .
I'm trying to reproduce it outside of the Flink, but as usual with dead locks
(especially in external systems) it might prove difficult to
reproduce/trackdown. Another idea might be upgrading to Kafka 1.0.0.
Never the less I don't think this is a release blocker.
> FlinkKafkaProducer011ITCase.testRunOutOfProducersInThePool timed out on Travis
> --
>
> Key: FLINK-8521
> URL: https://issues.apache.org/jira/browse/FLINK-8521
> Project: Flink
> Issue Type: Bug
> Components: Kafka Connector, Tests
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Piotr Nowojski
>Priority: Blocker
> Labels: test-stability
> Fix For: 1.5.0, 1.4.3
>
>
> The {{FlinkKafkaProducer011ITCase.testRunOutOfProducersInThePool}} timed out
> on Travis with producing no output for longer than 300s.
>
> https://travis-ci.org/tillrohrmann/flink/jobs/334642014
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (FLINK-8855) SQL client result serving gets stuck in result-mode=table
[
https://issues.apache.org/jira/browse/FLINK-8855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400385#comment-16400385
]
Fabian Hueske edited comment on FLINK-8855 at 3/15/18 1:19 PM:
---
I ran a simple {{SELECT a, COUNT(* ) FROM x GROUP BY a}} on a Table backed by a
Kafka topic (filled with ~3GB) in {{result-mode=table}}.
After sometime, the query result wasn't updated anymore.
I didn't look deeper into this.
was (Author: fhueske):
I ran a simple {{SELECT a, COUNT(*) FROM x GROUP BY a}} on a Table backed by a
Kafka topic (filled with ~3GB) in {{result-mode=table}}.
After sometime, the query result wasn't updated anymore.
I didn't look deeper into this.
> SQL client result serving gets stuck in result-mode=table
> -
>
> Key: FLINK-8855
> URL: https://issues.apache.org/jira/browse/FLINK-8855
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.5.0
>Reporter: Fabian Hueske
>Priority: Major
> Fix For: 1.5.0
>
>
> The result serving of a query in {{result-mode=table}} get stuck after some
> time when serving an updating result.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8855) SQL client result serving gets stuck in result-mode=table
[
https://issues.apache.org/jira/browse/FLINK-8855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400385#comment-16400385
]
Fabian Hueske commented on FLINK-8855:
--
I ran a simple {{SELECT a, COUNT(*) FROM x GROUP BY a}} on a Table backed by a
Kafka topic (filled with ~3GB) in {{result-mode=table}}.
After sometime, the query result wasn't updated anymore.
I didn't look deeper into this.
> SQL client result serving gets stuck in result-mode=table
> -
>
> Key: FLINK-8855
> URL: https://issues.apache.org/jira/browse/FLINK-8855
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.5.0
>Reporter: Fabian Hueske
>Priority: Major
> Fix For: 1.5.0
>
>
> The result serving of a query in {{result-mode=table}} get stuck after some
> time when serving an updating result.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8952) Support setting the parallelism of individual operators of a query
[
https://issues.apache.org/jira/browse/FLINK-8952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400350#comment-16400350
]
Xingcan Cui commented on FLINK-8952:
Hi [~fhueske], thanks for this ticket and I suppose it's related FLINK-8236.
What do you think of their relationships? BTW, I had a [basic
version|https://github.com/xccui/flink/tree/FLINK-8236] of FLINK-8236, but did
not create a PR then. Do you think it's necessary to merge these two issues?
> Support setting the parallelism of individual operators of a query
> --
>
> Key: FLINK-8952
> URL: https://issues.apache.org/jira/browse/FLINK-8952
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Reporter: Fabian Hueske
>Priority: Major
>
> Right now it is not possible to set the parallelism of individual operators
> that were generated for a SQL or Table API query.
> We could expose the optimized plan before it is translated to a
> {{DataStream}} or {{DataSet}} program to annotate operators with parallelism.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8949) Rest API failure with long URL
[
https://issues.apache.org/jira/browse/FLINK-8949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400326#comment-16400326
]
Chesnay Schepler commented on FLINK-8949:
-
huh, well that's a problem.
Unfortunately the UI actually requires the watermark for each subtask,
otherwise we could've just queried the aggregated minimum instead.
The metric querying was always a bit verbose but I don't see us reworking that
anytime soon; although with the aggregating handlers we conceptually know how
to do it,
I guess we'll just have to batch the requests in multiples of 50(?) or
something.
> Rest API failure with long URL
> --
>
> Key: FLINK-8949
> URL: https://issues.apache.org/jira/browse/FLINK-8949
> Project: Flink
> Issue Type: Bug
> Components: REST, Webfrontend
>Affects Versions: 1.5.0, 1.4.2
>Reporter: Truong Duc Kien
>Priority: Major
>
> When you have jobs with high parallelism, the URL for a REST request can get
> very long. When the URL is longer than 4096 bytes, the REST API will return
> error
> {{Failure: 404 Not Found}}
> This can easily be seen in the Web UI, when Flink queries for the watermark
> using the REST API:
> {{GET
> /jobs/:jobId/vertices/:vertexId/metrics?get=0.currentLowWatermark,1.currentLowWatermark,2.currentLo...}}
> The request will fail with more than 170 subtasks and the watermark will not
> be displayed in the Web UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (FLINK-8949) Rest API failure with long URL
[
https://issues.apache.org/jira/browse/FLINK-8949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400326#comment-16400326
]
Chesnay Schepler edited comment on FLINK-8949 at 3/15/18 12:41 PM:
---
huh, well that's a problem.
Unfortunately the UI actually requires the watermark for each subtask,
otherwise we could've just queried the aggregated minimum instead.
The metric querying was always a bit verbose but I don't see us reworking that
anytime soon; although with the aggregating handlers we conceptually know how
to do it,
I guess we'll just have to batch the requests in multiples of 50\(?) or
something.
was (Author: zentol):
huh, well that's a problem.
Unfortunately the UI actually requires the watermark for each subtask,
otherwise we could've just queried the aggregated minimum instead.
The metric querying was always a bit verbose but I don't see us reworking that
anytime soon; although with the aggregating handlers we conceptually know how
to do it,
I guess we'll just have to batch the requests in multiples of 50(?) or
something.
> Rest API failure with long URL
> --
>
> Key: FLINK-8949
> URL: https://issues.apache.org/jira/browse/FLINK-8949
> Project: Flink
> Issue Type: Bug
> Components: REST, Webfrontend
>Affects Versions: 1.5.0, 1.4.2
>Reporter: Truong Duc Kien
>Priority: Major
>
> When you have jobs with high parallelism, the URL for a REST request can get
> very long. When the URL is longer than 4096 bytes, the REST API will return
> error
> {{Failure: 404 Not Found}}
> This can easily be seen in the Web UI, when Flink queries for the watermark
> using the REST API:
> {{GET
> /jobs/:jobId/vertices/:vertexId/metrics?get=0.currentLowWatermark,1.currentLowWatermark,2.currentLo...}}
> The request will fail with more than 170 subtasks and the watermark will not
> be displayed in the Web UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (FLINK-6895) Add STR_TO_DATE supported in SQL
[
https://issues.apache.org/jira/browse/FLINK-6895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400182#comment-16400182
]
Fernando Díaz edited comment on FLINK-6895 at 3/15/18 12:27 PM:
Taking a look at this [MySQL
STR_TO_DATE|https://dev.mysql.com/doc/refman/5.5/en/date-and-time-functions.html#function_str-to-date]
it seems that the correct approach would be:
* Return a DATETIME ({{java.sql.Timestamp}}) value if the format string
contains both date and time parts.
* Return a DATE ({{java.sql.Date}}) if the string contains date only.
* Return a TIME ({{java.sql.Time}}) if the string contains time only.
{{java.util.Date}} it's a superclass of these three classes (see [Class
Date|https://docs.oracle.com/javase/7/docs/api/java/util/Date.html]). What do
you think? Maybe it's a good idea to use this as the return type.
was (Author: fdiazgon):
Taking a look at this [MySQL
STR_TO_DATE]https://dev.mysql.com/doc/refman/5.5/en/date-and-time-functions.html#function_str-to-date]
it seems that the correct approach would be:
* Return a DATETIME ({{java.sql.Timestamp}}) value if the format string
contains both date and time parts.
* Return a DATE ({{java.sql.Date}}) if the string contains date only.
* Return a TIME ({{java.sql.Time}}) if the string contains time only.
{{java.util.Date}} it's a superclass of these three classes (see [Class
Date|https://docs.oracle.com/javase/7/docs/api/java/util/Date.html]). What do
you think? Maybe it's a good idea to use this as the return type.
> Add STR_TO_DATE supported in SQL
>
>
> Key: FLINK-6895
> URL: https://issues.apache.org/jira/browse/FLINK-6895
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Affects Versions: 1.4.0
>Reporter: sunjincheng
>Assignee: Aegeaner
>Priority: Major
>
> STR_TO_DATE(str,format) This is the inverse of the DATE_FORMAT() function. It
> takes a string str and a format string format. STR_TO_DATE() returns a
> DATETIME value if the format string contains both date and time parts, or a
> DATE or TIME value if the string contains only date or time parts. If the
> date, time, or datetime value extracted from str is illegal, STR_TO_DATE()
> returns NULL and produces a warning.
> * Syntax:
> STR_TO_DATE(str,format)
> * Arguments
> **str: -
> **format: -
> * Return Types
> DATAETIME/DATE/TIME
> * Example:
> STR_TO_DATE('01,5,2013','%d,%m,%Y') -> '2013-05-01'
> SELECT STR_TO_DATE('a09:30:17','a%h:%i:%s') -> '09:30:17'
> * See more:
> ** [MySQL|
> https://dev.mysql.com/doc/refman/5.7/en/date-and-time-functions.html#function_str-to-date]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-8949) Rest API failure with long URL
[
https://issues.apache.org/jira/browse/FLINK-8949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-8949:
Affects Version/s: 1.5.0
> Rest API failure with long URL
> --
>
> Key: FLINK-8949
> URL: https://issues.apache.org/jira/browse/FLINK-8949
> Project: Flink
> Issue Type: Bug
> Components: REST, Webfrontend
>Affects Versions: 1.5.0, 1.4.2
>Reporter: Truong Duc Kien
>Priority: Major
>
> When you have jobs with high parallelism, the URL for a REST request can get
> very long. When the URL is longer than 4096 bytes, the REST API will return
> error
> {{Failure: 404 Not Found}}
> This can easily be seen in the Web UI, when Flink queries for the watermark
> using the REST API:
> {{GET
> /jobs/:jobId/vertices/:vertexId/metrics?get=0.currentLowWatermark,1.currentLowWatermark,2.currentLo...}}
> The request will fail with more than 170 subtasks and the watermark will not
> be displayed in the Web UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-8953) Resolve unresolved field references in FieldComputer expressions
Timo Walther created FLINK-8953:
---
Summary: Resolve unresolved field references in FieldComputer
expressions
Key: FLINK-8953
URL: https://issues.apache.org/jira/browse/FLINK-8953
Project: Flink
Issue Type: Improvement
Components: Table API & SQL
Reporter: Timo Walther
When implementing the {{FieldComputer.getExpression}} method, it is not
possible to use API classes but only internal expression case classes.
It would be great to also define timestamp extractors like:
{code}
def getExpression(fieldAccesses: Array[ResolvedFieldReference]): Expression =
{
// 'x.cast(Types.LONG)
// ExpressionParser.parseExpression("x.cast(LONG)")
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8906) Flip6DefaultCLI is not tested in org.apache.flink.client.cli tests
[
https://issues.apache.org/jira/browse/FLINK-8906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400258#comment-16400258
]
ASF GitHub Bot commented on FLINK-8906:
---
Github user NicoK commented on the issue:
https://github.com/apache/flink/pull/5671
Thanks for the review - indeed I copied that part too easily. Fixed now -
let's wait for travis to approve the changes though.
> Flip6DefaultCLI is not tested in org.apache.flink.client.cli tests
> --
>
> Key: FLINK-8906
> URL: https://issues.apache.org/jira/browse/FLINK-8906
> Project: Flink
> Issue Type: Bug
> Components: Client, Tests
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Blocker
> Labels: flip-6
> Fix For: 1.5.0, 1.6.0
>
>
> Various tests in {{org.apache.flink.client.cli}} only test with the
> {{DefaultCLI}} but should also test {{Flip6DefaultCLI}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink issue #5671: [FLINK-8906][flip6][tests] also test Flip6DefaultCLI in o...
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/5671 Thanks for the review - indeed I copied that part too easily. Fixed now - let's wait for travis to approve the changes though. ---
[jira] [Commented] (FLINK-8888) Upgrade AWS SDK in flink-connector-kinesis
[ https://issues.apache.org/jira/browse/FLINK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400252#comment-16400252 ] ASF GitHub Bot commented on FLINK-: --- Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/5663 Thanks for checking this out. Merging to 1.5 and 1.6 then... > Upgrade AWS SDK in flink-connector-kinesis > -- > > Key: FLINK- > URL: https://issues.apache.org/jira/browse/FLINK- > Project: Flink > Issue Type: Improvement >Reporter: Kailash Hassan Dayanand >Priority: Minor > > Bump up the java aws sdk version to 1.11.272. Evaluate also the impact of > this version upgrade for KCL and KPL versions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
