[jira] [Created] (FLINK-15323) SQL Client end-to-end test (Old planner) failed on travis
Yu Li created FLINK-15323:
-
Summary: SQL Client end-to-end test (Old planner) failed on travis
Key: FLINK-15323
URL: https://issues.apache.org/jira/browse/FLINK-15323
Project: Flink
Issue Type: Bug
Components: Table SQL / Legacy Planner
Affects Versions: 1.10.0
Reporter: Yu Li
Fix For: 1.10.0
As titled, the 'SQL Client end-to-end test (Old planner)' case failed with
below error:
{code}
org.apache.flink.table.client.gateway.SqlExecutionException: Invalid SQL update
statement.
at
org.apache.flink.table.client.gateway.local.LocalExecutor.applyUpdate(LocalExecutor.java:697)
at
org.apache.flink.table.client.gateway.local.LocalExecutor.executeUpdateInternal(LocalExecutor.java:576)
at
org.apache.flink.table.client.gateway.local.LocalExecutor.executeUpdate(LocalExecutor.java:527)
at
org.apache.flink.table.client.cli.CliClient.callInsertInto(CliClient.java:535)
at
org.apache.flink.table.client.cli.CliClient.lambda$submitUpdate$0(CliClient.java:231)
at java.util.Optional.map(Optional.java:215)
at
org.apache.flink.table.client.cli.CliClient.submitUpdate(CliClient.java:228)
at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:129)
at org.apache.flink.table.client.SqlClient.start(SqlClient.java:104)
at org.apache.flink.table.client.SqlClient.main(SqlClient.java:178)
Caused by: java.lang.RuntimeException: Error while applying rule
PushProjectIntoTableSourceScanRule, args
[rel#88:FlinkLogicalCalc.LOGICAL(input=RelSubset#87,expr#0..2={inputs},expr#3=IS
NOT NULL($t1),user=$t1,rowtime=$t0,$condition=$t3),
Scan(table:[default_catalog, default_database, JsonSourceTable],
fields:(rowtime, user, event), source:KafkaTableSource(rowtime, user, event))]
at
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:235)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631)
at
org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:327)
at
org.apache.flink.table.plan.Optimizer.runVolcanoPlanner(Optimizer.scala:280)
at
org.apache.flink.table.plan.Optimizer.optimizeLogicalPlan(Optimizer.scala:199)
at
org.apache.flink.table.plan.StreamOptimizer.optimize(StreamOptimizer.scala:66)
at
org.apache.flink.table.planner.StreamPlanner.writeToUpsertSink(StreamPlanner.scala:353)
at
org.apache.flink.table.planner.StreamPlanner.org$apache$flink$table$planner$StreamPlanner$$writeToSink(StreamPlanner.scala:281)
...
Caused by: org.apache.flink.table.api.ValidationException: Rowtime field
'rowtime' has invalid type LocalDateTime. Rowtime attributes must be of type
Timestamp.
at
org.apache.flink.table.sources.TableSourceUtil$$anonfun$3.apply(TableSourceUtil.scala:114)
at
org.apache.flink.table.sources.TableSourceUtil$$anonfun$3.apply(TableSourceUtil.scala:92)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
at
org.apache.flink.table.sources.TableSourceUtil$.computeIndexMapping(TableSourceUtil.scala:92)
{code}
https://api.travis-ci.org/v3/job/626790280/log.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15320) JobManager crashes in the standalone model when cancelling job which subtask' status is scheduled
[
https://issues.apache.org/jira/browse/FLINK-15320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999815#comment-16999815
]
Zhu Zhu commented on FLINK-15320:
-
To summarize, this issue happens when an external job cancel request comes when
the job is still allocating slots.
The root cause is that, the version of the canceled vertices are not
incremented in the case of external job cancel request, and the pending slot
requests are also not canceled in this case, so that the returned slot can be
used to fulfill an outdated deployment, which finally triggers the fatal error.
To fix it, I think we should always increment the version of a vertex before
canceling/failing it. Besides {{JobMaster#cancel}}, there are several other
cases including {{JobMaster#suspend}} and {{ExecutionGraph#failJob}}.
cc [~gjy]
> JobManager crashes in the standalone model when cancelling job which subtask'
> status is scheduled
> -
>
> Key: FLINK-15320
> URL: https://issues.apache.org/jira/browse/FLINK-15320
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: lining
>Assignee: Zhu Zhu
>Priority: Blocker
> Fix For: 1.10.0
>
>
> Use start-cluster.sh to start a standalone cluster, and then submit a job
> from the streaming's example which name is TopSpeedWindowing, parallelism is
> 20. Wait for one minute, cancel the job, jobmanager will crash. The exception
> stack is:
> 2019-12-19 10:12:11,060 ERROR
> org.apache.flink.runtime.util.FatalExitExceptionHandler - FATAL: Thread
> 'flink-akka.actor.default-dispatcher-2' produced an uncaught exception.
> Stopping the process...2019-12-19 10:12:11,060 ERROR
> org.apache.flink.runtime.util.FatalExitExceptionHandler - FATAL: Thread
> 'flink-akka.actor.default-dispatcher-2' produced an uncaught exception.
> Stopping the process...java.util.concurrent.CompletionException:
> java.util.concurrent.CompletionException: java.lang.IllegalStateException:
> Could not assign resource
> org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@583585c6 to
> current execution Attempt #0 (Window(GlobalWindows(), DeltaTrigger,
> TimeEvictor, ComparableAggregator, PassThroughWindowFunction) -> Sink: Print
> to Std. Out (19/20)) @ (unassigned) - [CANCELED]. at
> org.apache.flink.runtime.scheduler.DefaultScheduler.propagateIfNonNull(DefaultScheduler.java:387)
> at
> org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$deployAll$4(DefaultScheduler.java:372)
> at
> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
> at
> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
> at
> org.apache.flink.runtime.concurrent.FutureUtils$WaitingConjunctFuture.handleCompletedFuture(FutureUtils.java:705)
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
> at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at
> org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:170)
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
> at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at
> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.tryFulfillSlotRequestOrMakeAvailable(SlotPoolImpl.java:534)
> at
> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.releaseSingleSlot(SlotPoolImpl.java:479)
> at
> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.releaseSlot(SlotPoolImpl.java:390)
> at
> org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:557)
> at
> org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.releaseChild(SlotSharingManager.java:607)
> at
> org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.access$700(SlotSharingManager.java:352)
> at
> org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:716)
> at
>
[jira] [Updated] (FLINK-15308) Job failed when enable pipelined-shuffle.compression and numberOfTaskSlots > 1
[
https://issues.apache.org/jira/browse/FLINK-15308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhijiang updated FLINK-15308:
-
Fix Version/s: 1.10.0
> Job failed when enable pipelined-shuffle.compression and numberOfTaskSlots > 1
> --
>
> Key: FLINK-15308
> URL: https://issues.apache.org/jira/browse/FLINK-15308
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.10.0
> Environment: $ git log
> commit 4b54da2c67692b1c9d43e1184c00899b0151b3ae
> Author: bowen.li
> Date: Tue Dec 17 17:37:03 2019 -0800
>Reporter: Feng Jiajie
>Assignee: Yingjie Cao
>Priority: Blocker
> Fix For: 1.10.0
>
> Attachments: image-2019-12-19-10-55-30-644.png
>
>
> Job worked well with default flink-conf.yaml with
> pipelined-shuffle.compression:
> {code:java}
> taskmanager.numberOfTaskSlots: 1
> taskmanager.network.pipelined-shuffle.compression.enabled: true
> {code}
> But when I set taskmanager.numberOfTaskSlots to 4 or 6:
> {code:java}
> taskmanager.numberOfTaskSlots: 6
> taskmanager.network.pipelined-shuffle.compression.enabled: true
> {code}
> job failed:
> {code:java}
> $ bin/flink run -m yarn-cluster -p 16 -yjm 1024m -ytm 12288m
> ~/flink-example-1.0-SNAPSHOT.jar
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/data/build/flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/lib/slf4j-log4j12-1.7.15.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/data/sa_cluster/cloudera/parcels/CDH-5.14.4-1.cdh5.14.4.p0.3/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 2019-12-18 15:04:40,514 WARN org.apache.flink.yarn.cli.FlinkYarnSessionCli
> - The configuration directory
> ('/data/build/flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/conf')
> already contains a LOG4J config file.If you want to use logback, then please
> delete or rename the log configuration file.
> 2019-12-18 15:04:40,514 WARN org.apache.flink.yarn.cli.FlinkYarnSessionCli
> - The configuration directory
> ('/data/build/flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/conf')
> already contains a LOG4J config file.If you want to use logback, then please
> delete or rename the log configuration file.
> 2019-12-18 15:04:40,907 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - No path for the flink jar passed. Using the location of class
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
> 2019-12-18 15:04:41,084 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Cluster specification:
> ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=12288,
> numberTaskManagers=1, slotsPerTaskManager=6}
> 2019-12-18 15:04:42,344 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Submitting application master application_1576573857638_0026
> 2019-12-18 15:04:42,370 INFO
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted
> application application_1576573857638_0026
> 2019-12-18 15:04:42,371 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Waiting for the cluster to be allocated
> 2019-12-18 15:04:42,372 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Deploying cluster, current state ACCEPTED
> 2019-12-18 15:04:45,388 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - YARN application has been deployed successfully.
> 2019-12-18 15:04:45,390 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Found Web Interface debugboxcreate431x3.sa:36162 of
> application 'application_1576573857638_0026'.
> Job has been submitted with JobID 9140c70769f4271cc22ea8becaa26272
>
> The program finished with the following exception:
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: org.apache.flink.client.program.ProgramInvocationException:
> Job failed (JobID: 9140c70769f4271cc22ea8becaa26272)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:335)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:205)
> at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:138)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:664)
> at
[GitHub] [flink] lirui-apache opened a new pull request #10625: [FLINK-15259][hive] HiveInspector.toInspectors() should convert Flink…
lirui-apache opened a new pull request #10625: [FLINK-15259][hive] HiveInspector.toInspectors() should convert Flink… URL: https://github.com/apache/flink/pull/10625 … constant to Hive constant ## What is the purpose of the change Convert constant Flink object to Hive object before creating Hive's constant object inspectors. ## Brief change log - Convert the constant value to Hive object in `HiveInspectors::toInspectors` - Implement our own java constant objector inspectors for char, varchar and decimal - Add test case ## Verifying this change New test case ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? NA This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-15259) HiveInspector.toInspectors() should convert Flink constant to Hive constant
[
https://issues.apache.org/jira/browse/FLINK-15259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-15259:
---
Labels: pull-request-available (was: )
> HiveInspector.toInspectors() should convert Flink constant to Hive constant
>
>
> Key: FLINK-15259
> URL: https://issues.apache.org/jira/browse/FLINK-15259
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Bowen Li
>Assignee: Rui Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.9.2, 1.10.0, 1.11.0
>
>
> repro test:
> {code:java}
> public class HiveModuleITCase {
> @Test
> public void test() {
> TableEnvironment tEnv =
> HiveTestUtils.createTableEnvWithBlinkPlannerBatchMode();
> tEnv.unloadModule("core");
> tEnv.loadModule("hive", new HiveModule("2.3.4"));
> tEnv.sqlQuery("select concat('an', 'bn')");
> }
> }
> {code}
> seems that currently HiveInspector.toInspectors() didn't convert Flink
> constant to Hive constant before calling
> hiveShim.getObjectInspectorForConstant
> I don't think it's a blocker
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-15311) Lz4BlockCompressionFactory should use native compressor instead of java unsafe
[ https://issues.apache.org/jira/browse/FLINK-15311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhijiang reassigned FLINK-15311: Assignee: Yingjie Cao > Lz4BlockCompressionFactory should use native compressor instead of java unsafe > -- > > Key: FLINK-15311 > URL: https://issues.apache.org/jira/browse/FLINK-15311 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Reporter: Jingsong Lee >Assignee: Yingjie Cao >Priority: Critical > Fix For: 1.10.0 > > > According to: > [https://lz4.github.io/lz4-java/1.7.0/lz4-compression-benchmark/] > Java java unsafe compressor has lower performance than native lz4 compressor. > After FLINK-14845 , we use lz4 compression for shuffler. > In testing, I found shuffle using java unsafe compressor. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15311) Lz4BlockCompressionFactory should use native compressor instead of java unsafe
[ https://issues.apache.org/jira/browse/FLINK-15311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999797#comment-16999797 ] zhijiang commented on FLINK-15311: -- I guess it belongs to performance improvement, not a bug, because it does not affect the compression function and stable issue. Should it be a blocker for the release? > Lz4BlockCompressionFactory should use native compressor instead of java unsafe > -- > > Key: FLINK-15311 > URL: https://issues.apache.org/jira/browse/FLINK-15311 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Reporter: Jingsong Lee >Priority: Critical > Fix For: 1.10.0 > > > According to: > [https://lz4.github.io/lz4-java/1.7.0/lz4-compression-benchmark/] > Java java unsafe compressor has lower performance than native lz4 compressor. > After FLINK-14845 , we use lz4 compression for shuffler. > In testing, I found shuffle using java unsafe compressor. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15199) Benchmarks are not compiling
[
https://issues.apache.org/jira/browse/FLINK-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999791#comment-16999791
]
Yu Li commented on FLINK-15199:
---
Great to know, thanks for the update! [~pnowojski]
> Benchmarks are not compiling
>
>
> Key: FLINK-15199
> URL: https://issues.apache.org/jira/browse/FLINK-15199
> Project: Flink
> Issue Type: Bug
> Components: Benchmarks
>Affects Versions: 1.10.0
>Reporter: Piotr Nowojski
>Assignee: Piotr Nowojski
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Recent changes in FLINK-14926 caused:
> {noformat}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/jenkins/workspace/flink-master-benchmarks/flink-benchmarks/src/main/java/org/apache/flink/state/benchmark/BackendUtils.java:[61,57]
> cannot infer type arguments for
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder<>
> [INFO] 1 error
> [INFO] -
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 2.635 s
> [INFO] Finished at: 2019-12-11T14:58:37+01:00
> [INFO] Final Memory: 31M/751M
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile)
> on project flink-hackathon-benchmarks: Compilation failure
> [ERROR]
> /home/jenkins/workspace/flink-master-benchmarks/flink-benchmarks/src/main/java/org/apache/flink/state/benchmark/BackendUtils.java:[61,57]
> cannot infer type arguments for
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder<>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15322) Parquet test fails with Hive versions prior to 1.2.0
Rui Li created FLINK-15322: -- Summary: Parquet test fails with Hive versions prior to 1.2.0 Key: FLINK-15322 URL: https://issues.apache.org/jira/browse/FLINK-15322 Project: Flink Issue Type: Test Components: Connectors / Hive Reporter: Rui Li Fix For: 1.10.0 Some data types are not supported by parquet tables in older Hive versions. Related to: https://issues.apache.org/jira/browse/HIVE-6384 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15320) JobManager crashes in the standalone model when cancelling job which subtask' status is scheduled
[ https://issues.apache.org/jira/browse/FLINK-15320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated FLINK-15320: -- Fix Version/s: 1.10.0 > JobManager crashes in the standalone model when cancelling job which subtask' > status is scheduled > - > > Key: FLINK-15320 > URL: https://issues.apache.org/jira/browse/FLINK-15320 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.10.0 >Reporter: lining >Assignee: Zhu Zhu >Priority: Blocker > Fix For: 1.10.0 > > > Use start-cluster.sh to start a standalone cluster, and then submit a job > from the streaming's example which name is TopSpeedWindowing, parallelism is > 20. Wait for one minute, cancel the job, jobmanager will crash. The exception > stack is: > 2019-12-19 10:12:11,060 ERROR > org.apache.flink.runtime.util.FatalExitExceptionHandler - FATAL: Thread > 'flink-akka.actor.default-dispatcher-2' produced an uncaught exception. > Stopping the process...2019-12-19 10:12:11,060 ERROR > org.apache.flink.runtime.util.FatalExitExceptionHandler - FATAL: Thread > 'flink-akka.actor.default-dispatcher-2' produced an uncaught exception. > Stopping the process...java.util.concurrent.CompletionException: > java.util.concurrent.CompletionException: java.lang.IllegalStateException: > Could not assign resource > org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@583585c6 to > current execution Attempt #0 (Window(GlobalWindows(), DeltaTrigger, > TimeEvictor, ComparableAggregator, PassThroughWindowFunction) -> Sink: Print > to Std. Out (19/20)) @ (unassigned) - [CANCELED]. at > org.apache.flink.runtime.scheduler.DefaultScheduler.propagateIfNonNull(DefaultScheduler.java:387) > at > org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$deployAll$4(DefaultScheduler.java:372) > at > java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) > at > java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797) > at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > at > java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) > at > org.apache.flink.runtime.concurrent.FutureUtils$WaitingConjunctFuture.handleCompletedFuture(FutureUtils.java:705) > at > java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) > at > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) > at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > at > java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) > at > org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:170) > at > java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) > at > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) > at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > at > java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.tryFulfillSlotRequestOrMakeAvailable(SlotPoolImpl.java:534) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.releaseSingleSlot(SlotPoolImpl.java:479) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.releaseSlot(SlotPoolImpl.java:390) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:557) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.releaseChild(SlotSharingManager.java:607) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.access$700(SlotSharingManager.java:352) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:716) > at > org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.releaseSharedSlot(SchedulerImpl.java:552) > at > org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.cancelSlotRequest(SchedulerImpl.java:184) > at > org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.returnLogicalSlot(SchedulerImpl.java:195) > at > org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot.lambda$returnSlotToOwner$0(SingleLogicalSlot.java:181) > at > java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) > at > java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778) > at >
[jira] [Assigned] (FLINK-15320) JobManager crashes in the standalone model when cancelling job which subtask' status is scheduled
[ https://issues.apache.org/jira/browse/FLINK-15320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhu Zhu reassigned FLINK-15320: --- Assignee: Zhu Zhu > JobManager crashes in the standalone model when cancelling job which subtask' > status is scheduled > - > > Key: FLINK-15320 > URL: https://issues.apache.org/jira/browse/FLINK-15320 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.10.0 >Reporter: lining >Assignee: Zhu Zhu >Priority: Blocker > > Use start-cluster.sh to start a standalone cluster, and then submit a job > from the streaming's example which name is TopSpeedWindowing, parallelism is > 20. Wait for one minute, cancel the job, jobmanager will crash. The exception > stack is: > 2019-12-19 10:12:11,060 ERROR > org.apache.flink.runtime.util.FatalExitExceptionHandler - FATAL: Thread > 'flink-akka.actor.default-dispatcher-2' produced an uncaught exception. > Stopping the process...2019-12-19 10:12:11,060 ERROR > org.apache.flink.runtime.util.FatalExitExceptionHandler - FATAL: Thread > 'flink-akka.actor.default-dispatcher-2' produced an uncaught exception. > Stopping the process...java.util.concurrent.CompletionException: > java.util.concurrent.CompletionException: java.lang.IllegalStateException: > Could not assign resource > org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@583585c6 to > current execution Attempt #0 (Window(GlobalWindows(), DeltaTrigger, > TimeEvictor, ComparableAggregator, PassThroughWindowFunction) -> Sink: Print > to Std. Out (19/20)) @ (unassigned) - [CANCELED]. at > org.apache.flink.runtime.scheduler.DefaultScheduler.propagateIfNonNull(DefaultScheduler.java:387) > at > org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$deployAll$4(DefaultScheduler.java:372) > at > java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) > at > java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797) > at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > at > java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) > at > org.apache.flink.runtime.concurrent.FutureUtils$WaitingConjunctFuture.handleCompletedFuture(FutureUtils.java:705) > at > java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) > at > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) > at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > at > java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) > at > org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:170) > at > java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) > at > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) > at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > at > java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.tryFulfillSlotRequestOrMakeAvailable(SlotPoolImpl.java:534) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.releaseSingleSlot(SlotPoolImpl.java:479) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.releaseSlot(SlotPoolImpl.java:390) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:557) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.releaseChild(SlotSharingManager.java:607) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.access$700(SlotSharingManager.java:352) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:716) > at > org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.releaseSharedSlot(SchedulerImpl.java:552) > at > org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.cancelSlotRequest(SchedulerImpl.java:184) > at > org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.returnLogicalSlot(SchedulerImpl.java:195) > at > org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot.lambda$returnSlotToOwner$0(SingleLogicalSlot.java:181) > at > java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) > at > java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778) > at > java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140) > at >
[jira] [Commented] (FLINK-15258) HiveModuleFactory should be able to distinguish hive-version as property
[ https://issues.apache.org/jira/browse/FLINK-15258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999762#comment-16999762 ] Bowen Li commented on FLINK-15258: -- [~trohrmann] I thought the original title was pretty self explanatary, it probably can be rephrased as "HiveModuleFactory should be able to distinguish hive-version as property". Thanks for the reminder > HiveModuleFactory should be able to distinguish hive-version as property > > > Key: FLINK-15258 > URL: https://issues.apache.org/jira/browse/FLINK-15258 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.10.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Blocker > Fix For: 1.10.0, 1.11.0 > > > HiveModuleFactory should be able to distinguish hive-version as supported > property. it currently can't -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15258) HiveModuleFactory should be able to distinguish hive-version as property
[ https://issues.apache.org/jira/browse/FLINK-15258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li updated FLINK-15258: - Description: HiveModuleFactory should be able to distinguish hive-version as supported property. it currently can't (was: HiveModuleFactory should be able to distinguish hive-version as supported property) > HiveModuleFactory should be able to distinguish hive-version as property > > > Key: FLINK-15258 > URL: https://issues.apache.org/jira/browse/FLINK-15258 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.10.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Blocker > Fix For: 1.10.0, 1.11.0 > > > HiveModuleFactory should be able to distinguish hive-version as supported > property. it currently can't -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15258) HiveModuleFactory should take hive-version as property
[ https://issues.apache.org/jira/browse/FLINK-15258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li updated FLINK-15258: - Description: HiveModuleFactory should be able to distinguish hive-version as supported property (was: HiveModuleFactory should have hive-version as supported property) > HiveModuleFactory should take hive-version as property > -- > > Key: FLINK-15258 > URL: https://issues.apache.org/jira/browse/FLINK-15258 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.10.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Blocker > Fix For: 1.10.0, 1.11.0 > > > HiveModuleFactory should be able to distinguish hive-version as supported > property -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15258) HiveModuleFactory should be able to distinguish hive-version as property
[ https://issues.apache.org/jira/browse/FLINK-15258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li updated FLINK-15258: - Summary: HiveModuleFactory should be able to distinguish hive-version as property (was: HiveModuleFactory should take hive-version as property) > HiveModuleFactory should be able to distinguish hive-version as property > > > Key: FLINK-15258 > URL: https://issues.apache.org/jira/browse/FLINK-15258 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.10.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Blocker > Fix For: 1.10.0, 1.11.0 > > > HiveModuleFactory should be able to distinguish hive-version as supported > property -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15321) The result of sql(SELECT concat('a', cast(null as varchar), 'c');) is NULL;
[
https://issues.apache.org/jira/browse/FLINK-15321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999722#comment-16999722
]
hehuiyuan edited comment on FLINK-15321 at 12/19/19 3:58 AM:
-
You can try
{{CONCAT_WS()}}
was (Author: hehuiyuan):
You can try
{{CONCAT_WS(string1, string2, string3,...)}}
> The result of sql(SELECT concat('a', cast(null as varchar), 'c');) is NULL;
> ---
>
> Key: FLINK-15321
> URL: https://issues.apache.org/jira/browse/FLINK-15321
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.10.0
>Reporter: xiaojin.wy
>Priority: Major
>
> The sql is :
> SELECT concat('a', cast(null as varchar), 'c');
> The result of the sql is NULL after you execute it in a sqlClient
> environment;But actually the result should be 'ac'.
> The config is:
> execution:
> planner: blink
> type: batch
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15321) The result of sql(SELECT concat('a', cast(null as varchar), 'c');) is NULL;
[
https://issues.apache.org/jira/browse/FLINK-15321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999722#comment-16999722
]
hehuiyuan commented on FLINK-15321:
---
You can try
{{CONCAT_WS(string1, string2, string3,...)}}
> The result of sql(SELECT concat('a', cast(null as varchar), 'c');) is NULL;
> ---
>
> Key: FLINK-15321
> URL: https://issues.apache.org/jira/browse/FLINK-15321
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.10.0
>Reporter: xiaojin.wy
>Priority: Major
>
> The sql is :
> SELECT concat('a', cast(null as varchar), 'c');
> The result of the sql is NULL after you execute it in a sqlClient
> environment;But actually the result should be 'ac'.
> The config is:
> execution:
> planner: blink
> type: batch
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15321) The result of sql(SELECT concat('a', cast(null as varchar), 'c');) is NULL;
[
https://issues.apache.org/jira/browse/FLINK-15321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999718#comment-16999718
]
xiaojin.wy edited comment on FLINK-15321 at 12/19/19 3:50 AM:
--
Ok, I will close it if it's expected.
was (Author: xiaojin.wy):
ok, I will close it if it's expected.
> The result of sql(SELECT concat('a', cast(null as varchar), 'c');) is NULL;
> ---
>
> Key: FLINK-15321
> URL: https://issues.apache.org/jira/browse/FLINK-15321
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.10.0
>Reporter: xiaojin.wy
>Priority: Major
>
> The sql is :
> SELECT concat('a', cast(null as varchar), 'c');
> The result of the sql is NULL after you execute it in a sqlClient
> environment;But actually the result should be 'ac'.
> The config is:
> execution:
> planner: blink
> type: batch
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15321) The result of sql(SELECT concat('a', cast(null as varchar), 'c');) is NULL;
[
https://issues.apache.org/jira/browse/FLINK-15321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999718#comment-16999718
]
xiaojin.wy commented on FLINK-15321:
ok, I will close it if it's expected.
> The result of sql(SELECT concat('a', cast(null as varchar), 'c');) is NULL;
> ---
>
> Key: FLINK-15321
> URL: https://issues.apache.org/jira/browse/FLINK-15321
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.10.0
>Reporter: xiaojin.wy
>Priority: Major
>
> The sql is :
> SELECT concat('a', cast(null as varchar), 'c');
> The result of the sql is NULL after you execute it in a sqlClient
> environment;But actually the result should be 'ac'.
> The config is:
> execution:
> planner: blink
> type: batch
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-15321) The result of sql(SELECT concat('a', cast(null as varchar), 'c');) is NULL;
[
https://issues.apache.org/jira/browse/FLINK-15321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu closed FLINK-15321.
---
Fix Version/s: (was: 1.10.0)
Resolution: Not A Bug
> The result of sql(SELECT concat('a', cast(null as varchar), 'c');) is NULL;
> ---
>
> Key: FLINK-15321
> URL: https://issues.apache.org/jira/browse/FLINK-15321
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.10.0
>Reporter: xiaojin.wy
>Priority: Major
>
> The sql is :
> SELECT concat('a', cast(null as varchar), 'c');
> The result of the sql is NULL after you execute it in a sqlClient
> environment;But actually the result should be 'ac'.
> The config is:
> execution:
> planner: blink
> type: batch
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15321) The result of sql(SELECT concat('a', cast(null as varchar), 'c');) is NULL;
[
https://issues.apache.org/jira/browse/FLINK-15321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999716#comment-16999716
]
Jark Wu commented on FLINK-15321:
-
This is as expected. {{concat}} will return null if any argument is null. see
the documentation [1].
[1]:
https://ci.apache.org/projects/flink/flink-docs-master/dev/table/functions/systemFunctions.html
> The result of sql(SELECT concat('a', cast(null as varchar), 'c');) is NULL;
> ---
>
> Key: FLINK-15321
> URL: https://issues.apache.org/jira/browse/FLINK-15321
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.10.0
>Reporter: xiaojin.wy
>Priority: Major
> Fix For: 1.10.0
>
>
> The sql is :
> SELECT concat('a', cast(null as varchar), 'c');
> The result of the sql is NULL after you execute it in a sqlClient
> environment;But actually the result should be 'ac'.
> The config is:
> execution:
> planner: blink
> type: batch
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15321) The result of sql(SELECT concat('a', cast(null as varchar), 'c');) is NULL;
[
https://issues.apache.org/jira/browse/FLINK-15321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-15321:
Fix Version/s: 1.10.0
> The result of sql(SELECT concat('a', cast(null as varchar), 'c');) is NULL;
> ---
>
> Key: FLINK-15321
> URL: https://issues.apache.org/jira/browse/FLINK-15321
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.10.0
>Reporter: xiaojin.wy
>Priority: Major
> Fix For: 1.10.0
>
>
> The sql is :
> SELECT concat('a', cast(null as varchar), 'c');
> The result of the sql is NULL after you execute it in a sqlClient
> environment;But actually the result should be 'ac'.
> The config is:
> execution:
> planner: blink
> type: batch
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15321) The result of sql(SELECT concat('a', cast(null as varchar), 'c');) is NULL;
[
https://issues.apache.org/jira/browse/FLINK-15321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xiaojin.wy updated FLINK-15321:
---
Description:
The sql is :
SELECT concat('a', cast(null as varchar), 'c');
The result of the sql is NULL after you execute it in a sqlClient
environment;But actually the result should be 'ac'.
The config is:
execution:
planner: blink
type: batch
was:
The sql is :
SELECT concat('a', cast(null as varchar), 'c');
The result of the sql is NULL after you execute it in a sqlClient
environment;But actually the result should be 'ac'.
> The result of sql(SELECT concat('a', cast(null as varchar), 'c');) is NULL;
> ---
>
> Key: FLINK-15321
> URL: https://issues.apache.org/jira/browse/FLINK-15321
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.10.0
>Reporter: xiaojin.wy
>Priority: Major
>
> The sql is :
> SELECT concat('a', cast(null as varchar), 'c');
> The result of the sql is NULL after you execute it in a sqlClient
> environment;But actually the result should be 'ac'.
> The config is:
> execution:
> planner: blink
> type: batch
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15321) The result of sql(SELECT concat('a', cast(null as varchar), 'c');) is NULL;
xiaojin.wy created FLINK-15321:
--
Summary: The result of sql(SELECT concat('a', cast(null as
varchar), 'c');) is NULL;
Key: FLINK-15321
URL: https://issues.apache.org/jira/browse/FLINK-15321
Project: Flink
Issue Type: Bug
Components: Table SQL / Client
Affects Versions: 1.10.0
Reporter: xiaojin.wy
The sql is :
SELECT concat('a', cast(null as varchar), 'c');
The result of the sql is NULL after you execute it in a sqlClient
environment;But actually the result should be 'ac'.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15310) A timestamp result get by a select sql and a csvsink sql is different
[ https://issues.apache.org/jira/browse/FLINK-15310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-15310: Fix Version/s: 1.10.0 > A timestamp result get by a select sql and a csvsink sql is different > - > > Key: FLINK-15310 > URL: https://issues.apache.org/jira/browse/FLINK-15310 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.10.0 > Environment: execution: > planner: blink > type: batch >Reporter: xiaojin.wy >Priority: Major > Fix For: 1.10.0 > > Attachments: image-2019-12-18-16-33-24-510.png > > > *The sql is:* > CREATE TABLE `orders` ( > rowtime TIMESTAMP, > id INT, > product VARCHAR, > units INT > ) WITH ( > 'format.field-delimiter'='|', > 'connector.type'='filesystem', > 'format.derive-schema'='true', > > 'connector.path'='/defender_test_data/daily_regression_blink_sql_1.10/test_agg/sources/orders.csv', > 'format.type'='csv' > ); > select floor(rowtime to hour) as rowtime, count(*) as c from orders group by > floor(rowtime to hour) > The result got in a sqlClient environment which use the sql above is like > this: > !image-2019-12-18-16-33-24-510.png! > But the same sql write directly to a cvs batch sink will get a result like > this: > 1972-03-06 08:44:36.736|4 > 1972-03-06 08:44:36.736|1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15310) A timestamp result get by a select sql and a csvsink sql is different
[ https://issues.apache.org/jira/browse/FLINK-15310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-15310: Priority: Critical (was: Major) > A timestamp result get by a select sql and a csvsink sql is different > - > > Key: FLINK-15310 > URL: https://issues.apache.org/jira/browse/FLINK-15310 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.10.0 > Environment: execution: > planner: blink > type: batch >Reporter: xiaojin.wy >Priority: Critical > Fix For: 1.10.0 > > Attachments: image-2019-12-18-16-33-24-510.png > > > *The sql is:* > CREATE TABLE `orders` ( > rowtime TIMESTAMP, > id INT, > product VARCHAR, > units INT > ) WITH ( > 'format.field-delimiter'='|', > 'connector.type'='filesystem', > 'format.derive-schema'='true', > > 'connector.path'='/defender_test_data/daily_regression_blink_sql_1.10/test_agg/sources/orders.csv', > 'format.type'='csv' > ); > select floor(rowtime to hour) as rowtime, count(*) as c from orders group by > floor(rowtime to hour) > The result got in a sqlClient environment which use the sql above is like > this: > !image-2019-12-18-16-33-24-510.png! > But the same sql write directly to a cvs batch sink will get a result like > this: > 1972-03-06 08:44:36.736|4 > 1972-03-06 08:44:36.736|1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15308) Job failed when enable pipelined-shuffle.compression and numberOfTaskSlots > 1
[
https://issues.apache.org/jira/browse/FLINK-15308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999708#comment-16999708
]
Yingjie Cao commented on FLINK-15308:
-
[~fengjiajie] I also reproduced it.
> Job failed when enable pipelined-shuffle.compression and numberOfTaskSlots > 1
> --
>
> Key: FLINK-15308
> URL: https://issues.apache.org/jira/browse/FLINK-15308
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.10.0
> Environment: $ git log
> commit 4b54da2c67692b1c9d43e1184c00899b0151b3ae
> Author: bowen.li
> Date: Tue Dec 17 17:37:03 2019 -0800
>Reporter: Feng Jiajie
>Assignee: Yingjie Cao
>Priority: Blocker
> Attachments: image-2019-12-19-10-55-30-644.png
>
>
> Job worked well with default flink-conf.yaml with
> pipelined-shuffle.compression:
> {code:java}
> taskmanager.numberOfTaskSlots: 1
> taskmanager.network.pipelined-shuffle.compression.enabled: true
> {code}
> But when I set taskmanager.numberOfTaskSlots to 4 or 6:
> {code:java}
> taskmanager.numberOfTaskSlots: 6
> taskmanager.network.pipelined-shuffle.compression.enabled: true
> {code}
> job failed:
> {code:java}
> $ bin/flink run -m yarn-cluster -p 16 -yjm 1024m -ytm 12288m
> ~/flink-example-1.0-SNAPSHOT.jar
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/data/build/flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/lib/slf4j-log4j12-1.7.15.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/data/sa_cluster/cloudera/parcels/CDH-5.14.4-1.cdh5.14.4.p0.3/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 2019-12-18 15:04:40,514 WARN org.apache.flink.yarn.cli.FlinkYarnSessionCli
> - The configuration directory
> ('/data/build/flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/conf')
> already contains a LOG4J config file.If you want to use logback, then please
> delete or rename the log configuration file.
> 2019-12-18 15:04:40,514 WARN org.apache.flink.yarn.cli.FlinkYarnSessionCli
> - The configuration directory
> ('/data/build/flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/conf')
> already contains a LOG4J config file.If you want to use logback, then please
> delete or rename the log configuration file.
> 2019-12-18 15:04:40,907 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - No path for the flink jar passed. Using the location of class
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
> 2019-12-18 15:04:41,084 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Cluster specification:
> ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=12288,
> numberTaskManagers=1, slotsPerTaskManager=6}
> 2019-12-18 15:04:42,344 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Submitting application master application_1576573857638_0026
> 2019-12-18 15:04:42,370 INFO
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted
> application application_1576573857638_0026
> 2019-12-18 15:04:42,371 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Waiting for the cluster to be allocated
> 2019-12-18 15:04:42,372 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Deploying cluster, current state ACCEPTED
> 2019-12-18 15:04:45,388 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - YARN application has been deployed successfully.
> 2019-12-18 15:04:45,390 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Found Web Interface debugboxcreate431x3.sa:36162 of
> application 'application_1576573857638_0026'.
> Job has been submitted with JobID 9140c70769f4271cc22ea8becaa26272
>
> The program finished with the following exception:
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: org.apache.flink.client.program.ProgramInvocationException:
> Job failed (JobID: 9140c70769f4271cc22ea8becaa26272)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:335)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:205)
> at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:138)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:664)
> at
[jira] [Commented] (FLINK-15308) Job failed when enable pipelined-shuffle.compression and numberOfTaskSlots > 1
[
https://issues.apache.org/jira/browse/FLINK-15308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999706#comment-16999706
]
Feng Jiajie commented on FLINK-15308:
-
Hi [~kevin.cyj] ,
I can reproduce the problem every time.
YARN cluster: 3 node ( 8 core 32GB )
{code:java}
$ cat flink-conf.yaml | grep -v '^#' | grep -v '^$'
jobmanager.rpc.address: localhost
jobmanager.rpc.port: 6123
jobmanager.heap.size: 1024m
taskmanager.memory.total-process.size: 1024m
taskmanager.numberOfTaskSlots: 6
parallelism.default: 1
taskmanager.network.pipelined-shuffle.compression.enabled: true
jobmanager.execution.failover-strategy: region
{code}
> Job failed when enable pipelined-shuffle.compression and numberOfTaskSlots > 1
> --
>
> Key: FLINK-15308
> URL: https://issues.apache.org/jira/browse/FLINK-15308
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.10.0
> Environment: $ git log
> commit 4b54da2c67692b1c9d43e1184c00899b0151b3ae
> Author: bowen.li
> Date: Tue Dec 17 17:37:03 2019 -0800
>Reporter: Feng Jiajie
>Assignee: Yingjie Cao
>Priority: Blocker
> Attachments: image-2019-12-19-10-55-30-644.png
>
>
> Job worked well with default flink-conf.yaml with
> pipelined-shuffle.compression:
> {code:java}
> taskmanager.numberOfTaskSlots: 1
> taskmanager.network.pipelined-shuffle.compression.enabled: true
> {code}
> But when I set taskmanager.numberOfTaskSlots to 4 or 6:
> {code:java}
> taskmanager.numberOfTaskSlots: 6
> taskmanager.network.pipelined-shuffle.compression.enabled: true
> {code}
> job failed:
> {code:java}
> $ bin/flink run -m yarn-cluster -p 16 -yjm 1024m -ytm 12288m
> ~/flink-example-1.0-SNAPSHOT.jar
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/data/build/flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/lib/slf4j-log4j12-1.7.15.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/data/sa_cluster/cloudera/parcels/CDH-5.14.4-1.cdh5.14.4.p0.3/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 2019-12-18 15:04:40,514 WARN org.apache.flink.yarn.cli.FlinkYarnSessionCli
> - The configuration directory
> ('/data/build/flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/conf')
> already contains a LOG4J config file.If you want to use logback, then please
> delete or rename the log configuration file.
> 2019-12-18 15:04:40,514 WARN org.apache.flink.yarn.cli.FlinkYarnSessionCli
> - The configuration directory
> ('/data/build/flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/conf')
> already contains a LOG4J config file.If you want to use logback, then please
> delete or rename the log configuration file.
> 2019-12-18 15:04:40,907 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - No path for the flink jar passed. Using the location of class
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
> 2019-12-18 15:04:41,084 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Cluster specification:
> ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=12288,
> numberTaskManagers=1, slotsPerTaskManager=6}
> 2019-12-18 15:04:42,344 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Submitting application master application_1576573857638_0026
> 2019-12-18 15:04:42,370 INFO
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted
> application application_1576573857638_0026
> 2019-12-18 15:04:42,371 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Waiting for the cluster to be allocated
> 2019-12-18 15:04:42,372 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Deploying cluster, current state ACCEPTED
> 2019-12-18 15:04:45,388 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - YARN application has been deployed successfully.
> 2019-12-18 15:04:45,390 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Found Web Interface debugboxcreate431x3.sa:36162 of
> application 'application_1576573857638_0026'.
> Job has been submitted with JobID 9140c70769f4271cc22ea8becaa26272
>
> The program finished with the following exception:
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: org.apache.flink.client.program.ProgramInvocationException:
> Job failed (JobID:
[jira] [Issue Comment Deleted] (FLINK-15308) Job failed when enable pipelined-shuffle.compression and numberOfTaskSlots > 1
[
https://issues.apache.org/jira/browse/FLINK-15308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yingjie Cao updated FLINK-15308:
Comment: was deleted
(was: [~fengjiajie] I can not reproduce the problem in my test environment. Is
there any other settings?
!image-2019-12-19-10-55-30-644.png!)
> Job failed when enable pipelined-shuffle.compression and numberOfTaskSlots > 1
> --
>
> Key: FLINK-15308
> URL: https://issues.apache.org/jira/browse/FLINK-15308
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.10.0
> Environment: $ git log
> commit 4b54da2c67692b1c9d43e1184c00899b0151b3ae
> Author: bowen.li
> Date: Tue Dec 17 17:37:03 2019 -0800
>Reporter: Feng Jiajie
>Assignee: Yingjie Cao
>Priority: Blocker
> Attachments: image-2019-12-19-10-55-30-644.png
>
>
> Job worked well with default flink-conf.yaml with
> pipelined-shuffle.compression:
> {code:java}
> taskmanager.numberOfTaskSlots: 1
> taskmanager.network.pipelined-shuffle.compression.enabled: true
> {code}
> But when I set taskmanager.numberOfTaskSlots to 4 or 6:
> {code:java}
> taskmanager.numberOfTaskSlots: 6
> taskmanager.network.pipelined-shuffle.compression.enabled: true
> {code}
> job failed:
> {code:java}
> $ bin/flink run -m yarn-cluster -p 16 -yjm 1024m -ytm 12288m
> ~/flink-example-1.0-SNAPSHOT.jar
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/data/build/flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/lib/slf4j-log4j12-1.7.15.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/data/sa_cluster/cloudera/parcels/CDH-5.14.4-1.cdh5.14.4.p0.3/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 2019-12-18 15:04:40,514 WARN org.apache.flink.yarn.cli.FlinkYarnSessionCli
> - The configuration directory
> ('/data/build/flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/conf')
> already contains a LOG4J config file.If you want to use logback, then please
> delete or rename the log configuration file.
> 2019-12-18 15:04:40,514 WARN org.apache.flink.yarn.cli.FlinkYarnSessionCli
> - The configuration directory
> ('/data/build/flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/conf')
> already contains a LOG4J config file.If you want to use logback, then please
> delete or rename the log configuration file.
> 2019-12-18 15:04:40,907 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - No path for the flink jar passed. Using the location of class
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
> 2019-12-18 15:04:41,084 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Cluster specification:
> ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=12288,
> numberTaskManagers=1, slotsPerTaskManager=6}
> 2019-12-18 15:04:42,344 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Submitting application master application_1576573857638_0026
> 2019-12-18 15:04:42,370 INFO
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted
> application application_1576573857638_0026
> 2019-12-18 15:04:42,371 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Waiting for the cluster to be allocated
> 2019-12-18 15:04:42,372 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Deploying cluster, current state ACCEPTED
> 2019-12-18 15:04:45,388 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - YARN application has been deployed successfully.
> 2019-12-18 15:04:45,390 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Found Web Interface debugboxcreate431x3.sa:36162 of
> application 'application_1576573857638_0026'.
> Job has been submitted with JobID 9140c70769f4271cc22ea8becaa26272
>
> The program finished with the following exception:
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: org.apache.flink.client.program.ProgramInvocationException:
> Job failed (JobID: 9140c70769f4271cc22ea8becaa26272)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:335)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:205)
> at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:138)
> at
>
[jira] [Updated] (FLINK-15308) Job failed when enable pipelined-shuffle.compression and numberOfTaskSlots > 1
[
https://issues.apache.org/jira/browse/FLINK-15308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yingjie Cao updated FLINK-15308:
Attachment: image-2019-12-19-10-55-30-644.png
> Job failed when enable pipelined-shuffle.compression and numberOfTaskSlots > 1
> --
>
> Key: FLINK-15308
> URL: https://issues.apache.org/jira/browse/FLINK-15308
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.10.0
> Environment: $ git log
> commit 4b54da2c67692b1c9d43e1184c00899b0151b3ae
> Author: bowen.li
> Date: Tue Dec 17 17:37:03 2019 -0800
>Reporter: Feng Jiajie
>Assignee: Yingjie Cao
>Priority: Blocker
> Attachments: image-2019-12-19-10-55-30-644.png
>
>
> Job worked well with default flink-conf.yaml with
> pipelined-shuffle.compression:
> {code:java}
> taskmanager.numberOfTaskSlots: 1
> taskmanager.network.pipelined-shuffle.compression.enabled: true
> {code}
> But when I set taskmanager.numberOfTaskSlots to 4 or 6:
> {code:java}
> taskmanager.numberOfTaskSlots: 6
> taskmanager.network.pipelined-shuffle.compression.enabled: true
> {code}
> job failed:
> {code:java}
> $ bin/flink run -m yarn-cluster -p 16 -yjm 1024m -ytm 12288m
> ~/flink-example-1.0-SNAPSHOT.jar
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/data/build/flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/lib/slf4j-log4j12-1.7.15.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/data/sa_cluster/cloudera/parcels/CDH-5.14.4-1.cdh5.14.4.p0.3/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 2019-12-18 15:04:40,514 WARN org.apache.flink.yarn.cli.FlinkYarnSessionCli
> - The configuration directory
> ('/data/build/flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/conf')
> already contains a LOG4J config file.If you want to use logback, then please
> delete or rename the log configuration file.
> 2019-12-18 15:04:40,514 WARN org.apache.flink.yarn.cli.FlinkYarnSessionCli
> - The configuration directory
> ('/data/build/flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/conf')
> already contains a LOG4J config file.If you want to use logback, then please
> delete or rename the log configuration file.
> 2019-12-18 15:04:40,907 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - No path for the flink jar passed. Using the location of class
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
> 2019-12-18 15:04:41,084 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Cluster specification:
> ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=12288,
> numberTaskManagers=1, slotsPerTaskManager=6}
> 2019-12-18 15:04:42,344 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Submitting application master application_1576573857638_0026
> 2019-12-18 15:04:42,370 INFO
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted
> application application_1576573857638_0026
> 2019-12-18 15:04:42,371 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Waiting for the cluster to be allocated
> 2019-12-18 15:04:42,372 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Deploying cluster, current state ACCEPTED
> 2019-12-18 15:04:45,388 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - YARN application has been deployed successfully.
> 2019-12-18 15:04:45,390 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Found Web Interface debugboxcreate431x3.sa:36162 of
> application 'application_1576573857638_0026'.
> Job has been submitted with JobID 9140c70769f4271cc22ea8becaa26272
>
> The program finished with the following exception:
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: org.apache.flink.client.program.ProgramInvocationException:
> Job failed (JobID: 9140c70769f4271cc22ea8becaa26272)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:335)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:205)
> at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:138)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:664)
> at
[jira] [Commented] (FLINK-15308) Job failed when enable pipelined-shuffle.compression and numberOfTaskSlots > 1
[
https://issues.apache.org/jira/browse/FLINK-15308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999694#comment-16999694
]
Yingjie Cao commented on FLINK-15308:
-
[~fengjiajie] I can not reproduce the problem in my test environment. Is there
any other settings?
!image-2019-12-19-10-55-30-644.png!
> Job failed when enable pipelined-shuffle.compression and numberOfTaskSlots > 1
> --
>
> Key: FLINK-15308
> URL: https://issues.apache.org/jira/browse/FLINK-15308
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.10.0
> Environment: $ git log
> commit 4b54da2c67692b1c9d43e1184c00899b0151b3ae
> Author: bowen.li
> Date: Tue Dec 17 17:37:03 2019 -0800
>Reporter: Feng Jiajie
>Assignee: Yingjie Cao
>Priority: Blocker
> Attachments: image-2019-12-19-10-55-30-644.png
>
>
> Job worked well with default flink-conf.yaml with
> pipelined-shuffle.compression:
> {code:java}
> taskmanager.numberOfTaskSlots: 1
> taskmanager.network.pipelined-shuffle.compression.enabled: true
> {code}
> But when I set taskmanager.numberOfTaskSlots to 4 or 6:
> {code:java}
> taskmanager.numberOfTaskSlots: 6
> taskmanager.network.pipelined-shuffle.compression.enabled: true
> {code}
> job failed:
> {code:java}
> $ bin/flink run -m yarn-cluster -p 16 -yjm 1024m -ytm 12288m
> ~/flink-example-1.0-SNAPSHOT.jar
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/data/build/flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/lib/slf4j-log4j12-1.7.15.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/data/sa_cluster/cloudera/parcels/CDH-5.14.4-1.cdh5.14.4.p0.3/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 2019-12-18 15:04:40,514 WARN org.apache.flink.yarn.cli.FlinkYarnSessionCli
> - The configuration directory
> ('/data/build/flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/conf')
> already contains a LOG4J config file.If you want to use logback, then please
> delete or rename the log configuration file.
> 2019-12-18 15:04:40,514 WARN org.apache.flink.yarn.cli.FlinkYarnSessionCli
> - The configuration directory
> ('/data/build/flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/conf')
> already contains a LOG4J config file.If you want to use logback, then please
> delete or rename the log configuration file.
> 2019-12-18 15:04:40,907 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - No path for the flink jar passed. Using the location of class
> org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
> 2019-12-18 15:04:41,084 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Cluster specification:
> ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=12288,
> numberTaskManagers=1, slotsPerTaskManager=6}
> 2019-12-18 15:04:42,344 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Submitting application master application_1576573857638_0026
> 2019-12-18 15:04:42,370 INFO
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted
> application application_1576573857638_0026
> 2019-12-18 15:04:42,371 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Waiting for the cluster to be allocated
> 2019-12-18 15:04:42,372 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Deploying cluster, current state ACCEPTED
> 2019-12-18 15:04:45,388 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - YARN application has been deployed successfully.
> 2019-12-18 15:04:45,390 INFO org.apache.flink.yarn.YarnClusterDescriptor
> - Found Web Interface debugboxcreate431x3.sa:36162 of
> application 'application_1576573857638_0026'.
> Job has been submitted with JobID 9140c70769f4271cc22ea8becaa26272
>
> The program finished with the following exception:
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: org.apache.flink.client.program.ProgramInvocationException:
> Job failed (JobID: 9140c70769f4271cc22ea8becaa26272)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:335)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:205)
> at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:138)
> at
[jira] [Updated] (FLINK-15320) JobManager crashes in the standalone model when cancelling job which subtask' status is scheduled
[ https://issues.apache.org/jira/browse/FLINK-15320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-15320: --- Summary: JobManager crashes in the standalone model when cancelling job which subtask' status is scheduled (was: JobManager crash in the model of standalone) > JobManager crashes in the standalone model when cancelling job which subtask' > status is scheduled > - > > Key: FLINK-15320 > URL: https://issues.apache.org/jira/browse/FLINK-15320 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.10.0 >Reporter: lining >Priority: Blocker > > Use start-cluster.sh to start a standalone cluster, and then submit a job > from the streaming's example which name is TopSpeedWindowing, parallelism is > 20. Wait for one minute, cancel the job, jobmanager will crash. The exception > stack is: > 2019-12-19 10:12:11,060 ERROR > org.apache.flink.runtime.util.FatalExitExceptionHandler - FATAL: Thread > 'flink-akka.actor.default-dispatcher-2' produced an uncaught exception. > Stopping the process...2019-12-19 10:12:11,060 ERROR > org.apache.flink.runtime.util.FatalExitExceptionHandler - FATAL: Thread > 'flink-akka.actor.default-dispatcher-2' produced an uncaught exception. > Stopping the process...java.util.concurrent.CompletionException: > java.util.concurrent.CompletionException: java.lang.IllegalStateException: > Could not assign resource > org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@583585c6 to > current execution Attempt #0 (Window(GlobalWindows(), DeltaTrigger, > TimeEvictor, ComparableAggregator, PassThroughWindowFunction) -> Sink: Print > to Std. Out (19/20)) @ (unassigned) - [CANCELED]. at > org.apache.flink.runtime.scheduler.DefaultScheduler.propagateIfNonNull(DefaultScheduler.java:387) > at > org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$deployAll$4(DefaultScheduler.java:372) > at > java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) > at > java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797) > at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > at > java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) > at > org.apache.flink.runtime.concurrent.FutureUtils$WaitingConjunctFuture.handleCompletedFuture(FutureUtils.java:705) > at > java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) > at > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) > at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > at > java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) > at > org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:170) > at > java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) > at > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) > at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > at > java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.tryFulfillSlotRequestOrMakeAvailable(SlotPoolImpl.java:534) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.releaseSingleSlot(SlotPoolImpl.java:479) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.releaseSlot(SlotPoolImpl.java:390) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:557) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.releaseChild(SlotSharingManager.java:607) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.access$700(SlotSharingManager.java:352) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:716) > at > org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.releaseSharedSlot(SchedulerImpl.java:552) > at > org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.cancelSlotRequest(SchedulerImpl.java:184) > at > org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.returnLogicalSlot(SchedulerImpl.java:195) > at > org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot.lambda$returnSlotToOwner$0(SingleLogicalSlot.java:181) > at > java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) > at >
[jira] [Updated] (FLINK-15312) Remove PlanExposingEnvironment
[
https://issues.apache.org/jira/browse/FLINK-15312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-15312:
---
Labels: pull-request-available (was: )
> Remove PlanExposingEnvironment
> --
>
> Key: FLINK-15312
> URL: https://issues.apache.org/jira/browse/FLINK-15312
> Project: Flink
> Issue Type: Sub-task
> Components: API / DataSet, Tests
>Reporter: Zili Chen
>Assignee: Zili Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> We don't need it. {{env.createProgramPlan}} is sufficient, which is internal
> and can be used in testing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] TisonKun opened a new pull request #10624: [FLINK-15312][tests] Remove PlanExposingEnvironment
TisonKun opened a new pull request #10624: [FLINK-15312][tests] Remove PlanExposingEnvironment URL: https://github.com/apache/flink/pull/10624 ## What is the purpose of the change PlanExposingEnvironment is for testing only and we can replace it with `env.createProgramPlan` or an encapsulation of `PackagedProgramUtils#createPipeline`. We are towards a unique env entrypoint, I think. ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-15320) JobManager crash in the model of standalone
[ https://issues.apache.org/jira/browse/FLINK-15320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-15320: --- Priority: Blocker (was: Critical) > JobManager crash in the model of standalone > --- > > Key: FLINK-15320 > URL: https://issues.apache.org/jira/browse/FLINK-15320 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.10.0 >Reporter: lining >Priority: Blocker > > Use start-cluster.sh to start a standalone cluster, and then submit a job > from the streaming's example which name is TopSpeedWindowing, parallelism is > 20. Wait for one minute, cancel the job, jobmanager will crash. The exception > stack is: > 2019-12-19 10:12:11,060 ERROR > org.apache.flink.runtime.util.FatalExitExceptionHandler - FATAL: Thread > 'flink-akka.actor.default-dispatcher-2' produced an uncaught exception. > Stopping the process...2019-12-19 10:12:11,060 ERROR > org.apache.flink.runtime.util.FatalExitExceptionHandler - FATAL: Thread > 'flink-akka.actor.default-dispatcher-2' produced an uncaught exception. > Stopping the process...java.util.concurrent.CompletionException: > java.util.concurrent.CompletionException: java.lang.IllegalStateException: > Could not assign resource > org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@583585c6 to > current execution Attempt #0 (Window(GlobalWindows(), DeltaTrigger, > TimeEvictor, ComparableAggregator, PassThroughWindowFunction) -> Sink: Print > to Std. Out (19/20)) @ (unassigned) - [CANCELED]. at > org.apache.flink.runtime.scheduler.DefaultScheduler.propagateIfNonNull(DefaultScheduler.java:387) > at > org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$deployAll$4(DefaultScheduler.java:372) > at > java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) > at > java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797) > at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > at > java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) > at > org.apache.flink.runtime.concurrent.FutureUtils$WaitingConjunctFuture.handleCompletedFuture(FutureUtils.java:705) > at > java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) > at > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) > at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > at > java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) > at > org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:170) > at > java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) > at > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) > at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > at > java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.tryFulfillSlotRequestOrMakeAvailable(SlotPoolImpl.java:534) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.releaseSingleSlot(SlotPoolImpl.java:479) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.releaseSlot(SlotPoolImpl.java:390) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:557) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.releaseChild(SlotSharingManager.java:607) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.access$700(SlotSharingManager.java:352) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:716) > at > org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.releaseSharedSlot(SchedulerImpl.java:552) > at > org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.cancelSlotRequest(SchedulerImpl.java:184) > at > org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.returnLogicalSlot(SchedulerImpl.java:195) > at > org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot.lambda$returnSlotToOwner$0(SingleLogicalSlot.java:181) > at > java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) > at > java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778) > at > java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140) > at > org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot.returnSlotToOwner(SingleLogicalSlot.java:178) > at >
[jira] [Updated] (FLINK-15320) JobManager crash in the model of standalone
[ https://issues.apache.org/jira/browse/FLINK-15320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-15320: --- Description: Use start-cluster.sh to start a standalone cluster, and then submit a job from the streaming's example which name is TopSpeedWindowing, parallelism is 20. Wait for one minute, cancel the job, jobmanager will crash. The exception stack is: 2019-12-19 10:12:11,060 ERROR org.apache.flink.runtime.util.FatalExitExceptionHandler - FATAL: Thread 'flink-akka.actor.default-dispatcher-2' produced an uncaught exception. Stopping the process...2019-12-19 10:12:11,060 ERROR org.apache.flink.runtime.util.FatalExitExceptionHandler - FATAL: Thread 'flink-akka.actor.default-dispatcher-2' produced an uncaught exception. Stopping the process...java.util.concurrent.CompletionException: java.util.concurrent.CompletionException: java.lang.IllegalStateException: Could not assign resource org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@583585c6 to current execution Attempt #0 (Window(GlobalWindows(), DeltaTrigger, TimeEvictor, ComparableAggregator, PassThroughWindowFunction) -> Sink: Print to Std. Out (19/20)) @ (unassigned) - [CANCELED]. at org.apache.flink.runtime.scheduler.DefaultScheduler.propagateIfNonNull(DefaultScheduler.java:387) at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$deployAll$4(DefaultScheduler.java:372) at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) at org.apache.flink.runtime.concurrent.FutureUtils$WaitingConjunctFuture.handleCompletedFuture(FutureUtils.java:705) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:170) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.tryFulfillSlotRequestOrMakeAvailable(SlotPoolImpl.java:534) at org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.releaseSingleSlot(SlotPoolImpl.java:479) at org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.releaseSlot(SlotPoolImpl.java:390) at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:557) at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.releaseChild(SlotSharingManager.java:607) at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.access$700(SlotSharingManager.java:352) at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:716) at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.releaseSharedSlot(SchedulerImpl.java:552) at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.cancelSlotRequest(SchedulerImpl.java:184) at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.returnLogicalSlot(SchedulerImpl.java:195) at org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot.lambda$returnSlotToOwner$0(SingleLogicalSlot.java:181) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778) at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140) at org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot.returnSlotToOwner(SingleLogicalSlot.java:178) at org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot.releaseSlot(SingleLogicalSlot.java:125) at org.apache.flink.runtime.executiongraph.Execution.releaseAssignedResource(Execution.java:1451) at org.apache.flink.runtime.executiongraph.Execution.finishCancellation(Execution.java:1170) at org.apache.flink.runtime.executiongraph.Execution.completeCancelling(Execution.java:1150) at org.apache.flink.runtime.executiongraph.Execution.completeCancelling(Execution.java:1129) at org.apache.flink.runtime.executiongraph.Execution.cancelAtomically(Execution.java:) at
[jira] [Created] (FLINK-15320) JobManager crash in the model of standalone
lining created FLINK-15320: -- Summary: JobManager crash in the model of standalone Key: FLINK-15320 URL: https://issues.apache.org/jira/browse/FLINK-15320 Project: Flink Issue Type: Bug Components: Runtime / Coordination Affects Versions: 1.10.0 Reporter: lining Use start-cluster.sh to start an independent cluster, and then submit a job from the stream example. The job name is TopSpeedWindowing, set concurrent. 20. Wait for a while, cancel the job, jobmanager will crash. The exception stack is: 2019-12-19 10:12:11,060 ERROR org.apache.flink.runtime.util.FatalExitExceptionHandler - FATAL: Thread 'flink-akka.actor.default-dispatcher-2' produced an uncaught exception. Stopping the process...2019-12-19 10:12:11,060 ERROR org.apache.flink.runtime.util.FatalExitExceptionHandler - FATAL: Thread 'flink-akka.actor.default-dispatcher-2' produced an uncaught exception. Stopping the process...java.util.concurrent.CompletionException: java.util.concurrent.CompletionException: java.lang.IllegalStateException: Could not assign resource org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@583585c6 to current execution Attempt #0 (Window(GlobalWindows(), DeltaTrigger, TimeEvictor, ComparableAggregator, PassThroughWindowFunction) -> Sink: Print to Std. Out (19/20)) @ (unassigned) - [CANCELED]. at org.apache.flink.runtime.scheduler.DefaultScheduler.propagateIfNonNull(DefaultScheduler.java:387) at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$deployAll$4(DefaultScheduler.java:372) at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) at org.apache.flink.runtime.concurrent.FutureUtils$WaitingConjunctFuture.handleCompletedFuture(FutureUtils.java:705) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:170) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.tryFulfillSlotRequestOrMakeAvailable(SlotPoolImpl.java:534) at org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.releaseSingleSlot(SlotPoolImpl.java:479) at org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.releaseSlot(SlotPoolImpl.java:390) at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:557) at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.releaseChild(SlotSharingManager.java:607) at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.access$700(SlotSharingManager.java:352) at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:716) at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.releaseSharedSlot(SchedulerImpl.java:552) at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.cancelSlotRequest(SchedulerImpl.java:184) at org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.returnLogicalSlot(SchedulerImpl.java:195) at org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot.lambda$returnSlotToOwner$0(SingleLogicalSlot.java:181) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778) at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140) at org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot.returnSlotToOwner(SingleLogicalSlot.java:178) at org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot.releaseSlot(SingleLogicalSlot.java:125) at org.apache.flink.runtime.executiongraph.Execution.releaseAssignedResource(Execution.java:1451) at org.apache.flink.runtime.executiongraph.Execution.finishCancellation(Execution.java:1170) at org.apache.flink.runtime.executiongraph.Execution.completeCancelling(Execution.java:1150) at
[jira] [Commented] (FLINK-10114) Support Orc for StreamingFileSink
[ https://issues.apache.org/jira/browse/FLINK-10114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999669#comment-16999669 ] vinoyang commented on FLINK-10114: -- Hi [~mbellani] thanks for paying attention to this feature and sharing your thought and design. I am sorry I have no time to join you and learn the context. And I am sorry I can not reassign this ticket. Based on the new contribution constitution, before implementing, it would be better to agree on the design with the community. In addition, it would be better to find a committer as your mentor for the feature. Ping [~aljoscha] and [~trohrm...@apache.org] to give more professional options. > Support Orc for StreamingFileSink > - > > Key: FLINK-10114 > URL: https://issues.apache.org/jira/browse/FLINK-10114 > Project: Flink > Issue Type: Sub-task > Components: Connectors / FileSystem >Reporter: zhangminglei >Assignee: vinoyang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14172) Implement KubeClient with official Java client library for kubernetes
[ https://issues.apache.org/jira/browse/FLINK-14172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999625#comment-16999625 ] Jian He commented on FLINK-14172: - I also see fabric8 client is used more widely than the official client > Implement KubeClient with official Java client library for kubernetes > - > > Key: FLINK-14172 > URL: https://issues.apache.org/jira/browse/FLINK-14172 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes >Reporter: Yang Wang >Priority: Major > > Official Java client library for kubernetes is become more and more active. > The new features(such as leader election) and some client > implementations(informer, lister, cache) are better. So we should use the > official java client for kubernetes in flink. > https://github.com/kubernetes-client/java -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-12160) Support to ship user config files on kubernetes
[
https://issues.apache.org/jira/browse/FLINK-12160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999624#comment-16999624
]
Jian He commented on FLINK-12160:
-
As the link noted, the configMap is not suitable for large files like jars,
reusing hdfs might be a better idea.
> Support to ship user config files on kubernetes
> ---
>
> Key: FLINK-12160
> URL: https://issues.apache.org/jira/browse/FLINK-12160
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Reporter: Yang Wang
>Assignee: Yang Wang
>Priority: Major
>
> When deploying flink cluster on Yarn, `yarnShip` cli option could be used to
> ship user config files and jars. The files are registered as Yarn local
> resource which is saved on hdfs.
>
> However, we do not have a default existing dfs. The `ConfigMap` could be used
> to ship small files. The files will be stored in etcd and then mounted to
> jobmanager and taskmanager pod so that they could use it as locally.
> {code:java}
> -kt,--kubernetesship Ship files in the specified
> directory
> (t for transfer). Only small
> files(<1MB) could be supported.
> {code}
> [https://stackoverflow.com/questions/53012798/kubernetes-configmap-size-limitation]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-15256) unable to drop table in HiveCatalogITCase
[
https://issues.apache.org/jira/browse/FLINK-15256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bowen Li closed FLINK-15256.
Resolution: Invalid
> unable to drop table in HiveCatalogITCase
> -
>
> Key: FLINK-15256
> URL: https://issues.apache.org/jira/browse/FLINK-15256
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Reporter: Bowen Li
>Assignee: Terry Wang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {code:java}
> @Test
> public void testCsvTableViaSQL() throws Exception {
> EnvironmentSettings settings =
> EnvironmentSettings.newInstance().useBlinkPlanner().inBatchMode().build();
> TableEnvironment tableEnv = TableEnvironment.create(settings);
> tableEnv.registerCatalog("myhive", hiveCatalog);
> tableEnv.useCatalog("myhive");
> String path =
> this.getClass().getResource("/csv/test.csv").getPath();
> tableEnv.sqlUpdate("create table test2 (name String, age Int)
> with (\n" +
> " 'connector.type' = 'filesystem',\n" +
> " 'connector.path' = 'file://" + path + "',\n" +
> " 'format.type' = 'csv'\n" +
> ")");
> Table t = tableEnv.sqlQuery("SELECT * FROM
> myhive.`default`.test2");
> List result = TableUtils.collectToList(t);
> // assert query result
> assertEquals(
> new HashSet<>(Arrays.asList(
> Row.of("1", 1),
> Row.of("2", 2),
> Row.of("3", 3))),
> new HashSet<>(result)
> );
> tableEnv.sqlUpdate("drop table myhive.`default`.tests2");
> }
> {code}
> The last drop table statement reports error as:
> {code:java}
> org.apache.flink.table.api.ValidationException: Could not execute DropTable
> in path `myhive`.`default`.`tests2`
> at
> org.apache.flink.table.catalog.CatalogManager.execute(CatalogManager.java:568)
> at
> org.apache.flink.table.catalog.CatalogManager.dropTable(CatalogManager.java:543)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlUpdate(TableEnvironmentImpl.java:519)
> at
> org.apache.flink.table.catalog.hive.HiveCatalogITCase.testCsvTableViaSQL(HiveCatalogITCase.java:123)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at
> org.apache.flink.connectors.hive.FlinkStandaloneHiveRunner.runTestMethod(FlinkStandaloneHiveRunner.java:169)
> at
> org.apache.flink.connectors.hive.FlinkStandaloneHiveRunner.runChild(FlinkStandaloneHiveRunner.java:154)
> at
> org.apache.flink.connectors.hive.FlinkStandaloneHiveRunner.runChild(FlinkStandaloneHiveRunner.java:92)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
>
[jira] [Commented] (FLINK-15256) unable to drop table in HiveCatalogITCase
[
https://issues.apache.org/jira/browse/FLINK-15256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999470#comment-16999470
]
Bowen Li commented on FLINK-15256:
--
[~Terry1897] I ran it today and it succeeded, maybe my local testing env having
issues. Thanks for looking into it.
[~gjy] the merged PR has a wrong jira id.
Closing this as invalid
> unable to drop table in HiveCatalogITCase
> -
>
> Key: FLINK-15256
> URL: https://issues.apache.org/jira/browse/FLINK-15256
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Reporter: Bowen Li
>Assignee: Terry Wang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {code:java}
> @Test
> public void testCsvTableViaSQL() throws Exception {
> EnvironmentSettings settings =
> EnvironmentSettings.newInstance().useBlinkPlanner().inBatchMode().build();
> TableEnvironment tableEnv = TableEnvironment.create(settings);
> tableEnv.registerCatalog("myhive", hiveCatalog);
> tableEnv.useCatalog("myhive");
> String path =
> this.getClass().getResource("/csv/test.csv").getPath();
> tableEnv.sqlUpdate("create table test2 (name String, age Int)
> with (\n" +
> " 'connector.type' = 'filesystem',\n" +
> " 'connector.path' = 'file://" + path + "',\n" +
> " 'format.type' = 'csv'\n" +
> ")");
> Table t = tableEnv.sqlQuery("SELECT * FROM
> myhive.`default`.test2");
> List result = TableUtils.collectToList(t);
> // assert query result
> assertEquals(
> new HashSet<>(Arrays.asList(
> Row.of("1", 1),
> Row.of("2", 2),
> Row.of("3", 3))),
> new HashSet<>(result)
> );
> tableEnv.sqlUpdate("drop table myhive.`default`.tests2");
> }
> {code}
> The last drop table statement reports error as:
> {code:java}
> org.apache.flink.table.api.ValidationException: Could not execute DropTable
> in path `myhive`.`default`.`tests2`
> at
> org.apache.flink.table.catalog.CatalogManager.execute(CatalogManager.java:568)
> at
> org.apache.flink.table.catalog.CatalogManager.dropTable(CatalogManager.java:543)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlUpdate(TableEnvironmentImpl.java:519)
> at
> org.apache.flink.table.catalog.hive.HiveCatalogITCase.testCsvTableViaSQL(HiveCatalogITCase.java:123)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at
> org.apache.flink.connectors.hive.FlinkStandaloneHiveRunner.runTestMethod(FlinkStandaloneHiveRunner.java:169)
> at
> org.apache.flink.connectors.hive.FlinkStandaloneHiveRunner.runChild(FlinkStandaloneHiveRunner.java:154)
> at
> org.apache.flink.connectors.hive.FlinkStandaloneHiveRunner.runChild(FlinkStandaloneHiveRunner.java:92)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>
[jira] [Assigned] (FLINK-15247) Closing (Testing)MiniCluster may cause ConcurrentModificationException
[
https://issues.apache.org/jira/browse/FLINK-15247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Zagrebin reassigned FLINK-15247:
---
Assignee: Andrey Zagrebin
> Closing (Testing)MiniCluster may cause ConcurrentModificationException
> --
>
> Key: FLINK-15247
> URL: https://issues.apache.org/jira/browse/FLINK-15247
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.10.0
>Reporter: Gary Yao
>Assignee: Andrey Zagrebin
>Priority: Blocker
> Labels: test-stability
> Fix For: 1.10.0
>
>
> {noformat}
> Test
> operatorsBecomeBackPressured(org.apache.flink.test.streaming.runtime.BackPressureITCase)
> failed with:
> org.apache.flink.util.FlinkException: Could not close resource.
> at
> org.apache.flink.util.AutoCloseableAsync.close(AutoCloseableAsync.java:42)org.apache.flink.test.streaming.runtime.BackPressureITCase.tearDown(BackPressureITCase.java:165)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> Caused by: org.apache.flink.util.FlinkException: Error while shutting the
> TaskExecutor down.
> at
> org.apache.flink.runtime.taskexecutor.TaskExecutor.handleOnStopException(TaskExecutor.java:397)
> at
> org.apache.flink.runtime.taskexecutor.TaskExecutor.lambda$onStop$0(TaskExecutor.java:382)
> at
> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
> at
> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
>
[jira] [Commented] (FLINK-15199) Benchmarks are not compiling
[
https://issues.apache.org/jira/browse/FLINK-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999369#comment-16999369
]
Piotr Nowojski commented on FLINK-15199:
It's already merged to release-1.10 as 535845153ba45dc8e564f235e353e86aa1429fdd.
I probably forgot to specify it in my previous comment.
> Benchmarks are not compiling
>
>
> Key: FLINK-15199
> URL: https://issues.apache.org/jira/browse/FLINK-15199
> Project: Flink
> Issue Type: Bug
> Components: Benchmarks
>Affects Versions: 1.10.0
>Reporter: Piotr Nowojski
>Assignee: Piotr Nowojski
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Recent changes in FLINK-14926 caused:
> {noformat}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/jenkins/workspace/flink-master-benchmarks/flink-benchmarks/src/main/java/org/apache/flink/state/benchmark/BackendUtils.java:[61,57]
> cannot infer type arguments for
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder<>
> [INFO] 1 error
> [INFO] -
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 2.635 s
> [INFO] Finished at: 2019-12-11T14:58:37+01:00
> [INFO] Final Memory: 31M/751M
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile)
> on project flink-hackathon-benchmarks: Compilation failure
> [ERROR]
> /home/jenkins/workspace/flink-master-benchmarks/flink-benchmarks/src/main/java/org/apache/flink/state/benchmark/BackendUtils.java:[61,57]
> cannot infer type arguments for
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder<>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-10114) Support Orc for StreamingFileSink
[ https://issues.apache.org/jira/browse/FLINK-10114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999359#comment-16999359 ] Manish Bellani commented on FLINK-10114: Hi, Is there still interest in this work? I have an implementation of an S3/Orc sink that handles these requirements. We're currently using this at GitHub internally to write several billions of events/terabytes of data per day to S3(exactly once). If there's interest I can kick off a conversation at GitHub, in the meantime do you mind sharing how can I contribute to flink? I found this: [https://flink.apache.org/contributing/contribute-code.html] but is there anything else that I need to do? Thanks Manish > Support Orc for StreamingFileSink > - > > Key: FLINK-10114 > URL: https://issues.apache.org/jira/browse/FLINK-10114 > Project: Flink > Issue Type: Sub-task > Components: Connectors / FileSystem >Reporter: zhangminglei >Assignee: vinoyang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (FLINK-9749) Rework Bucketing Sink
[
https://issues.apache.org/jira/browse/FLINK-9749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manish Bellani updated FLINK-9749:
--
Comment: was deleted
(was: Hi,
Is there still interest in this work? I have an implementation of an S3/Orc
sink that handles these requirements. We're currently using this at GitHub
internally to write several billions of events/terabytes of data per day to
S3(exactly once). If there's interest I can kick off a conversation at GitHub,
in the meantime do you mind sharing how can I contribute to flink? I found
this: [https://flink.apache.org/contributing/contribute-code.html] but is there
anything else that I need to do?
Thanks
Manish)
> Rework Bucketing Sink
> -
>
> Key: FLINK-9749
> URL: https://issues.apache.org/jira/browse/FLINK-9749
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / FileSystem
>Reporter: Stephan Ewen
>Assignee: Kostas Kloudas
>Priority: Major
>
> The BucketingSink has a series of deficits at the moment.
> Due to the long list of issues, I would suggest to add a new
> StreamingFileSink with a new and cleaner design
> h3. Encoders, Parquet, ORC
> - It only efficiently supports row-wise data formats (avro, jso, sequence
> files.
> - Efforts to add (columnar) compression for blocks of data is inefficient,
> because blocks cannot span checkpoints due to persistence-on-checkpoint.
> - The encoders are part of the \{{flink-connector-filesystem project}},
> rather than in orthogonal formats projects. This blows up the dependencies of
> the \{{flink-connector-filesystem project}} project. As an example, the
> rolling file sink has dependencies on Hadoop and Avro, which messes up
> dependency management.
> h3. Use of FileSystems
> - The BucketingSink works only on Hadoop's FileSystem abstraction not
> support Flink's own FileSystem abstraction and cannot work with the packaged
> S3, maprfs, and swift file systems
> - The sink hence needs Hadoop as a dependency
> - The sink relies on "trying out" whether truncation works, which requires
> write access to the users working directory
> - The sink relies on enumerating and counting files, rather than maintaining
> its own state, making less efficient
> h3. Correctness and Efficiency on S3
> - The BucketingSink relies on strong consistency in the file enumeration,
> hence may work incorrectly on S3.
> - The BucketingSink relies on persisting streams at intermediate points.
> This is not working properly on S3, hence there may be data loss on S3.
> h3. .valid-length companion file
> - The valid length file makes it hard for consumers of the data and should
> be dropped
> We track this design in a series of sub issues.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-11956) Remove shading from S3 filesystems build
[ https://issues.apache.org/jira/browse/FLINK-11956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski closed FLINK-11956. -- Fix Version/s: 1.10.0 Resolution: Fixed merged to master as 9fdedf552822d27efbd36f87d21b9b0f2637b757 merged to release-1.10 as 8ec545d56f007645ca8f2a2374386882132ffc7a > Remove shading from S3 filesystems build > > > Key: FLINK-11956 > URL: https://issues.apache.org/jira/browse/FLINK-11956 > Project: Flink > Issue Type: Sub-task > Components: Connectors / FileSystem >Affects Versions: 1.9.0 >Reporter: Stefan Richter >Assignee: Arvid Heise >Priority: Critical > Fix For: 1.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-11956) Remove shading from S3 filesystems build
[ https://issues.apache.org/jira/browse/FLINK-11956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski updated FLINK-11956: --- Summary: Remove shading from S3 filesystems build (was: Remove shading from filesystems build) > Remove shading from S3 filesystems build > > > Key: FLINK-11956 > URL: https://issues.apache.org/jira/browse/FLINK-11956 > Project: Flink > Issue Type: Sub-task > Components: Connectors / FileSystem >Affects Versions: 1.9.0 >Reporter: Stefan Richter >Assignee: Arvid Heise >Priority: Critical > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (FLINK-15241) Revert the unexpected change of the configuration for Mesos CPU cores
[ https://issues.apache.org/jira/browse/FLINK-15241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann resolved FLINK-15241. --- Resolution: Fixed Fixed via master: 4fc55c747d001b75ba4652e867c98a5e2d62fd69 1.10.0: b360019433f25395a8fc151b23d94eb7bb81359d > Revert the unexpected change of the configuration for Mesos CPU cores > - > > Key: FLINK-15241 > URL: https://issues.apache.org/jira/browse/FLINK-15241 > Project: Flink > Issue Type: Bug > Components: Deployment / Mesos >Reporter: Yangze Guo >Assignee: Yangze Guo >Priority: Blocker > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Before 1.10, the default config of cpu cores for Mesos Workers is equal to > taskmanager.numberOfTaskSlots. However, it is now 0.0 by default. I feel > sorry I've not catch it in > [FLINK-15196|https://issues.apache.org/jira/browse/FLINK-15196]. I think this > change is unexpected to user thus we need to fix it. > cc [~trohrmann] Could you assigned this ticket to me? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15199) Benchmarks are not compiling
[
https://issues.apache.org/jira/browse/FLINK-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999320#comment-16999320
]
Yu Li commented on FLINK-15199:
---
Waiting for this one to be merged also into release-1.10
> Benchmarks are not compiling
>
>
> Key: FLINK-15199
> URL: https://issues.apache.org/jira/browse/FLINK-15199
> Project: Flink
> Issue Type: Bug
> Components: Benchmarks
>Affects Versions: 1.10.0
>Reporter: Piotr Nowojski
>Assignee: Piotr Nowojski
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Recent changes in FLINK-14926 caused:
> {noformat}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/jenkins/workspace/flink-master-benchmarks/flink-benchmarks/src/main/java/org/apache/flink/state/benchmark/BackendUtils.java:[61,57]
> cannot infer type arguments for
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder<>
> [INFO] 1 error
> [INFO] -
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 2.635 s
> [INFO] Finished at: 2019-12-11T14:58:37+01:00
> [INFO] Final Memory: 31M/751M
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile)
> on project flink-hackathon-benchmarks: Compilation failure
> [ERROR]
> /home/jenkins/workspace/flink-master-benchmarks/flink-benchmarks/src/main/java/org/apache/flink/state/benchmark/BackendUtils.java:[61,57]
> cannot infer type arguments for
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder<>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (FLINK-14926) Make sure no resource leak of RocksObject
[
https://issues.apache.org/jira/browse/FLINK-14926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li resolved FLINK-14926.
---
Fix Version/s: 1.10.0
Resolution: Fixed
Fixed in master via:
* e05618b427
* 0602ccf1d2
* 93deb1a709
* 31dd8c3624
* 2405111bd6
* 07f08fb19a
Fixed in 1.10.0 via:
* 13c577f6d2
* 5326730225
* 386dad8705
* f5fa21b7d0
* 6f777d5a51
* e94bf2fe70
(Thanks [~sewen] for reviewing and merging the PR)
> Make sure no resource leak of RocksObject
> -
>
> Key: FLINK-14926
> URL: https://issues.apache.org/jira/browse/FLINK-14926
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
>Affects Versions: 1.6.4, 1.7.2, 1.8.2, 1.9.1
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When investigating FLINK-14484 to allow setting {{WriteBufferManager}} via
> options, we find it necessary to supply a {{close}} method in
> {{OptionsFactory}} to make sure resources configured in options like
> {{WriteBufferManager}} could/would be released.
> What's more, we're having potential risk of resource leak in
> {{PredefinedOptions}} such as the created [BloomFilter
> instance|https://github.com/apache/flink/blob/e14bd50fed42e89674ba9d01a231d5c5e59f490c/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/PredefinedOptions.java#L155]
> is never closed (regression caused by FLINK-7220 changes), and we should
> also fix this and add a {{close}} method in {{PredefinedOptions}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14926) Make sure no resource leak of RocksObject
[
https://issues.apache.org/jira/browse/FLINK-14926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999315#comment-16999315
]
Yu Li commented on FLINK-14926:
---
We also need to merge the changes of FLINK-15199 into release-1.10 to prevent
compilation issue when running flink-benchmark against release-1.10, and I
guess the plan was to include all related commits here when closing it. But yes
I think we could also update FLINK-15199 to indicate the merging into
release-1.10.
Let me add some comments about changes for this JIRA and close it. Thanks for
the reminder [~gjy].
> Make sure no resource leak of RocksObject
> -
>
> Key: FLINK-14926
> URL: https://issues.apache.org/jira/browse/FLINK-14926
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
>Affects Versions: 1.6.4, 1.7.2, 1.8.2, 1.9.1
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When investigating FLINK-14484 to allow setting {{WriteBufferManager}} via
> options, we find it necessary to supply a {{close}} method in
> {{OptionsFactory}} to make sure resources configured in options like
> {{WriteBufferManager}} could/would be released.
> What's more, we're having potential risk of resource leak in
> {{PredefinedOptions}} such as the created [BloomFilter
> instance|https://github.com/apache/flink/blob/e14bd50fed42e89674ba9d01a231d5c5e59f490c/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/PredefinedOptions.java#L155]
> is never closed (regression caused by FLINK-7220 changes), and we should
> also fix this and add a {{close}} method in {{PredefinedOptions}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15317) State TTL Heap backend end-to-end test fails on Travis
[
https://issues.apache.org/jira/browse/FLINK-15317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999314#comment-16999314
]
Yu Li commented on FLINK-15317:
---
Thanks for logging the issue [~gjy] and the quick fix [~arvid heise].
Correct me if I'm wrong, is this a regression of FLINK-15076? If so, let's add
some link to the issue. Thanks.
> State TTL Heap backend end-to-end test fails on Travis
> --
>
> Key: FLINK-15317
> URL: https://issues.apache.org/jira/browse/FLINK-15317
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network, Tests
>Affects Versions: 1.10.0
>Reporter: Gary Yao
>Assignee: Arvid Heise
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> https://api.travis-ci.org/v3/job/626286529/log.txt
> {noformat}
> Checking for errors...
> Found error in log files:
> ...
> java.lang.IllegalStateException: Released
> at
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:483)
> at
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.pollNext(SingleInputGate.java:474)
> at
> org.apache.flink.runtime.taskmanager.InputGateWithMetrics.pollNext(InputGateWithMetrics.java:75)
> at
> org.apache.flink.streaming.runtime.io.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:125)
> at
> org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:133)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:69)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:311)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:187)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:488)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:470)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:702)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:527)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13417) Bump Zookeeper to 3.5.5
[ https://issues.apache.org/jira/browse/FLINK-13417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-13417: --- Labels: pull-request-available (was: ) > Bump Zookeeper to 3.5.5 > --- > > Key: FLINK-13417 > URL: https://issues.apache.org/jira/browse/FLINK-13417 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.9.0 >Reporter: Konstantin Knauf >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > > User might want to secure their Zookeeper connection via SSL. > This requires a Zookeeper version >= 3.5.1. We might as well try to bump it > to 3.5.5, which is the latest version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] tillrohrmann opened a new pull request #10623: [WIP][FLINK-13417] Bump ZooKeeper from 3.4.10 to 3.5.6
tillrohrmann opened a new pull request #10623: [WIP][FLINK-13417] Bump ZooKeeper from 3.4.10 to 3.5.6 URL: https://github.com/apache/flink/pull/10623 This is a WIP PR for the ZooKeeper version upgrade. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14364) Allow comments fail when not ignore parse errors in CsvRowDeserializationSchema
[
https://issues.apache.org/jira/browse/FLINK-14364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999291#comment-16999291
]
Jiayi Liao commented on FLINK-14364:
[~lzljs3620320] [~ykt836]
Sorry for the late reply, I was busy on something else recently :(.
I've added a validation in {{CsvValidator}} to improve the usability in this
[PR|https://github.com/apache/flink/pull/10622]. I'll appreciate it if you can
spare time and take a look.
> Allow comments fail when not ignore parse errors in
> CsvRowDeserializationSchema
> ---
>
> Key: FLINK-14364
> URL: https://issues.apache.org/jira/browse/FLINK-14364
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Reporter: Jingsong Lee
>Assignee: Jiayi Liao
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Use CsvRowDeserializationSchema, when setIgnoreParseErrors(false) and
> setAllowComments(true).
> If there are some comments in msg, will throw MismatchedInputException.
> If this a bug? and we should catch MismatchedInputException and return null?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-13417) Bump Zookeeper to 3.5.5
[ https://issues.apache.org/jira/browse/FLINK-13417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999287#comment-16999287 ] Till Rohrmann commented on FLINK-13417: --- For future reference: [Here|https://lists.apache.org/thread.html/be402a11bc986219eabd9dd8af507f36f49784d5400d0873e9ec0c2e%40%3Cdev.flink.apache.org%3E] is the link to the dev ML discussion about dropping vendor specific repositories. > Bump Zookeeper to 3.5.5 > --- > > Key: FLINK-13417 > URL: https://issues.apache.org/jira/browse/FLINK-13417 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.9.0 >Reporter: Konstantin Knauf >Priority: Major > Fix For: 1.11.0 > > > User might want to secure their Zookeeper connection via SSL. > This requires a Zookeeper version >= 3.5.1. We might as well try to bump it > to 3.5.5, which is the latest version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14364) Allow comments fail when not ignore parse errors in CsvRowDeserializationSchema
[ https://issues.apache.org/jira/browse/FLINK-14364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-14364: --- Labels: pull-request-available (was: ) > Allow comments fail when not ignore parse errors in > CsvRowDeserializationSchema > --- > > Key: FLINK-14364 > URL: https://issues.apache.org/jira/browse/FLINK-14364 > Project: Flink > Issue Type: Bug > Components: Table SQL / API >Reporter: Jingsong Lee >Assignee: Jiayi Liao >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > > Use CsvRowDeserializationSchema, when setIgnoreParseErrors(false) and > setAllowComments(true). > If there are some comments in msg, will throw MismatchedInputException. > If this a bug? and we should catch MismatchedInputException and return null? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] buptljy opened a new pull request #10622: [FLINK-14364] [flink-formats] Add validation when format.allow-comments is used with format.ignore-parse-errors in CsvValidator
buptljy opened a new pull request #10622: [FLINK-14364] [flink-formats] Add validation when format.allow-comments is used with format.ignore-parse-errors in CsvValidator URL: https://github.com/apache/flink/pull/10622 ## What is the purpose of the change As described in [FLINK-14364](https://issues.apache.org/jira/browse/FLINK-14364), we can only use `format.allow-comments` when `format.ignore-parse-errors` is set. So we add a validation here to improve the usability. ## Brief change log Add a new check in `CsvValidator`. ## Verifying this change Unit testing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (FLINK-13417) Bump Zookeeper to 3.5.5
[ https://issues.apache.org/jira/browse/FLINK-13417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999279#comment-16999279 ] Zili Chen edited comment on FLINK-13417 at 12/18/19 3:40 PM: - Thanks for reviving this thread [~sewen] & [~trohrmann]. One thing I'd like to emphasize again is that the bump to zk 3.5 will force user update version of their zookeeper ensemble. See also [this comment|https://issues.apache.org/jira/browse/FLINK-13417?focusedCommentId=16921386=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16921386] was (Author: tison): Thanks for reviving this thread [~sewen][~trohrmann]. One thing I'd like to emphasize again is that the bump to zk 3.5 will force user update version of their zookeeper ensemble. See also [this comment|https://issues.apache.org/jira/browse/FLINK-13417?focusedCommentId=16921386=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16921386] > Bump Zookeeper to 3.5.5 > --- > > Key: FLINK-13417 > URL: https://issues.apache.org/jira/browse/FLINK-13417 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.9.0 >Reporter: Konstantin Knauf >Priority: Major > Fix For: 1.11.0 > > > User might want to secure their Zookeeper connection via SSL. > This requires a Zookeeper version >= 3.5.1. We might as well try to bump it > to 3.5.5, which is the latest version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-13417) Bump Zookeeper to 3.5.5
[ https://issues.apache.org/jira/browse/FLINK-13417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999279#comment-16999279 ] Zili Chen commented on FLINK-13417: --- Thanks for reviving this thread [~sewen][~trohrmann]. One thing I'd like to emphasize again is that the bump to zk 3.5 will force user update version of their zookeeper ensemble. See also [this comment|https://issues.apache.org/jira/browse/FLINK-13417?focusedCommentId=16921386=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16921386] > Bump Zookeeper to 3.5.5 > --- > > Key: FLINK-13417 > URL: https://issues.apache.org/jira/browse/FLINK-13417 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.9.0 >Reporter: Konstantin Knauf >Priority: Major > Fix For: 1.11.0 > > > User might want to secure their Zookeeper connection via SSL. > This requires a Zookeeper version >= 3.5.1. We might as well try to bump it > to 3.5.5, which is the latest version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-13417) Bump Zookeeper to 3.5.5
[ https://issues.apache.org/jira/browse/FLINK-13417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999263#comment-16999263 ] Till Rohrmann commented on FLINK-13417: --- Hi [~tison], thanks a lot for driving this effort. I would like to help with finalizing it so that we can include it in the upcoming 1.10 release. As a first step I would propose to drop the vendor specific repos. I'll start a discussion on dev for this. Then the version upgrade should be a bit easier. > Bump Zookeeper to 3.5.5 > --- > > Key: FLINK-13417 > URL: https://issues.apache.org/jira/browse/FLINK-13417 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.9.0 >Reporter: Konstantin Knauf >Priority: Major > Fix For: 1.11.0 > > > User might want to secure their Zookeeper connection via SSL. > This requires a Zookeeper version >= 3.5.1. We might as well try to bump it > to 3.5.5, which is the latest version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15247) Closing (Testing)MiniCluster may cause ConcurrentModificationException
[
https://issues.apache.org/jira/browse/FLINK-15247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999249#comment-16999249
]
Andrey Zagrebin commented on FLINK-15247:
-
It seems the problem is a bit deeper.
We introduced the delayed TaskExecutor#stopTaskExecutorServices in case of
stopping TM in FLINK-11630 to gracefully cancel the tasks and let them release
the resources of services before the final shutdown of services. Previously,
slots were static, there were no resources attached to slots and TaskSlotTable
was stateless, that did not cause issues.
Now TaskSlotTable has dynamically allocated slots after
[FLINK-14189|https://jira.apache.org/jira/browse/FLINK-14189] and memory
manager is attached to the slot after
[FLINK-14400|https://jira.apache.org/jira/browse/FLINK-14400]. It means that
when we cancel now tasks it triggers stopping of the services (including
TaskSlotTable) concurrently with freeing slots and their resources. Even we put
stopping of the services into the main thread, freeing of slots can be after
stopping. It would be cleaner to firstly cancel tasks, wait for the subsequent
freeing of the slots and then shutdown services.
cc [~trohrm...@apache.org]
> Closing (Testing)MiniCluster may cause ConcurrentModificationException
> --
>
> Key: FLINK-15247
> URL: https://issues.apache.org/jira/browse/FLINK-15247
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.10.0
>Reporter: Gary Yao
>Priority: Blocker
> Labels: test-stability
> Fix For: 1.10.0
>
>
> {noformat}
> Test
> operatorsBecomeBackPressured(org.apache.flink.test.streaming.runtime.BackPressureITCase)
> failed with:
> org.apache.flink.util.FlinkException: Could not close resource.
> at
> org.apache.flink.util.AutoCloseableAsync.close(AutoCloseableAsync.java:42)org.apache.flink.test.streaming.runtime.BackPressureITCase.tearDown(BackPressureITCase.java:165)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158)
> at
>
[jira] [Updated] (FLINK-15317) State TTL Heap backend end-to-end test fails on Travis
[
https://issues.apache.org/jira/browse/FLINK-15317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-15317:
---
Labels: pull-request-available (was: )
> State TTL Heap backend end-to-end test fails on Travis
> --
>
> Key: FLINK-15317
> URL: https://issues.apache.org/jira/browse/FLINK-15317
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network, Tests
>Affects Versions: 1.10.0
>Reporter: Gary Yao
>Assignee: Arvid Heise
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.10.0
>
>
> https://api.travis-ci.org/v3/job/626286529/log.txt
> {noformat}
> Checking for errors...
> Found error in log files:
> ...
> java.lang.IllegalStateException: Released
> at
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:483)
> at
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.pollNext(SingleInputGate.java:474)
> at
> org.apache.flink.runtime.taskmanager.InputGateWithMetrics.pollNext(InputGateWithMetrics.java:75)
> at
> org.apache.flink.streaming.runtime.io.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:125)
> at
> org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:133)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:69)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:311)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:187)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:488)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:470)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:702)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:527)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] AHeise opened a new pull request #10621: [FLINK-15317][runtime] Handle closed input gates more gracefully during
AHeise opened a new pull request #10621: [FLINK-15317][runtime] Handle closed input gates more gracefully during URL: https://github.com/apache/flink/pull/10621 cancellation. ## What is the purpose of the change During task cancellation, input gates can be closed in the task cancellation thread, before the main invokable has been interrupted, which resulted in reads from a closed gate. This commit will handle this case in a more graceful manner such that the end-user is not seeing any additional exception. ## Brief change log - Use `TaskCancelException` instead of `IllegalStateException` when input gates are closed in a similar fashion as this exception has been used in `LocalInputChannel`. - Add a special handling of `TaskCancelException` to `StreamTask#runMainLoop()`, to make sure the exception is not appearing in the log. ## Verifying this change This fix is indirectly covered by many e2e tests that rely on this exception being swallowed. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-15244) FileUtils#deleteDirectoryQuietly will delete files in the symbolic link which point to a directory
[
https://issues.apache.org/jira/browse/FLINK-15244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao closed FLINK-15244.
Resolution: Fixed
1.10: 337f1cc534aac7f2806465687eeefb4d4984e3f3
master: 3c8a3ac0e7c5784a125d2bce3d2058fd5d1066a8
> FileUtils#deleteDirectoryQuietly will delete files in the symbolic link which
> point to a directory
> --
>
> Key: FLINK-15244
> URL: https://issues.apache.org/jira/browse/FLINK-15244
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
>Affects Versions: 1.10.0
>Reporter: Wei Zhong
>Assignee: Wei Zhong
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> _FileUtils.deleteDirectoryQuietly_ will delete files in symbolic link which
> point to a directory. Currently the PyFlink uses this method to delete
> temporary folders generated during the job submission and python UDF
> execution, which contains the symbolic links which may point to users'
> libraries and directories in distributed cache.
> To resolve this problem we need to check if the directory is symbolic link in
> _FileUtils.deleteDirectoryInternal:_
> {code:java}
> private static void deleteDirectoryInternal(File directory) throws
> IOException {
> // **We should check if the directory is symbolic link.**
> if (directory.isDirectory()) {
> // directory exists and is a directory
> // empty the directory first
> try {
> cleanDirectoryInternal(directory);
> }
> catch (FileNotFoundException ignored) {
> // someone concurrently deleted the directory, nothing to do for us
> return;
> }
> // delete the directory. this fails if the directory is not empty,
> meaning
> // if new files got concurrently created. we want to fail then.
> // if someone else deleted the empty directory concurrently, we don't
> mind
> // the result is the same for us, after all
> Files.deleteIfExists(directory.toPath());
>}
>else if (directory.exists()) {
> // exists but is file, not directory
> // either an error from the caller, or concurrently a file got created
> throw new IOException(directory + " is not a directory");
>}
>// else: does not exist, which is okay (as if deleted)
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15244) FileUtils#deleteDirectoryQuietly will delete files in the symbolic link which point to a directory
[
https://issues.apache.org/jira/browse/FLINK-15244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao updated FLINK-15244:
-
Fix Version/s: (was: 1.11.0)
> FileUtils#deleteDirectoryQuietly will delete files in the symbolic link which
> point to a directory
> --
>
> Key: FLINK-15244
> URL: https://issues.apache.org/jira/browse/FLINK-15244
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
>Affects Versions: 1.10.0
>Reporter: Wei Zhong
>Assignee: Wei Zhong
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> _FileUtils.deleteDirectoryQuietly_ will delete files in symbolic link which
> point to a directory. Currently the PyFlink uses this method to delete
> temporary folders generated during the job submission and python UDF
> execution, which contains the symbolic links which may point to users'
> libraries and directories in distributed cache.
> To resolve this problem we need to check if the directory is symbolic link in
> _FileUtils.deleteDirectoryInternal:_
> {code:java}
> private static void deleteDirectoryInternal(File directory) throws
> IOException {
> // **We should check if the directory is symbolic link.**
> if (directory.isDirectory()) {
> // directory exists and is a directory
> // empty the directory first
> try {
> cleanDirectoryInternal(directory);
> }
> catch (FileNotFoundException ignored) {
> // someone concurrently deleted the directory, nothing to do for us
> return;
> }
> // delete the directory. this fails if the directory is not empty,
> meaning
> // if new files got concurrently created. we want to fail then.
> // if someone else deleted the empty directory concurrently, we don't
> mind
> // the result is the same for us, after all
> Files.deleteIfExists(directory.toPath());
>}
>else if (directory.exists()) {
> // exists but is file, not directory
> // either an error from the caller, or concurrently a file got created
> throw new IOException(directory + " is not a directory");
>}
>// else: does not exist, which is okay (as if deleted)
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15219) LocalEnvironment is not initializing plugins
[
https://issues.apache.org/jira/browse/FLINK-15219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999166#comment-16999166
]
Arvid Heise commented on FLINK-15219:
-
Piotr suggested to provide a manual hook for users to initialize
`Plugins.initialize()` in their respective main. Then we can later support
non-filesystem plugins.
> LocalEnvironment is not initializing plugins
>
>
> Key: FLINK-15219
> URL: https://issues.apache.org/jira/browse/FLINK-15219
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.10.0
>Reporter: Arvid Heise
>Priority: Major
>
> Currently, plugins cannot be used in `LocalEnvironment` as `FileSystem`s are
> only initialized without `PluginManager`. In all other environments, the
> plugin manager is initialized in `TaskManagerRunner`.
>
> That means unless a user manually adds
> {code:java}
>final Configuration configuration = new Configuration();
>FileSystem.initialize(configuration,
> PluginUtils.createPluginManagerFromRootFolder(configuration));
> {code}
> to his entry point, he cannot use plugins, which means no s3 filesystems
> after FLINK-11956.
> We should add the proper initialization to the LocalExecutionEnvironment (at
> LocalExecutor?), so that the user does not have to it manually.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-13197) Fix Hive view row type mismatch when expanding in planner
[ https://issues.apache.org/jira/browse/FLINK-13197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Wysakowicz closed FLINK-13197. Resolution: Fixed Fixed in: master: 4e4dcf521748538aa164124fd613c9fe096ea604 1.10: 3d70c75477705b71acc83c010e7b89fce6c78c19 > Fix Hive view row type mismatch when expanding in planner > - > > Key: FLINK-13197 > URL: https://issues.apache.org/jira/browse/FLINK-13197 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Table SQL / Planner >Reporter: Bowen Li >Assignee: Rui Li >Priority: Major > Fix For: 1.10.0 > > > One goal of HiveCatalog and hive integration is to enable Flink-Hive > interoperability, that is Flink should understand existing Hive meta-objects, > and Hive meta-objects created thru Flink should be understood by Hive. > Taking an example of a Hive view v1 in HiveCatalog and database hc.db. Unlike > an equivalent Flink view whose full path in expanded query should be > hc.db.v1, the Hive view's full path in the expanded query should be db.v1 > such that Hive can understand it, no matter it's created by Hive or Flink. > [~lirui] can you help to ensure that Flink can also query Hive's view in both > Flink planner and Blink planner? > cc [~xuefuz] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13197) Fix Hive view row type mismatch when expanding in planner
[ https://issues.apache.org/jira/browse/FLINK-13197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Wysakowicz updated FLINK-13197: - Component/s: Table SQL / Planner > Fix Hive view row type mismatch when expanding in planner > - > > Key: FLINK-13197 > URL: https://issues.apache.org/jira/browse/FLINK-13197 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Table SQL / Planner >Reporter: Bowen Li >Assignee: Rui Li >Priority: Major > Fix For: 1.10.0 > > > One goal of HiveCatalog and hive integration is to enable Flink-Hive > interoperability, that is Flink should understand existing Hive meta-objects, > and Hive meta-objects created thru Flink should be understood by Hive. > Taking an example of a Hive view v1 in HiveCatalog and database hc.db. Unlike > an equivalent Flink view whose full path in expanded query should be > hc.db.v1, the Hive view's full path in the expanded query should be db.v1 > such that Hive can understand it, no matter it's created by Hive or Flink. > [~lirui] can you help to ensure that Flink can also query Hive's view in both > Flink planner and Blink planner? > cc [~xuefuz] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13197) Fix Hive view row type mismatch when expanding in planner
[ https://issues.apache.org/jira/browse/FLINK-13197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Wysakowicz updated FLINK-13197: - Issue Type: Bug (was: Test) > Fix Hive view row type mismatch when expanding in planner > - > > Key: FLINK-13197 > URL: https://issues.apache.org/jira/browse/FLINK-13197 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Reporter: Bowen Li >Assignee: Rui Li >Priority: Major > Fix For: 1.10.0 > > > One goal of HiveCatalog and hive integration is to enable Flink-Hive > interoperability, that is Flink should understand existing Hive meta-objects, > and Hive meta-objects created thru Flink should be understood by Hive. > Taking an example of a Hive view v1 in HiveCatalog and database hc.db. Unlike > an equivalent Flink view whose full path in expanded query should be > hc.db.v1, the Hive view's full path in the expanded query should be db.v1 > such that Hive can understand it, no matter it's created by Hive or Flink. > [~lirui] can you help to ensure that Flink can also query Hive's view in both > Flink planner and Blink planner? > cc [~xuefuz] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-15186) Column names and nullability mismatch when querying CatalogViews
[ https://issues.apache.org/jira/browse/FLINK-15186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Wysakowicz closed FLINK-15186. Fix Version/s: (was: 1.10.0) Resolution: Duplicate > Column names and nullability mismatch when querying CatalogViews > > > Key: FLINK-15186 > URL: https://issues.apache.org/jira/browse/FLINK-15186 > Project: Flink > Issue Type: Bug > Components: Table SQL / Legacy Planner, Table SQL / Planner >Reporter: Rui Li >Priority: Major > > When querying a CatalogView, the column names and nullability we get from the > catalog may differ from what we arrive at after compiling the view's query. > And we fail to expand the view due to this mismatch. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-15289) Run sql appear error of "Zero-length character strings have no serializable string representation".
[ https://issues.apache.org/jira/browse/FLINK-15289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu reassigned FLINK-15289: --- Assignee: Jingsong Lee > Run sql appear error of "Zero-length character strings have no serializable > string representation". > --- > > Key: FLINK-15289 > URL: https://issues.apache.org/jira/browse/FLINK-15289 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.10.0 >Reporter: xiaojin.wy >Assignee: Jingsong Lee >Priority: Critical > Fix For: 1.10.0 > > > *The sql is:* > CREATE TABLE `INT8_TBL` ( > q1 BIGINT, > q2 BIGINT > ) WITH ( > 'format.field-delimiter'=',', > 'connector.type'='filesystem', > 'format.derive-schema'='true', > > 'connector.path'='/defender_test_data/daily_regression_batch_postgres_1.10/test_bigint/sources/INT8_TBL.csv', > 'format.type'='csv' > ); > SELECT '' AS five, q1 AS plus, -q1 AS xm FROM INT8_TBL; > *The error detail is :* > 2019-12-17 15:35:07,026 ERROR org.apache.flink.table.client.SqlClient - SQL > Client must stop. Unexpected exception. This is a bug. Please consider filing > an issue. > org.apache.flink.table.api.TableException: Zero-length character strings > have no serializable string representation. > at > org.apache.flink.table.types.logical.CharType.asSerializableString(CharType.java:116) > at > org.apache.flink.table.descriptors.DescriptorProperties.putTableSchema(DescriptorProperties.java:218) > at > org.apache.flink.table.catalog.CatalogTableImpl.toProperties(CatalogTableImpl.java:75) > at > org.apache.flink.table.factories.TableFactoryUtil.findAndCreateTableSink(TableFactoryUtil.java:85) > at > org.apache.flink.table.client.gateway.local.LocalExecutor.executeQueryAndPersistInternal(LocalExecutor.java:688) > at > org.apache.flink.table.client.gateway.local.LocalExecutor.executeQueryAndPersist(LocalExecutor.java:488) > at org.apache.flink.table.client.cli.CliClient.callSelect(CliClient.java:601) > at > org.apache.flink.table.client.cli.CliClient.callCommand(CliClient.java:385) > at java.util.Optional.ifPresent(Optional.java:159) > at > org.apache.flink.table.client.cli.CliClient.submitSQLFile(CliClient.java:271) > at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:125) > at org.apache.flink.table.client.SqlClient.start(SqlClient.java:104) > at org.apache.flink.table.client.SqlClient.main(SqlClient.java:180) > *The input data is:* > 123,456 > 123,4567890123456789 > 4567890123456789,123 > 4567890123456789,4567890123456789 > 4567890123456789,-4567890123456789 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-15313) Can not insert decimal with precision into sink using TypeInformation
[
https://issues.apache.org/jira/browse/FLINK-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu reassigned FLINK-15313:
---
Assignee: Jark Wu
> Can not insert decimal with precision into sink using TypeInformation
> -
>
> Key: FLINK-15313
> URL: https://issues.apache.org/jira/browse/FLINK-15313
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: Jingsong Lee
>Assignee: Jark Wu
>Priority: Blocker
> Fix For: 1.10.0
>
>
> Sink DDL:
>
> {code:java}
> val sinkDDL =
> s"""
> |CREATE TABLE T2 (
> | d DECIMAL(10, 2),
> | cnt INT
> |) with (
> | 'connector.type' = 'filesystem',
> | 'connector.path' = '$sinkFilePath',
> | 'format.type' = 'csv',
> | 'format.field-delimiter' = ','
> |)
> """.stripMargin
> {code}
> Using blink with batch mode. (ensure insert BinaryRow into sink table)
>
> In FLINK-15124 , but we still use wrong precision to construct
> DataFormatConverter. This will lead to exception when inserting BinaryRow.
> (BinaryRow need correct precision to get)
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15289) Run sql appear error of "Zero-length character strings have no serializable string representation".
[ https://issues.apache.org/jira/browse/FLINK-15289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999128#comment-16999128 ] Timo Walther commented on FLINK-15289: -- Yes, we should do that. > Run sql appear error of "Zero-length character strings have no serializable > string representation". > --- > > Key: FLINK-15289 > URL: https://issues.apache.org/jira/browse/FLINK-15289 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.10.0 >Reporter: xiaojin.wy >Priority: Critical > Fix For: 1.10.0 > > > *The sql is:* > CREATE TABLE `INT8_TBL` ( > q1 BIGINT, > q2 BIGINT > ) WITH ( > 'format.field-delimiter'=',', > 'connector.type'='filesystem', > 'format.derive-schema'='true', > > 'connector.path'='/defender_test_data/daily_regression_batch_postgres_1.10/test_bigint/sources/INT8_TBL.csv', > 'format.type'='csv' > ); > SELECT '' AS five, q1 AS plus, -q1 AS xm FROM INT8_TBL; > *The error detail is :* > 2019-12-17 15:35:07,026 ERROR org.apache.flink.table.client.SqlClient - SQL > Client must stop. Unexpected exception. This is a bug. Please consider filing > an issue. > org.apache.flink.table.api.TableException: Zero-length character strings > have no serializable string representation. > at > org.apache.flink.table.types.logical.CharType.asSerializableString(CharType.java:116) > at > org.apache.flink.table.descriptors.DescriptorProperties.putTableSchema(DescriptorProperties.java:218) > at > org.apache.flink.table.catalog.CatalogTableImpl.toProperties(CatalogTableImpl.java:75) > at > org.apache.flink.table.factories.TableFactoryUtil.findAndCreateTableSink(TableFactoryUtil.java:85) > at > org.apache.flink.table.client.gateway.local.LocalExecutor.executeQueryAndPersistInternal(LocalExecutor.java:688) > at > org.apache.flink.table.client.gateway.local.LocalExecutor.executeQueryAndPersist(LocalExecutor.java:488) > at org.apache.flink.table.client.cli.CliClient.callSelect(CliClient.java:601) > at > org.apache.flink.table.client.cli.CliClient.callCommand(CliClient.java:385) > at java.util.Optional.ifPresent(Optional.java:159) > at > org.apache.flink.table.client.cli.CliClient.submitSQLFile(CliClient.java:271) > at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:125) > at org.apache.flink.table.client.SqlClient.start(SqlClient.java:104) > at org.apache.flink.table.client.SqlClient.main(SqlClient.java:180) > *The input data is:* > 123,456 > 123,4567890123456789 > 4567890123456789,123 > 4567890123456789,4567890123456789 > 4567890123456789,-4567890123456789 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15319) flink-end-to-end-tests-common-kafka fails due to timeout
Yun Tang created FLINK-15319:
Summary: flink-end-to-end-tests-common-kafka fails due to timeout
Key: FLINK-15319
URL: https://issues.apache.org/jira/browse/FLINK-15319
Project: Flink
Issue Type: Bug
Components: Connectors / Kafka
Affects Versions: 1.11.0
Reporter: Yun Tang
{code:java}
17:47:50.589 [INFO] Running
org.apache.flink.tests.util.kafka.StreamingKafkaITCase
17:51:35.273 [ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time
elapsed: 224.68 s <<< FAILURE! - in
org.apache.flink.tests.util.kafka.StreamingKafkaITCase
17:51:35.273 [ERROR] testKafka[2:
kafka-version:2.2.0](org.apache.flink.tests.util.kafka.StreamingKafkaITCase)
Time elapsed: 120.008 s <<< ERROR!
java.io.IOException: Process failed due to timeout.
{code}
Instance: [https://api.travis-ci.com/v3/job/268159458/log.txt]
It seems logs not uploaded to transfer.sh
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15318) RocksDBWriteBatchPerformanceTest.benchMark fails on ppc64le
[ https://issues.apache.org/jira/browse/FLINK-15318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999114#comment-16999114 ] Yun Tang commented on FLINK-15318: -- Which version of Flink did you verify? Did you observe a stable performance behavior before and then suddenly fail due to sync with new commits or you just come across this error for the first time when you just want to try it on ppc64le platform. Actually, Flink community lacks of benchmark on ppc64le environment and I noticed that RocksDB on ppc64le behaves not as good as those on linux64. > RocksDBWriteBatchPerformanceTest.benchMark fails on ppc64le > --- > > Key: FLINK-15318 > URL: https://issues.apache.org/jira/browse/FLINK-15318 > Project: Flink > Issue Type: Bug > Components: Benchmarks, Runtime / State Backends > Environment: arch: ppc64le > os: rhel7.6, ubuntu 18.04 > jdk: 8, 11 > mvn: 3.3.9, 3.6.2 >Reporter: Siddhesh Ghadi >Priority: Major > Attachments: surefire-report.txt > > > RocksDBWriteBatchPerformanceTest.benchMark fails due to TestTimedOut, however > when test-timeout is increased from 2s to 5s in > org/apache/flink/contrib/streaming/state/benchmark/RocksDBWriteBatchPerformanceTest.java:75, > it passes. Is this acceptable solution? > Note: Tests are ran inside a container. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-14944) Unstable test FlinkFnExecutionSyncTests.test_flink_fn_execution_pb2_synced
[
https://issues.apache.org/jira/browse/FLINK-14944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hequn Cheng closed FLINK-14944.
---
Resolution: Resolved
> Unstable test FlinkFnExecutionSyncTests.test_flink_fn_execution_pb2_synced
> --
>
> Key: FLINK-14944
> URL: https://issues.apache.org/jira/browse/FLINK-14944
> Project: Flink
> Issue Type: Improvement
> Components: API / Python
>Affects Versions: 1.10.0
>Reporter: Dian Fu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This tests failed occasionally:
> {code:java}
> force = 'True', output_dir = '/tmp/tmpkh6rmeig'
> def generate_proto_files(force=True,
> output_dir=DEFAULT_PYTHON_OUTPUT_PATH):
> try:
> import grpc_tools # noqa # pylint: disable=unused-import
> except ImportError:
> warnings.warn('Installing grpcio-tools is recommended for
> development.')
>
> proto_dirs = [os.path.join(PYFLINK_ROOT_PATH, path) for path in
> PROTO_PATHS]
> proto_files = sum(
> [glob.glob(os.path.join(d, '*.proto')) for d in proto_dirs], [])
> out_dir = os.path.join(PYFLINK_ROOT_PATH, output_dir)
> out_files = [path for path in glob.glob(os.path.join(out_dir,
> '*_pb2.py'))]
>
> if out_files and not proto_files and not force:
> # We have out_files but no protos; assume they're up to date.
> # This is actually the common case (e.g. installation from an
> sdist).
> logging.info('No proto files; using existing generated files.')
> return
>
> elif not out_files and not proto_files:
> raise RuntimeError(
> 'No proto files found in %s.' % proto_dirs)
>
> # Regenerate iff the proto files or this file are newer.
> elif force or not out_files or len(out_files) < len(proto_files) or (
> min(os.path.getmtime(path) for path in out_files)
> <= max(os.path.getmtime(path)
>for path in proto_files +
> [os.path.realpath(__file__)])):
> try:
> > from grpc_tools import protoc
> E ModuleNotFoundError: No module named 'grpc_tools'
> pyflink/gen_protos.py:70: ModuleNotFoundError
> During handling of the above exception, another exception occurred:
> self =
> testMethod=test_flink_fn_execution_pb2_synced>
> def test_flink_fn_execution_pb2_synced(self):
> > generate_proto_files('True', self.tempdir)
> pyflink/fn_execution/tests/test_flink_fn_execution_pb2_synced.py:35:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> pyflink/gen_protos.py:83: in generate_proto_files
> target=_install_grpcio_tools_and_generate_proto_files(force, output_dir))
> pyflink/gen_protos.py:131: in _install_grpcio_tools_and_generate_proto_files
> '--upgrade', GRPC_TOOLS, "-I"])
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> popenargs =
> (['/home/travis/build/apache/flink/flink-python/.tox/py36/bin/python', '-m',
> 'pip', 'install', '--prefix',
> '/home/travis/build/apache/flink/flink-python/pyflink/../.eggs/grpcio-wheels',
> ...],)
> kwargs = {}, retcode = 2
> cmd = ['/home/travis/build/apache/flink/flink-python/.tox/py36/bin/python',
> '-m', 'pip', 'install', '--prefix',
> '/home/travis/build/apache/flink/flink-python/pyflink/../.eggs/grpcio-wheels',
> ...]
> def check_call(*popenargs, **kwargs):
> """Run command with arguments. Wait for command to complete. If
> the exit code was zero then return, otherwise raise
> CalledProcessError. The CalledProcessError object will have the
> return code in the returncode attribute.
>
> The arguments are the same as for the call function. Example:
>
> check_call(["ls", "-l"])
> """
> retcode = call(*popenargs, **kwargs)
> if retcode:
> cmd = kwargs.get("args")
> if cmd is None:
> cmd = popenargs[0]
> > raise CalledProcessError(retcode, cmd)
> E subprocess.CalledProcessError: Command
> '['/home/travis/build/apache/flink/flink-python/.tox/py36/bin/python', '-m',
> 'pip', 'install', '--prefix',
> '/home/travis/build/apache/flink/flink-python/pyflink/../.eggs/grpcio-wheels',
> '--build',
> '/home/travis/build/apache/flink/flink-python/pyflink/../.eggs/grpcio-wheels-build',
> '--upgrade', 'grpcio-tools>=1.3.5,<=1.14.2', '-I']' returned non-zero exit
> status 2.
> dev/.conda/envs/3.6/lib/python3.6/subprocess.py:311: CalledProcessError
> Captured stdout setup
[jira] [Updated] (FLINK-15268) Shaded Hadoop S3A end-to-end test fails on travis
[
https://issues.apache.org/jira/browse/FLINK-15268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao updated FLINK-15268:
-
Fix Version/s: 1.10.0
> Shaded Hadoop S3A end-to-end test fails on travis
> -
>
> Key: FLINK-15268
> URL: https://issues.apache.org/jira/browse/FLINK-15268
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Tests
>Affects Versions: 1.10.0
>Reporter: Yu Li
>Priority: Blocker
> Fix For: 1.10.0
>
>
> As titled, the 'Shaded Hadoop S3A end-to-end test' case failed with below
> error:
> {code}
> java.io.IOException: regular upload failed: java.lang.NoClassDefFoundError:
> javax/xml/bind/JAXBException
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AUtils.extractException(S3AUtils.java:291)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3ABlockOutputStream.putObject(S3ABlockOutputStream.java:448)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3ABlockOutputStream.close(S3ABlockOutputStream.java:360)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
> at
> org.apache.flink.fs.s3.common.hadoop.HadoopDataOutputStream.close(HadoopDataOutputStream.java:52)
> at
> org.apache.flink.core.fs.ClosingFSDataOutputStream.close(ClosingFSDataOutputStream.java:64)
> at
> java.base/java.io.FilterOutputStream.close(FilterOutputStream.java:188)
> at java.base/sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:341)
> at java.base/sun.nio.cs.StreamEncoder.close(StreamEncoder.java:161)
> at
> java.base/java.io.OutputStreamWriter.close(OutputStreamWriter.java:258)
> at
> org.apache.flink.api.java.io.CsvOutputFormat.close(CsvOutputFormat.java:170)
> at
> org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:227)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:702)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:527)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException
> at
> org.apache.flink.fs.s3base.shaded.com.amazonaws.util.Md5Utils.md5AsBase64(Md5Utils.java:104)
> at
> org.apache.flink.fs.s3base.shaded.com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1647)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AFileSystem.putObjectDirect(S3AFileSystem.java:1531)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.WriteOperationHelper.lambda$putObject$5(WriteOperationHelper.java:426)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:260)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:317)
> {code}
> https://api.travis-ci.org/v3/job/625037121/log.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15268) Shaded Hadoop S3A end-to-end test fails on travis
[
https://issues.apache.org/jira/browse/FLINK-15268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999092#comment-16999092
]
Arvid Heise commented on FLINK-15268:
-
Is this related to Java 11 not having jaxb anymore?
> Shaded Hadoop S3A end-to-end test fails on travis
> -
>
> Key: FLINK-15268
> URL: https://issues.apache.org/jira/browse/FLINK-15268
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Tests
>Affects Versions: 1.10.0
>Reporter: Yu Li
>Priority: Blocker
>
> As titled, the 'Shaded Hadoop S3A end-to-end test' case failed with below
> error:
> {code}
> java.io.IOException: regular upload failed: java.lang.NoClassDefFoundError:
> javax/xml/bind/JAXBException
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AUtils.extractException(S3AUtils.java:291)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3ABlockOutputStream.putObject(S3ABlockOutputStream.java:448)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3ABlockOutputStream.close(S3ABlockOutputStream.java:360)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
> at
> org.apache.flink.fs.s3.common.hadoop.HadoopDataOutputStream.close(HadoopDataOutputStream.java:52)
> at
> org.apache.flink.core.fs.ClosingFSDataOutputStream.close(ClosingFSDataOutputStream.java:64)
> at
> java.base/java.io.FilterOutputStream.close(FilterOutputStream.java:188)
> at java.base/sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:341)
> at java.base/sun.nio.cs.StreamEncoder.close(StreamEncoder.java:161)
> at
> java.base/java.io.OutputStreamWriter.close(OutputStreamWriter.java:258)
> at
> org.apache.flink.api.java.io.CsvOutputFormat.close(CsvOutputFormat.java:170)
> at
> org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:227)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:702)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:527)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException
> at
> org.apache.flink.fs.s3base.shaded.com.amazonaws.util.Md5Utils.md5AsBase64(Md5Utils.java:104)
> at
> org.apache.flink.fs.s3base.shaded.com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1647)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AFileSystem.putObjectDirect(S3AFileSystem.java:1531)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.WriteOperationHelper.lambda$putObject$5(WriteOperationHelper.java:426)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:260)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:317)
> {code}
> https://api.travis-ci.org/v3/job/625037121/log.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-9009) Error| You are creating too many HashedWheelTimer instances. HashedWheelTimer is a shared resource that must be reused across the application, so that only a few
[ https://issues.apache.org/jira/browse/FLINK-9009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999056#comment-16999056 ] Subramanyam Ramanathan edited comment on FLINK-9009 at 12/18/19 11:52 AM: -- Hi, I'm seeing a similar issue when using flink with Pulsar source + Sink. I am using flink 1.8.2 and pulsar v2.4.2, on an 8 cpu 16GB RAM vm, running centos 7 I have 20 map transformations, each with it's own source and sink, and parallelism set to 8. If the source and sink are Kafka, then I don't see any error, and top command shows me 4% memory usage. When I use pulsar source+sink, the java process consumes *40 %* memory which is around 6GB. This happens even if I have not streamed any data. TaskManager heap size was set to 1024M and I don't see any outOfMemory errors. I think the increase in memory usage is because flink uses the off heap memory which gets set by flink to -XX:MaxDirectMemorySize=8388607T,and something with Pulsar Source/Sink is causing it to consume a lot of it. I also see the message in the logs mentioned in the title : *"Error: You are creating too many HashedWheelTimer instances. HashedWheelTimer is a shared resource."* Can you please help me understand the behaviour of the off heap memory in this case, and why it grows so much? Is there any fix that is planned for this ? Or any way I can work around this ? was (Author: subbu-ramanathan107): Hi, I'm seeing a similar issue when using flink with Pulsar source + Sink. I am using flink 1.8.2 and pulsar v2.4.2, on an 8 cpu 16GB RAM vm, running centos 7 I have 20 map transformations, each with it's own source and sink, and parallelism set to 8. If the source and sink are Kafka, then I don't see any error, and top command shows me 4% memory usage. When I use pulsar source+sink, the java process consumes *40 %* memory. This happens even if I have not streamed any data. TaskManager heap size was set to 1024M and I don't see any outOfMemory errors. I think the increase in memory usage is because flink uses the off heap memory which gets set by flink to -XX:MaxDirectMemorySize=8388607T,and something with Pulsar Source/Sink is causing it to consume a lot of it. I also see the message in the logs mentioned in the title : *"Error: You are creating too many HashedWheelTimer instances. HashedWheelTimer is a shared resource."* Can you please help me understand the behaviour of the off heap memory in this case, and why it grows so much? Is there any fix that is planned for this ? Or any way I can work around this ? > Error| You are creating too many HashedWheelTimer instances. > HashedWheelTimer is a shared resource that must be reused across the > application, so that only a few instances are created. > - > > Key: FLINK-9009 > URL: https://issues.apache.org/jira/browse/FLINK-9009 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka > Environment: Pass platform: Openshift >Reporter: Pankaj >Priority: Major > > Steps to reproduce: > 1- Flink with Kafka as a consumer -> Writing stream to Cassandra using flink > cassandra sink. > 2- In memory Job manager and task manager with checkpointing 5000ms. > 3- env.setpararllelism(10)-> As kafka topic has 10 partition. > 4- There are around 13 unique streams in a single flink run time environment > which are reading from kafka -> processing and writing to cassandra. > Hardware: CPU 200 milli core . It is deployed on Paas platform on one node > Memory: 526 MB. > > When i start the server, It starts flink and all off sudden stops with above > error. It also shows out of memory error. > > It would be nice if any body can suggest if something is wrong. > > Maven: > flink-connector-cassandra_2.11: 1.3.2 > flink-streaming-java_2.11: 1.4.0 > flink-connector-kafka-0.11_2.11:1.4.0 > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-9009) Error| You are creating too many HashedWheelTimer instances. HashedWheelTimer is a shared resource that must be reused across the application, so that only a few
[ https://issues.apache.org/jira/browse/FLINK-9009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999056#comment-16999056 ] Subramanyam Ramanathan edited comment on FLINK-9009 at 12/18/19 11:52 AM: -- Hi, I'm seeing a similar issue when using flink with Pulsar source + Sink. I am using flink 1.8.2 and pulsar v2.4.2, on an 8 cpu 16GB RAM vm, running centos 7 I have 20 map transformations, each with it's own source and sink, and parallelism set to 8. If the source and sink are Kafka, then I don't see any error, and top command shows me 4% memory usage. When I use pulsar source+sink, the java process consumes *40 %* memory. This happens even if I have not streamed any data. TaskManager heap size was set to 1024M and I don't see any outOfMemory errors. I think the increase in memory usage is because flink uses the off heap memory which gets set by flink to -XX:MaxDirectMemorySize=8388607T,and something with Pulsar Source/Sink is causing it to consume a lot of it. I also see the message in the logs mentioned in the title : *"Error: You are creating too many HashedWheelTimer instances. HashedWheelTimer is a shared resource."* Can you please help me understand the behaviour of the off heap memory in this case, and why it grows so much? Is there any fix that is planned for this ? Or any way I can work around this ? was (Author: subbu-ramanathan107): Hi, I'm seeing a similar issue when using flink with Pulsar source + Sink. I am using flink 1.8.2 and pulsar v2.4.2, on an 8 cpu 16GB RAM vm, running centos 7 I have 20 map transformations, each with it's own source and sink, and parallelism set to 8. If the source and sink are Kafka, then I don't see any error, and top command shows me 4% memory usage. When I use pulsar source+sink, the java process consumes *40 %* memory. This happens even if I have not streamed any data. My heap size was set to 1024M and I don't see any outOfMemory errors. I think the increase in memory usage is because flink uses the off heap memory which gets set by flink to -XX:MaxDirectMemorySize=8388607T,and something with Pulsar Source/Sink is causing it to consume a lot of it. I also see the message in the logs mentioned in the title : *"Error: You are creating too many HashedWheelTimer instances. HashedWheelTimer is a shared resource."* Can you please help me understand the behaviour of the off heap memory in this case, and why it grows so much? Is there any fix that is planned for this ? Or any way I can work around this ? > Error| You are creating too many HashedWheelTimer instances. > HashedWheelTimer is a shared resource that must be reused across the > application, so that only a few instances are created. > - > > Key: FLINK-9009 > URL: https://issues.apache.org/jira/browse/FLINK-9009 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka > Environment: Pass platform: Openshift >Reporter: Pankaj >Priority: Major > > Steps to reproduce: > 1- Flink with Kafka as a consumer -> Writing stream to Cassandra using flink > cassandra sink. > 2- In memory Job manager and task manager with checkpointing 5000ms. > 3- env.setpararllelism(10)-> As kafka topic has 10 partition. > 4- There are around 13 unique streams in a single flink run time environment > which are reading from kafka -> processing and writing to cassandra. > Hardware: CPU 200 milli core . It is deployed on Paas platform on one node > Memory: 526 MB. > > When i start the server, It starts flink and all off sudden stops with above > error. It also shows out of memory error. > > It would be nice if any body can suggest if something is wrong. > > Maven: > flink-connector-cassandra_2.11: 1.3.2 > flink-streaming-java_2.11: 1.4.0 > flink-connector-kafka-0.11_2.11:1.4.0 > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-9009) Error| You are creating too many HashedWheelTimer instances. HashedWheelTimer is a shared resource that must be reused across the application, so that only a few
[ https://issues.apache.org/jira/browse/FLINK-9009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999056#comment-16999056 ] Subramanyam Ramanathan edited comment on FLINK-9009 at 12/18/19 11:52 AM: -- Hi, I'm seeing a similar issue when using flink with Pulsar source + Sink. I am using flink 1.8.2 and pulsar v2.4.2, on an 8 cpu 16GB RAM vm, running centos 7 I have 20 map transformations, each with it's own source and sink, and parallelism set to 8. If the source and sink are Kafka, then I don't see any error, and top command shows me 4% memory usage. When I use pulsar source+sink, the java process consumes *40 %* memory which is around 6GB. This happens even if I have not streamed any data. TaskManager heap size was set to 1024M and I don't see any outOfMemory errors. I think the increase in memory usage is because flink uses the off heap memory which gets set by flink to -XX:MaxDirectMemorySize=8388607T,and something with Pulsar Source/Sink is causing it to consume a lot of it. I also see the error message in the logs mentioned in the title : *"Error: You are creating too many HashedWheelTimer instances. HashedWheelTimer is a shared resource."* Can you please help me understand the behaviour of the off heap memory in this case, and why it grows so much? Is there any fix that is planned for this ? Or any way I can work around this ? was (Author: subbu-ramanathan107): Hi, I'm seeing a similar issue when using flink with Pulsar source + Sink. I am using flink 1.8.2 and pulsar v2.4.2, on an 8 cpu 16GB RAM vm, running centos 7 I have 20 map transformations, each with it's own source and sink, and parallelism set to 8. If the source and sink are Kafka, then I don't see any error, and top command shows me 4% memory usage. When I use pulsar source+sink, the java process consumes *40 %* memory which is around 6GB. This happens even if I have not streamed any data. TaskManager heap size was set to 1024M and I don't see any outOfMemory errors. I think the increase in memory usage is because flink uses the off heap memory which gets set by flink to -XX:MaxDirectMemorySize=8388607T,and something with Pulsar Source/Sink is causing it to consume a lot of it. I also see the message in the logs mentioned in the title : *"Error: You are creating too many HashedWheelTimer instances. HashedWheelTimer is a shared resource."* Can you please help me understand the behaviour of the off heap memory in this case, and why it grows so much? Is there any fix that is planned for this ? Or any way I can work around this ? > Error| You are creating too many HashedWheelTimer instances. > HashedWheelTimer is a shared resource that must be reused across the > application, so that only a few instances are created. > - > > Key: FLINK-9009 > URL: https://issues.apache.org/jira/browse/FLINK-9009 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka > Environment: Pass platform: Openshift >Reporter: Pankaj >Priority: Major > > Steps to reproduce: > 1- Flink with Kafka as a consumer -> Writing stream to Cassandra using flink > cassandra sink. > 2- In memory Job manager and task manager with checkpointing 5000ms. > 3- env.setpararllelism(10)-> As kafka topic has 10 partition. > 4- There are around 13 unique streams in a single flink run time environment > which are reading from kafka -> processing and writing to cassandra. > Hardware: CPU 200 milli core . It is deployed on Paas platform on one node > Memory: 526 MB. > > When i start the server, It starts flink and all off sudden stops with above > error. It also shows out of memory error. > > It would be nice if any body can suggest if something is wrong. > > Maven: > flink-connector-cassandra_2.11: 1.3.2 > flink-streaming-java_2.11: 1.4.0 > flink-connector-kafka-0.11_2.11:1.4.0 > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14926) Make sure no resource leak of RocksObject
[
https://issues.apache.org/jira/browse/FLINK-14926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999082#comment-16999082
]
Gary Yao commented on FLINK-14926:
--
What's the state here? There are already commits in release-1.10, e.g.,
208de7c29f2ca71b526f98f603cf4a55103c5520.
> Make sure no resource leak of RocksObject
> -
>
> Key: FLINK-14926
> URL: https://issues.apache.org/jira/browse/FLINK-14926
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
>Affects Versions: 1.6.4, 1.7.2, 1.8.2, 1.9.1
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When investigating FLINK-14484 to allow setting {{WriteBufferManager}} via
> options, we find it necessary to supply a {{close}} method in
> {{OptionsFactory}} to make sure resources configured in options like
> {{WriteBufferManager}} could/would be released.
> What's more, we're having potential risk of resource leak in
> {{PredefinedOptions}} such as the created [BloomFilter
> instance|https://github.com/apache/flink/blob/e14bd50fed42e89674ba9d01a231d5c5e59f490c/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/PredefinedOptions.java#L155]
> is never closed (regression caused by FLINK-7220 changes), and we should
> also fix this and add a {{close}} method in {{PredefinedOptions}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14944) Unstable test FlinkFnExecutionSyncTests.test_flink_fn_execution_pb2_synced
[
https://issues.apache.org/jira/browse/FLINK-14944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hequn Cheng updated FLINK-14944:
Issue Type: Improvement (was: Bug)
> Unstable test FlinkFnExecutionSyncTests.test_flink_fn_execution_pb2_synced
> --
>
> Key: FLINK-14944
> URL: https://issues.apache.org/jira/browse/FLINK-14944
> Project: Flink
> Issue Type: Improvement
> Components: API / Python
>Affects Versions: 1.10.0
>Reporter: Dian Fu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This tests failed occasionally:
> {code:java}
> force = 'True', output_dir = '/tmp/tmpkh6rmeig'
> def generate_proto_files(force=True,
> output_dir=DEFAULT_PYTHON_OUTPUT_PATH):
> try:
> import grpc_tools # noqa # pylint: disable=unused-import
> except ImportError:
> warnings.warn('Installing grpcio-tools is recommended for
> development.')
>
> proto_dirs = [os.path.join(PYFLINK_ROOT_PATH, path) for path in
> PROTO_PATHS]
> proto_files = sum(
> [glob.glob(os.path.join(d, '*.proto')) for d in proto_dirs], [])
> out_dir = os.path.join(PYFLINK_ROOT_PATH, output_dir)
> out_files = [path for path in glob.glob(os.path.join(out_dir,
> '*_pb2.py'))]
>
> if out_files and not proto_files and not force:
> # We have out_files but no protos; assume they're up to date.
> # This is actually the common case (e.g. installation from an
> sdist).
> logging.info('No proto files; using existing generated files.')
> return
>
> elif not out_files and not proto_files:
> raise RuntimeError(
> 'No proto files found in %s.' % proto_dirs)
>
> # Regenerate iff the proto files or this file are newer.
> elif force or not out_files or len(out_files) < len(proto_files) or (
> min(os.path.getmtime(path) for path in out_files)
> <= max(os.path.getmtime(path)
>for path in proto_files +
> [os.path.realpath(__file__)])):
> try:
> > from grpc_tools import protoc
> E ModuleNotFoundError: No module named 'grpc_tools'
> pyflink/gen_protos.py:70: ModuleNotFoundError
> During handling of the above exception, another exception occurred:
> self =
> testMethod=test_flink_fn_execution_pb2_synced>
> def test_flink_fn_execution_pb2_synced(self):
> > generate_proto_files('True', self.tempdir)
> pyflink/fn_execution/tests/test_flink_fn_execution_pb2_synced.py:35:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> pyflink/gen_protos.py:83: in generate_proto_files
> target=_install_grpcio_tools_and_generate_proto_files(force, output_dir))
> pyflink/gen_protos.py:131: in _install_grpcio_tools_and_generate_proto_files
> '--upgrade', GRPC_TOOLS, "-I"])
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> popenargs =
> (['/home/travis/build/apache/flink/flink-python/.tox/py36/bin/python', '-m',
> 'pip', 'install', '--prefix',
> '/home/travis/build/apache/flink/flink-python/pyflink/../.eggs/grpcio-wheels',
> ...],)
> kwargs = {}, retcode = 2
> cmd = ['/home/travis/build/apache/flink/flink-python/.tox/py36/bin/python',
> '-m', 'pip', 'install', '--prefix',
> '/home/travis/build/apache/flink/flink-python/pyflink/../.eggs/grpcio-wheels',
> ...]
> def check_call(*popenargs, **kwargs):
> """Run command with arguments. Wait for command to complete. If
> the exit code was zero then return, otherwise raise
> CalledProcessError. The CalledProcessError object will have the
> return code in the returncode attribute.
>
> The arguments are the same as for the call function. Example:
>
> check_call(["ls", "-l"])
> """
> retcode = call(*popenargs, **kwargs)
> if retcode:
> cmd = kwargs.get("args")
> if cmd is None:
> cmd = popenargs[0]
> > raise CalledProcessError(retcode, cmd)
> E subprocess.CalledProcessError: Command
> '['/home/travis/build/apache/flink/flink-python/.tox/py36/bin/python', '-m',
> 'pip', 'install', '--prefix',
> '/home/travis/build/apache/flink/flink-python/pyflink/../.eggs/grpcio-wheels',
> '--build',
> '/home/travis/build/apache/flink/flink-python/pyflink/../.eggs/grpcio-wheels-build',
> '--upgrade', 'grpcio-tools>=1.3.5,<=1.14.2', '-I']' returned non-zero exit
> status 2.
> dev/.conda/envs/3.6/lib/python3.6/subprocess.py:311: CalledProcessError
>
[jira] [Commented] (FLINK-14944) Unstable test FlinkFnExecutionSyncTests.test_flink_fn_execution_pb2_synced
[
https://issues.apache.org/jira/browse/FLINK-14944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999080#comment-16999080
]
Hequn Cheng commented on FLINK-14944:
-
Resloved
in 1.11.0 via a7884abde4646745a30f964181c10c9f04b03b4a
in 1.10.0 via 729e87cf3371d8b42383564d45caaf4518afc4c9
> Unstable test FlinkFnExecutionSyncTests.test_flink_fn_execution_pb2_synced
> --
>
> Key: FLINK-14944
> URL: https://issues.apache.org/jira/browse/FLINK-14944
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.10.0
>Reporter: Dian Fu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This tests failed occasionally:
> {code:java}
> force = 'True', output_dir = '/tmp/tmpkh6rmeig'
> def generate_proto_files(force=True,
> output_dir=DEFAULT_PYTHON_OUTPUT_PATH):
> try:
> import grpc_tools # noqa # pylint: disable=unused-import
> except ImportError:
> warnings.warn('Installing grpcio-tools is recommended for
> development.')
>
> proto_dirs = [os.path.join(PYFLINK_ROOT_PATH, path) for path in
> PROTO_PATHS]
> proto_files = sum(
> [glob.glob(os.path.join(d, '*.proto')) for d in proto_dirs], [])
> out_dir = os.path.join(PYFLINK_ROOT_PATH, output_dir)
> out_files = [path for path in glob.glob(os.path.join(out_dir,
> '*_pb2.py'))]
>
> if out_files and not proto_files and not force:
> # We have out_files but no protos; assume they're up to date.
> # This is actually the common case (e.g. installation from an
> sdist).
> logging.info('No proto files; using existing generated files.')
> return
>
> elif not out_files and not proto_files:
> raise RuntimeError(
> 'No proto files found in %s.' % proto_dirs)
>
> # Regenerate iff the proto files or this file are newer.
> elif force or not out_files or len(out_files) < len(proto_files) or (
> min(os.path.getmtime(path) for path in out_files)
> <= max(os.path.getmtime(path)
>for path in proto_files +
> [os.path.realpath(__file__)])):
> try:
> > from grpc_tools import protoc
> E ModuleNotFoundError: No module named 'grpc_tools'
> pyflink/gen_protos.py:70: ModuleNotFoundError
> During handling of the above exception, another exception occurred:
> self =
> testMethod=test_flink_fn_execution_pb2_synced>
> def test_flink_fn_execution_pb2_synced(self):
> > generate_proto_files('True', self.tempdir)
> pyflink/fn_execution/tests/test_flink_fn_execution_pb2_synced.py:35:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> pyflink/gen_protos.py:83: in generate_proto_files
> target=_install_grpcio_tools_and_generate_proto_files(force, output_dir))
> pyflink/gen_protos.py:131: in _install_grpcio_tools_and_generate_proto_files
> '--upgrade', GRPC_TOOLS, "-I"])
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> popenargs =
> (['/home/travis/build/apache/flink/flink-python/.tox/py36/bin/python', '-m',
> 'pip', 'install', '--prefix',
> '/home/travis/build/apache/flink/flink-python/pyflink/../.eggs/grpcio-wheels',
> ...],)
> kwargs = {}, retcode = 2
> cmd = ['/home/travis/build/apache/flink/flink-python/.tox/py36/bin/python',
> '-m', 'pip', 'install', '--prefix',
> '/home/travis/build/apache/flink/flink-python/pyflink/../.eggs/grpcio-wheels',
> ...]
> def check_call(*popenargs, **kwargs):
> """Run command with arguments. Wait for command to complete. If
> the exit code was zero then return, otherwise raise
> CalledProcessError. The CalledProcessError object will have the
> return code in the returncode attribute.
>
> The arguments are the same as for the call function. Example:
>
> check_call(["ls", "-l"])
> """
> retcode = call(*popenargs, **kwargs)
> if retcode:
> cmd = kwargs.get("args")
> if cmd is None:
> cmd = popenargs[0]
> > raise CalledProcessError(retcode, cmd)
> E subprocess.CalledProcessError: Command
> '['/home/travis/build/apache/flink/flink-python/.tox/py36/bin/python', '-m',
> 'pip', 'install', '--prefix',
> '/home/travis/build/apache/flink/flink-python/pyflink/../.eggs/grpcio-wheels',
> '--build',
> '/home/travis/build/apache/flink/flink-python/pyflink/../.eggs/grpcio-wheels-build',
> '--upgrade', 'grpcio-tools>=1.3.5,<=1.14.2', '-I']' returned non-zero exit
> status
[jira] [Reopened] (FLINK-15316) SQL Client end-to-end test (Old planner) failed on Travis
[
https://issues.apache.org/jira/browse/FLINK-15316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao reopened FLINK-15316:
--
> SQL Client end-to-end test (Old planner) failed on Travis
> -
>
> Key: FLINK-15316
> URL: https://issues.apache.org/jira/browse/FLINK-15316
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client, Tests
>Affects Versions: 1.10.0
>Reporter: Gary Yao
>Priority: Critical
> Labels: test-stability
> Fix For: 1.10.0
>
>
> https://api.travis-ci.org/v3/job/626286501/log.txt
> {noformat}
> Checking for errors...
> Found error in log files:
> ...
> org.apache.flink.table.client.gateway.SqlExecutionException: Invalid SQL
> update statement.
> at
> org.apache.flink.table.client.gateway.local.LocalExecutor.applyUpdate(LocalExecutor.java:697)
> at
> org.apache.flink.table.client.gateway.local.LocalExecutor.executeUpdateInternal(LocalExecutor.java:576)
> at
> org.apache.flink.table.client.gateway.local.LocalExecutor.executeUpdate(LocalExecutor.java:527)
> at
> org.apache.flink.table.client.cli.CliClient.callInsertInto(CliClient.java:535)
> at
> org.apache.flink.table.client.cli.CliClient.lambda$submitUpdate$0(CliClient.java:231)
> at java.util.Optional.map(Optional.java:215)
> at
> org.apache.flink.table.client.cli.CliClient.submitUpdate(CliClient.java:228)
> at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:129)
> at org.apache.flink.table.client.SqlClient.start(SqlClient.java:104)
> at org.apache.flink.table.client.SqlClient.main(SqlClient.java:178)
> Caused by: java.lang.RuntimeException: Error while applying rule
> PushProjectIntoTableSourceScanRule, args
> [rel#88:FlinkLogicalCalc.LOGICAL(input=RelSubset#87,expr#0..2={inputs},expr#3=IS
> NOT NULL($t1),user=$t1,rowtime=$t0,$condition=$t3),
> Scan(table:[default_catalog, default_database, JsonSourceTable],
> fields:(rowtime, user, event), source:KafkaTableSource(rowtime, user, event))]
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:235)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631)
> at
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:327)
> at
> org.apache.flink.table.plan.Optimizer.runVolcanoPlanner(Optimizer.scala:280)
> at
> org.apache.flink.table.plan.Optimizer.optimizeLogicalPlan(Optimizer.scala:199)
> at
> org.apache.flink.table.plan.StreamOptimizer.optimize(StreamOptimizer.scala:66)
> at
> org.apache.flink.table.planner.StreamPlanner.writeToUpsertSink(StreamPlanner.scala:353)
> at
> org.apache.flink.table.planner.StreamPlanner.org$apache$flink$table$planner$StreamPlanner$$writeToSink(StreamPlanner.scala:281)
> at
> org.apache.flink.table.planner.StreamPlanner$$anonfun$2.apply(StreamPlanner.scala:166)
> at
> org.apache.flink.table.planner.StreamPlanner$$anonfun$2.apply(StreamPlanner.scala:145)
> at scala.Option.map(Option.scala:146)
> at
> org.apache.flink.table.planner.StreamPlanner.org$apache$flink$table$planner$StreamPlanner$$translate(StreamPlanner.scala:145)
> at
> org.apache.flink.table.planner.StreamPlanner$$anonfun$translate$1.apply(StreamPlanner.scala:117)
> at
> org.apache.flink.table.planner.StreamPlanner$$anonfun$translate$1.apply(StreamPlanner.scala:117)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at scala.collection.Iterator$class.foreach(Iterator.scala:891)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
> at scala.collection.AbstractTraversable.map(Traversable.scala:104)
> at
> org.apache.flink.table.planner.StreamPlanner.translate(StreamPlanner.scala:117)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.translate(TableEnvironmentImpl.java:680)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlUpdate(TableEnvironmentImpl.java:493)
> at
> org.apache.flink.table.api.java.internal.StreamTableEnvironmentImpl.sqlUpdate(StreamTableEnvironmentImpl.java:331)
> at
> org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$applyUpdate$15(LocalExecutor.java:690)
> at
> org.apache.flink.table.client.gateway.local.ExecutionContext.wrapClassLoader(ExecutionContext.java:240)
> at
>
[jira] [Closed] (FLINK-15316) SQL Client end-to-end test (Old planner) failed on Travis
[
https://issues.apache.org/jira/browse/FLINK-15316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao closed FLINK-15316.
Resolution: Fixed
cc: [~dwysakowicz]
> SQL Client end-to-end test (Old planner) failed on Travis
> -
>
> Key: FLINK-15316
> URL: https://issues.apache.org/jira/browse/FLINK-15316
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client, Tests
>Affects Versions: 1.10.0
>Reporter: Gary Yao
>Priority: Critical
> Labels: test-stability
> Fix For: 1.10.0
>
>
> https://api.travis-ci.org/v3/job/626286501/log.txt
> {noformat}
> Checking for errors...
> Found error in log files:
> ...
> org.apache.flink.table.client.gateway.SqlExecutionException: Invalid SQL
> update statement.
> at
> org.apache.flink.table.client.gateway.local.LocalExecutor.applyUpdate(LocalExecutor.java:697)
> at
> org.apache.flink.table.client.gateway.local.LocalExecutor.executeUpdateInternal(LocalExecutor.java:576)
> at
> org.apache.flink.table.client.gateway.local.LocalExecutor.executeUpdate(LocalExecutor.java:527)
> at
> org.apache.flink.table.client.cli.CliClient.callInsertInto(CliClient.java:535)
> at
> org.apache.flink.table.client.cli.CliClient.lambda$submitUpdate$0(CliClient.java:231)
> at java.util.Optional.map(Optional.java:215)
> at
> org.apache.flink.table.client.cli.CliClient.submitUpdate(CliClient.java:228)
> at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:129)
> at org.apache.flink.table.client.SqlClient.start(SqlClient.java:104)
> at org.apache.flink.table.client.SqlClient.main(SqlClient.java:178)
> Caused by: java.lang.RuntimeException: Error while applying rule
> PushProjectIntoTableSourceScanRule, args
> [rel#88:FlinkLogicalCalc.LOGICAL(input=RelSubset#87,expr#0..2={inputs},expr#3=IS
> NOT NULL($t1),user=$t1,rowtime=$t0,$condition=$t3),
> Scan(table:[default_catalog, default_database, JsonSourceTable],
> fields:(rowtime, user, event), source:KafkaTableSource(rowtime, user, event))]
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:235)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631)
> at
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:327)
> at
> org.apache.flink.table.plan.Optimizer.runVolcanoPlanner(Optimizer.scala:280)
> at
> org.apache.flink.table.plan.Optimizer.optimizeLogicalPlan(Optimizer.scala:199)
> at
> org.apache.flink.table.plan.StreamOptimizer.optimize(StreamOptimizer.scala:66)
> at
> org.apache.flink.table.planner.StreamPlanner.writeToUpsertSink(StreamPlanner.scala:353)
> at
> org.apache.flink.table.planner.StreamPlanner.org$apache$flink$table$planner$StreamPlanner$$writeToSink(StreamPlanner.scala:281)
> at
> org.apache.flink.table.planner.StreamPlanner$$anonfun$2.apply(StreamPlanner.scala:166)
> at
> org.apache.flink.table.planner.StreamPlanner$$anonfun$2.apply(StreamPlanner.scala:145)
> at scala.Option.map(Option.scala:146)
> at
> org.apache.flink.table.planner.StreamPlanner.org$apache$flink$table$planner$StreamPlanner$$translate(StreamPlanner.scala:145)
> at
> org.apache.flink.table.planner.StreamPlanner$$anonfun$translate$1.apply(StreamPlanner.scala:117)
> at
> org.apache.flink.table.planner.StreamPlanner$$anonfun$translate$1.apply(StreamPlanner.scala:117)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at scala.collection.Iterator$class.foreach(Iterator.scala:891)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
> at scala.collection.AbstractTraversable.map(Traversable.scala:104)
> at
> org.apache.flink.table.planner.StreamPlanner.translate(StreamPlanner.scala:117)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.translate(TableEnvironmentImpl.java:680)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlUpdate(TableEnvironmentImpl.java:493)
> at
> org.apache.flink.table.api.java.internal.StreamTableEnvironmentImpl.sqlUpdate(StreamTableEnvironmentImpl.java:331)
> at
> org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$applyUpdate$15(LocalExecutor.java:690)
> at
> org.apache.flink.table.client.gateway.local.ExecutionContext.wrapClassLoader(ExecutionContext.java:240)
>
[jira] [Closed] (FLINK-15316) SQL Client end-to-end test (Old planner) failed on Travis
[
https://issues.apache.org/jira/browse/FLINK-15316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao closed FLINK-15316.
Resolution: Duplicate
> SQL Client end-to-end test (Old planner) failed on Travis
> -
>
> Key: FLINK-15316
> URL: https://issues.apache.org/jira/browse/FLINK-15316
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client, Tests
>Affects Versions: 1.10.0
>Reporter: Gary Yao
>Priority: Critical
> Labels: test-stability
> Fix For: 1.10.0
>
>
> https://api.travis-ci.org/v3/job/626286501/log.txt
> {noformat}
> Checking for errors...
> Found error in log files:
> ...
> org.apache.flink.table.client.gateway.SqlExecutionException: Invalid SQL
> update statement.
> at
> org.apache.flink.table.client.gateway.local.LocalExecutor.applyUpdate(LocalExecutor.java:697)
> at
> org.apache.flink.table.client.gateway.local.LocalExecutor.executeUpdateInternal(LocalExecutor.java:576)
> at
> org.apache.flink.table.client.gateway.local.LocalExecutor.executeUpdate(LocalExecutor.java:527)
> at
> org.apache.flink.table.client.cli.CliClient.callInsertInto(CliClient.java:535)
> at
> org.apache.flink.table.client.cli.CliClient.lambda$submitUpdate$0(CliClient.java:231)
> at java.util.Optional.map(Optional.java:215)
> at
> org.apache.flink.table.client.cli.CliClient.submitUpdate(CliClient.java:228)
> at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:129)
> at org.apache.flink.table.client.SqlClient.start(SqlClient.java:104)
> at org.apache.flink.table.client.SqlClient.main(SqlClient.java:178)
> Caused by: java.lang.RuntimeException: Error while applying rule
> PushProjectIntoTableSourceScanRule, args
> [rel#88:FlinkLogicalCalc.LOGICAL(input=RelSubset#87,expr#0..2={inputs},expr#3=IS
> NOT NULL($t1),user=$t1,rowtime=$t0,$condition=$t3),
> Scan(table:[default_catalog, default_database, JsonSourceTable],
> fields:(rowtime, user, event), source:KafkaTableSource(rowtime, user, event))]
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:235)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631)
> at
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:327)
> at
> org.apache.flink.table.plan.Optimizer.runVolcanoPlanner(Optimizer.scala:280)
> at
> org.apache.flink.table.plan.Optimizer.optimizeLogicalPlan(Optimizer.scala:199)
> at
> org.apache.flink.table.plan.StreamOptimizer.optimize(StreamOptimizer.scala:66)
> at
> org.apache.flink.table.planner.StreamPlanner.writeToUpsertSink(StreamPlanner.scala:353)
> at
> org.apache.flink.table.planner.StreamPlanner.org$apache$flink$table$planner$StreamPlanner$$writeToSink(StreamPlanner.scala:281)
> at
> org.apache.flink.table.planner.StreamPlanner$$anonfun$2.apply(StreamPlanner.scala:166)
> at
> org.apache.flink.table.planner.StreamPlanner$$anonfun$2.apply(StreamPlanner.scala:145)
> at scala.Option.map(Option.scala:146)
> at
> org.apache.flink.table.planner.StreamPlanner.org$apache$flink$table$planner$StreamPlanner$$translate(StreamPlanner.scala:145)
> at
> org.apache.flink.table.planner.StreamPlanner$$anonfun$translate$1.apply(StreamPlanner.scala:117)
> at
> org.apache.flink.table.planner.StreamPlanner$$anonfun$translate$1.apply(StreamPlanner.scala:117)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at scala.collection.Iterator$class.foreach(Iterator.scala:891)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
> at scala.collection.AbstractTraversable.map(Traversable.scala:104)
> at
> org.apache.flink.table.planner.StreamPlanner.translate(StreamPlanner.scala:117)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.translate(TableEnvironmentImpl.java:680)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlUpdate(TableEnvironmentImpl.java:493)
> at
> org.apache.flink.table.api.java.internal.StreamTableEnvironmentImpl.sqlUpdate(StreamTableEnvironmentImpl.java:331)
> at
> org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$applyUpdate$15(LocalExecutor.java:690)
> at
> org.apache.flink.table.client.gateway.local.ExecutionContext.wrapClassLoader(ExecutionContext.java:240)
> at
>
[jira] [Updated] (FLINK-14007) Add doc for how to using Java user-defined source/sink in Python API
[ https://issues.apache.org/jira/browse/FLINK-14007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Yao updated FLINK-14007: - Fix Version/s: (was: 1.11.0) > Add doc for how to using Java user-defined source/sink in Python API > > > Key: FLINK-14007 > URL: https://issues.apache.org/jira/browse/FLINK-14007 > Project: Flink > Issue Type: Improvement > Components: API / Python, Documentation >Reporter: sunjincheng >Assignee: Hequn Cheng >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Currently, user can not find out the doc for how to using Java user-defined > source/sink in Python API. So we should add the detail doc. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15061) create/alter table/databases properties should be case sensitive stored in catalog
[ https://issues.apache.org/jira/browse/FLINK-15061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Yao updated FLINK-15061: - Fix Version/s: (was: 1.11.0) > create/alter table/databases properties should be case sensitive stored in > catalog > -- > > Key: FLINK-15061 > URL: https://issues.apache.org/jira/browse/FLINK-15061 > Project: Flink > Issue Type: Bug > Components: Table SQL / API >Reporter: Terry Wang >Assignee: Terry Wang >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Now in the class `SqlToOperationConverter`, when creating a table the logic > will convert all properties key to lower format, which will cause the > properties stored in catalog to lose the case style and not intuitively be > observed to user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15318) RocksDBWriteBatchPerformanceTest.benchMark fails on ppc64le
Siddhesh Ghadi created FLINK-15318: -- Summary: RocksDBWriteBatchPerformanceTest.benchMark fails on ppc64le Key: FLINK-15318 URL: https://issues.apache.org/jira/browse/FLINK-15318 Project: Flink Issue Type: Bug Components: Benchmarks, Runtime / State Backends Environment: arch: ppc64le os: rhel7.6, ubuntu 18.04 jdk: 8, 11 mvn: 3.3.9, 3.6.2 Reporter: Siddhesh Ghadi Attachments: surefire-report.txt RocksDBWriteBatchPerformanceTest.benchMark fails due to TestTimedOut, however when test-timeout is increased from 2s to 5s in org/apache/flink/contrib/streaming/state/benchmark/RocksDBWriteBatchPerformanceTest.java:75, it passes. Is this acceptable solution? Note: Tests are ran inside a container. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-14170) Support hadoop < 2.7 with StreamingFileSink.BulkFormatBuilder
[ https://issues.apache.org/jira/browse/FLINK-14170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kostas Kloudas closed FLINK-14170. -- Resolution: Fixed Merged on master with f1c4ddf00a3c40dd451723d80e21adeb8b31fa2e and on release-1.10 with 9c5313e1f634ade92d645dd20ea8b92552a0a3d0 > Support hadoop < 2.7 with StreamingFileSink.BulkFormatBuilder > - > > Key: FLINK-14170 > URL: https://issues.apache.org/jira/browse/FLINK-14170 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Affects Versions: 1.8.0, 1.8.1, 1.8.2, 1.9.0 >Reporter: Bhagavan >Assignee: Bhagavan >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, StreamingFileSink is supported only with Hadoop >= 2.7 > irrespective of Row/bulk format builder. This restriction is due to truncate > is not supported in Hadoop < 2.7 > However, BulkFormatBuilder does not use truncate method to restore the file. > So the restricting StreamingFileSink.BulkFormatBuilder to be used only with > Hadoop >= 2.7 is not necessary. > So requested improvement is to remove the precondition on > HadoopRecoverableWriter and allow BulkFormatBuilder (Parquet) to be used in > Hadoop 2.6 ( Most of the enterprises still on CDH 5.x) > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-14170) Support hadoop < 2.7 with StreamingFileSink.BulkFormatBuilder
[ https://issues.apache.org/jira/browse/FLINK-14170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kostas Kloudas reassigned FLINK-14170: -- Assignee: Bhagavan > Support hadoop < 2.7 with StreamingFileSink.BulkFormatBuilder > - > > Key: FLINK-14170 > URL: https://issues.apache.org/jira/browse/FLINK-14170 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Affects Versions: 1.8.0, 1.8.1, 1.8.2, 1.9.0 >Reporter: Bhagavan >Assignee: Bhagavan >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Currently, StreamingFileSink is supported only with Hadoop >= 2.7 > irrespective of Row/bulk format builder. This restriction is due to truncate > is not supported in Hadoop < 2.7 > However, BulkFormatBuilder does not use truncate method to restore the file. > So the restricting StreamingFileSink.BulkFormatBuilder to be used only with > Hadoop >= 2.7 is not necessary. > So requested improvement is to remove the precondition on > HadoopRecoverableWriter and allow BulkFormatBuilder (Parquet) to be used in > Hadoop 2.6 ( Most of the enterprises still on CDH 5.x) > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15128) Document support for Hive timestamp type
[ https://issues.apache.org/jira/browse/FLINK-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Yao updated FLINK-15128: - Fix Version/s: (was: 1.11.0) > Document support for Hive timestamp type > > > Key: FLINK-15128 > URL: https://issues.apache.org/jira/browse/FLINK-15128 > Project: Flink > Issue Type: Task > Components: Connectors / Hive, Documentation >Reporter: Rui Li >Assignee: Rui Li >Priority: Blocker > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14170) Support hadoop < 2.7 with StreamingFileSink.BulkFormatBuilder
[ https://issues.apache.org/jira/browse/FLINK-14170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kostas Kloudas updated FLINK-14170: --- Fix Version/s: 1.10.0 > Support hadoop < 2.7 with StreamingFileSink.BulkFormatBuilder > - > > Key: FLINK-14170 > URL: https://issues.apache.org/jira/browse/FLINK-14170 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Affects Versions: 1.8.0, 1.8.1, 1.8.2, 1.9.0 >Reporter: Bhagavan >Assignee: Bhagavan >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, StreamingFileSink is supported only with Hadoop >= 2.7 > irrespective of Row/bulk format builder. This restriction is due to truncate > is not supported in Hadoop < 2.7 > However, BulkFormatBuilder does not use truncate method to restore the file. > So the restricting StreamingFileSink.BulkFormatBuilder to be used only with > Hadoop >= 2.7 is not necessary. > So requested improvement is to remove the precondition on > HadoopRecoverableWriter and allow BulkFormatBuilder (Parquet) to be used in > Hadoop 2.6 ( Most of the enterprises still on CDH 5.x) > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15179) Kubernetes should not have a CustomCommandLine.
[
https://issues.apache.org/jira/browse/FLINK-15179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao updated FLINK-15179:
-
Fix Version/s: 1.10.0
> Kubernetes should not have a CustomCommandLine.
> ---
>
> Key: FLINK-15179
> URL: https://issues.apache.org/jira/browse/FLINK-15179
> Project: Flink
> Issue Type: Sub-task
> Components: Deployment / Kubernetes
>Affects Versions: 1.10.0
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> As part of FLIP-73, all command line options are mapped to config options.
> Given this 1-to-1 mapping, the Kubernetes command line could simply forward
> the command line arguments to ConfigOptions directly, instead of introducing
> new command line options. In this case, the user is expected to simply write:
>
> {\{bin/run -e (or --executor) kubernetes-session-cluster -D
> kubernetes.container.image=MY_IMAGE ...}}
> and the CLI will parse the -e to figure out the correct
> {{ClusterClientFactory}} and {{ExecutorFactory}} and then forward to that the
> config options specified with {{-D}}.
> For this, we need to introduce a {{GenericCustomCommandLine}} that simply
> forward the specified parameters to the executors.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15190) add documentation for DDL in FLIP-69
[ https://issues.apache.org/jira/browse/FLINK-15190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Yao updated FLINK-15190: - Fix Version/s: (was: 1.11.0) > add documentation for DDL in FLIP-69 > > > Key: FLINK-15190 > URL: https://issues.apache.org/jira/browse/FLINK-15190 > Project: Flink > Issue Type: Task > Components: Documentation >Reporter: Bowen Li >Assignee: Terry Wang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > > in > https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/sql.html#ddl -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15203) rephrase Hive's data types doc
[ https://issues.apache.org/jira/browse/FLINK-15203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Yao updated FLINK-15203: - Fix Version/s: (was: 1.11.0) > rephrase Hive's data types doc > -- > > Key: FLINK-15203 > URL: https://issues.apache.org/jira/browse/FLINK-15203 > Project: Flink > Issue Type: Task > Components: Documentation >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15228) Drop vendor specific deployment documentation
[ https://issues.apache.org/jira/browse/FLINK-15228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Yao updated FLINK-15228: - Fix Version/s: (was: 1.11.0) > Drop vendor specific deployment documentation > - > > Key: FLINK-15228 > URL: https://issues.apache.org/jira/browse/FLINK-15228 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Based on a mailing list discussion we want to drop vendor specific deployment > documentation > ml discussion: > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Drop-vendor-specific-deployment-documentation-td35457.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
