[jira] [Created] (FLINK-9189) Add a SBT and Gradle Quickstarts
Stephan Ewen created FLINK-9189: --- Summary: Add a SBT and Gradle Quickstarts Key: FLINK-9189 URL: https://issues.apache.org/jira/browse/FLINK-9189 Project: Flink Issue Type: Improvement Components: Quickstarts Reporter: Stephan Ewen Having a proper project template helps a lot in getting dependencies right. For example, setting the core dependencies to "provided", the connector / library dependencies to "compile", etc. The Maven quickstarts are in good shape by now, but I observed SBT and Gradle users to get this wrong quite often. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-9190) YarnResourceManager sometimes does not request new Containers
Gary Yao created FLINK-9190: --- Summary: YarnResourceManager sometimes does not request new Containers Key: FLINK-9190 URL: https://issues.apache.org/jira/browse/FLINK-9190 Project: Flink Issue Type: Bug Components: Distributed Coordination, YARN Affects Versions: 1.5.0 Environment: Hadoop 2.8.3 ZooKeeper 3.4.5 Flink 71c3cd2781d36e0a03d022a38cc4503d343f7ff8 Reporter: Gary Yao Attachments: yarn-logs *Description* The {{YarnResourceManager}} does not request new containers if {{TaskManagers}} are killed rapidly in succession. After 5 minutes the job is restarted due to {{NoResourceAvailableException}}, and the job runs normally afterwards. I suspect that {{TaskManager}} failures are not registered if the failure occurs before the {{TaskManager}} registers with the master. Logs are attached; I added additional log statements to {{YarnResourceManager.onContainersCompleted}} and YarnResourceManager.onContainersAllocated}}. *Expected Behavior* The {{YarnResourceManager}} should recognize that the container is completed and keep requesting new containers. The job should run as soon as resources are available. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-9191) Misleading URLs for website source
Sebb created FLINK-9191: --- Summary: Misleading URLs for website source Key: FLINK-9191 URL: https://issues.apache.org/jira/browse/FLINK-9191 Project: Flink Issue Type: Bug Components: Project Website Reporter: Sebb The project website has the following URLs: Website repositories ASF writable: https://git-wip-us.apache.org/repos/asf/flink-web.git ASF read-only: git://git.apache.org/flink-web.git GitHub mirror: https://github.com/apache/flink-web.git However these all link to the master branch, which is not maintained. If the default branch cannot be changed to asf-site, then the URLs could be changed, e.g: https://git-wip-us.apache.org/repos/asf?p=flink-web.git;a=shortlog;h=refs/heads/asf-site etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-9192) Undo parameterization of StateMachine Example
Stephan Ewen created FLINK-9192: --- Summary: Undo parameterization of StateMachine Example Key: FLINK-9192 URL: https://issues.apache.org/jira/browse/FLINK-9192 Project: Flink Issue Type: Improvement Reporter: Stephan Ewen The example has been changed to add parametrization and a different sink. I would vote to undo these changes, the make the example less nice and use non-recommended sinks: - For state backend, incremental checkpoints, async checkpoints, etc. having these settings in the example blows up the parameter list of the example and distracts from what the example is about. - If the main reason for this is an end-to-end test, then these settings should go into the test's Flink config. - The {{writeAsText}} is a sink that is not recommended to use, because it is not integrated with checkpoints and has no well defined semantics. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-9193) Deprecate non-well-defined output methods on DataStream
Aljoscha Krettek created FLINK-9193: --- Summary: Deprecate non-well-defined output methods on DataStream Key: FLINK-9193 URL: https://issues.apache.org/jira/browse/FLINK-9193 Project: Flink Issue Type: Improvement Components: DataStream API Reporter: Aljoscha Krettek Assignee: Aljoscha Krettek Fix For: 1.5.0 Some output methods on {{DataStream}} that write text to files are not safe to use in a streaming program as they have no consistency guarantees. They are: - {{writeAsText()}} - {{writeAsCsv()}} - {{writeToSocket()}} - {{writeUsingOutputFormat()}} Along with those we should also deprecate the {{SinkFunctions}} that they use. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [jira] [Created] (FLINK-9148) when deploying flink on kubernetes, the taskmanager report "java.net.UnknownHostException: flink-jobmanager: Temporary failure in name resolution"
You'll have create the job manager service like mentioned in the link. http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Flink-on-Kubernetes-issue-tp21664.html You should create job manager service before deploying the task manager. http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Flink-on-Kubernetes-issue-tp21664.html On Sun, Apr 8, 2018 at 8:28 PM, You Chu (JIRA) wrote: > You Chu created FLINK-9148: > -- > > Summary: when deploying flink on kubernetes, the taskmanager > report "java.net.UnknownHostException: flink-jobmanager: Temporary > failure in name resolution" > Key: FLINK-9148 > URL: https://issues.apache.org/jira/browse/FLINK-9148 > Project: Flink > Issue Type: Bug > Components: Docker > Affects Versions: 1.4.2 > Environment: kubernetes 1.9 > docker 1.4 > see :https://ci.apache.org/projects/flink/flink-docs- > release-1.4/ops/deployment/kubernetes.html > Reporter: You Chu > > > refer tohttps://ci.apache.org/projects/flink/flink-docs- > release-1.4/ops/deployment/kubernetes.html: > > I deploy flink1.4 on kubernetes 1.9, the jobmanager container is ok, but > the taskmanager contains failed with error: > > > java.net.UnknownHostException: flink-jobmanager: Temporary failure in name > resolution > at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) > at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928) > at java.net.InetAddress.getAddressesFromNameService( > InetAddress.java:1323) > at java.net.InetAddress.getAllByName0(InetAddress.java:1276) > at java.net.InetAddress.getAllByName(InetAddress.java:1192) > at java.net.InetAddress.getAllByName(InetAddress.java:1126) > at java.net.InetAddress.getByName(InetAddress.java:1076) > at org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils. > getRpcUrl(AkkaRpcServiceUtils.java:172) > at org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils. > getRpcUrl(AkkaRpcServiceUtils.java:137) > at org.apache.flink.runtime.highavailability. > HighAvailabilityServicesUtils.createHighAvailabilityServices( > HighAvailabilityServicesUtils.java:79) > at org.apache.flink.runtime.taskmanager.TaskManager$. > selectNetworkInterfaceAndRunTaskManager(TaskManager.scala:1681) > at org.apache.flink.runtime.taskmanager.TaskManager$$anon$ > 2.call(TaskManager.scala:1592) > at org.apache.flink.runtime.taskmanager.TaskManager$$anon$ > 2.call(TaskManager.scala:1590) > at java.security.AccessController.doPrivileged(Native Method) > > > > -- > This message was sent by Atlassian JIRA > (v7.6.3#76005) >
[jira] [Created] (FLINK-9194) Finished jobs are not archived to HistoryServer
Gary Yao created FLINK-9194: --- Summary: Finished jobs are not archived to HistoryServer Key: FLINK-9194 URL: https://issues.apache.org/jira/browse/FLINK-9194 Project: Flink Issue Type: Bug Components: History Server, JobManager Affects Versions: 1.5.0 Environment: Flink 2af481a Reporter: Gary Yao In flip6 mode, jobs are not archived to the HistoryServer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-9195) Delete non-well-defined output methods on DataStream
Aljoscha Krettek created FLINK-9195: --- Summary: Delete non-well-defined output methods on DataStream Key: FLINK-9195 URL: https://issues.apache.org/jira/browse/FLINK-9195 Project: Flink Issue Type: Improvement Reporter: Aljoscha Krettek Assignee: Aljoscha Krettek Fix For: 1.6.0 Some output methods on {{DataStream}} that write text to files are not safe to use in a streaming program as they have no consistency guarantees. They are: - {{writeAsText()}} - {{writeAsCsv()}} - {{writeToSocket()}} - {{writeUsingOutputFormat()}} Along with those we should also delete the {{SinkFunctions}} that they use. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-9196) YARN: Flink binaries are not deleted from HDFS after cluster shutdown
Gary Yao created FLINK-9196: --- Summary: YARN: Flink binaries are not deleted from HDFS after cluster shutdown Key: FLINK-9196 URL: https://issues.apache.org/jira/browse/FLINK-9196 Project: Flink Issue Type: Bug Components: YARN Affects Versions: 1.5.0 Reporter: Gary Yao When deploying on YARN in flip6 mode, the Flink binaries are not deleted from HDFS after the cluster shuts down. *Steps to reproduce* # Submit job in YARN job mode, non-detached: {noformat} HADOOP_CLASSPATH=`hadoop classpath` bin/flink run -m yarn-cluster -yjm 2048 -ytm 2048 -yn 2 ./examples/streaming/WordCount.jar {noformat} # Check contents of {{/user/hadoop/.flink/}} on HDFS after job is finished: {noformat} [hadoop@ip-172-31-43-78 flink-1.5.0]$ hdfs dfs -ls /user/hadoop/.flink/application_1523966184826_0016 Found 6 items -rw-r--r-- 1 hadoop hadoop583 2018-04-17 14:54 /user/hadoop/.flink/application_1523966184826_0016/90cf5b3a-039e-4d52-8266-4e9563d74827-taskmanager-conf.yaml -rw-r--r-- 1 hadoop hadoop332 2018-04-17 14:54 /user/hadoop/.flink/application_1523966184826_0016/application_1523966184826_0016-flink-conf.yaml3818971235442577934.tmp -rw-r--r-- 1 hadoop hadoop 89779342 2018-04-02 17:08 /user/hadoop/.flink/application_1523966184826_0016/flink-dist_2.11-1.5.0.jar drwxrwxrwx - hadoop hadoop 0 2018-04-17 14:54 /user/hadoop/.flink/application_1523966184826_0016/lib -rw-r--r-- 1 hadoop hadoop 1939 2018-04-02 15:37 /user/hadoop/.flink/application_1523966184826_0016/log4j.properties -rw-r--r-- 1 hadoop hadoop 2331 2018-04-02 15:37 /user/hadoop/.flink/application_1523966184826_0016/logback.xml {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-9197) Improve error message for TypyInformation and TypeHint with generics
Stephan Ewen created FLINK-9197: --- Summary: Improve error message for TypyInformation and TypeHint with generics Key: FLINK-9197 URL: https://issues.apache.org/jira/browse/FLINK-9197 Project: Flink Issue Type: Improvement Components: Core Affects Versions: 1.4.2 Reporter: Stephan Ewen Assignee: Stephan Ewen Fix For: 1.5.0 User feedback: When using a {{TypeHint}} with a generic type variable, the error message could be better. Similarly, when using {{TypeInformation.of(Tuple2.class)}}, the error message should refer the user to the TypeHint method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
CaseClassSerializer and/or TraversableSerializer may still not be threadsafe?
Hello all, I am running Flink 1.4.0 on Amazon EMR, and find that asynchronous snapshots fail when using the Filesystem back-end. Synchronous snapshots succeed, and RocksDB snapshots succeed (both async and sync), but async Filesystem snapshots fail with this error: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 at java.util.ArrayList.rangeCheck(ArrayList.java:657) at java.util.ArrayList.set(ArrayList.java:448) at com.esotericsoftware.kryo.util.MapReferenceResolver.setReadObject(MapReferenceResolver.java:56) at com.esotericsoftware.kryo.Kryo.reference(Kryo.java:875) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:710) at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.copy(KryoSerializer.java:189) at org.apache.flink.api.scala.typeutils.CaseClassSerializer.copy(CaseClassSerializer.scala:101) at org.apache.flink.api.scala.typeutils.CaseClassSerializer.copy(CaseClassSerializer.scala:32) at org.apache.flink.api.scala.typeutils.TraversableSerializer$$anonfun$copy$1.apply(TraversableSerializer.scala:69) at org.apache.flink.api.scala.typeutils.TraversableSerializer$$anonfun$copy$1.apply(TraversableSerializer.scala:69) at scala.collection.Iterator$class.foreach(Iterator.scala:891) at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at org.apache.flink.api.scala.typeutils.TraversableSerializer.copy(TraversableSerializer.scala:69) at org.apache.flink.api.scala.typeutils.TraversableSerializer.copy(TraversableSerializer.scala:33) at org.apache.flink.runtime.state.heap.CopyOnWriteStateTable.get(CopyOnWriteStateTable.java:282) at org.apache.flink.runtime.state.heap.CopyOnWriteStateTable.get(CopyOnWriteStateTable.java:306) at org.apache.flink.runtime.state.heap.HeapValueState.value(HeapValueState.java:55) at net.districtm.segmentsync.processing.JoinSegmentMappingWithSegmentAssignments.enqueueSegmentAssignment(JoinSegmentMappingWithSegmentAssignments.scala:102) at net.districtm.segmentsync.processing.JoinSegmentMappingWithSegmentAssignments.processElement2(JoinSegmentMappingWithSegmentAssignments.scala:218) at net.districtm.segmentsync.processing.JoinSegmentMappingWithSegmentAssignments.processElement2(JoinSegmentMappingWithSegmentAssignments.scala:76) at org.apache.flink.streaming.api.operators.co.KeyedCoProcessOperator.processElement2(KeyedCoProcessOperator.java:86) at org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.processInput(StreamTwoInputProcessor.java:270) at org.apache.flink.streaming.runtime.tasks.TwoInputStreamTask.run(TwoInputStreamTask.java:91) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:264) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718) at java.lang.Thread.run(Thread.java:748) This stack trace occurs when I am trying to access the value of a `ValueState[scala.collection.mutable.PriorityQueue[(AJavaObjectThatUsesTwitterChillProtoSerialization, Long, scala.collection.mutable.BitSet)]` while a checkpoint is going on. I have found similar errors occurring in already-fixed tickets like this one: https://issues.apache.org/jira/browse/FLINK-7484 which is part of this umbrella issue: https://issues.apache.org/jira/browse/FLINK-7830 However these tickets are apparently resolved, maybe the bug has not been completely fixed? Or maybe I am making a mistake in programming? When I get the value of the state, I do mutate it, and I also mutate the mutable.BitSet before persisting again. But as far as I know this is perfectly ok by flink yes? Thanks for any help or pointers! -- Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
[jira] [Created] (FLINK-9198) Improve error messages in AbstractDeserializationSchema for type extraction
Stephan Ewen created FLINK-9198: --- Summary: Improve error messages in AbstractDeserializationSchema for type extraction Key: FLINK-9198 URL: https://issues.apache.org/jira/browse/FLINK-9198 Project: Flink Issue Type: Improvement Affects Versions: 1.4.2 Reporter: Stephan Ewen Assignee: Stephan Ewen Fix For: 1.5.0 User feedback: When type extraction fails in the {{AbstractDeserializationSchema}}, the error message does not explain fully how to fix this. I suggest to improve the error message and add some convenience constructors to directly pass TypeInformation when needed. We can also simplify the class a bit, because TypeInformation needs no longer be dropped during serialization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[Discuss] FLINK-8533 MasterTriggerRestoreHook state initialization
Hello, We see an issue with the use of `MasterTriggerRestoreHook` to synchronize the state of a source function with that of an external system. I'd like the fix to be considered for 1.5. There's a patch ready: https://github.com/apache/flink/pull/5427 Thanks!
[jira] [Created] (FLINK-9199) SubtaskExecutionAttemptAccumulatorsHeaders & SubtaskExecutionAttemptDetailsHeaders has malfunctioning URL
Rong Rong created FLINK-9199: Summary: SubtaskExecutionAttemptAccumulatorsHeaders & SubtaskExecutionAttemptDetailsHeaders has malfunctioning URL Key: FLINK-9199 URL: https://issues.apache.org/jira/browse/FLINK-9199 Project: Flink Issue Type: Bug Components: REST Reporter: Rong Rong -- This message was sent by Atlassian JIRA (v7.6.3#76005)