[jira] [Created] (FLINK-9189) Add a SBT and Gradle Quickstarts
Stephan Ewen created FLINK-9189: --- Summary: Add a SBT and Gradle Quickstarts Key: FLINK-9189 URL: https://issues.apache.org/jira/browse/FLINK-9189 Project: Flink Issue Type: Improvement Components: Quickstarts Reporter: Stephan Ewen Having a proper project template helps a lot in getting dependencies right. For example, setting the core dependencies to "provided", the connector / library dependencies to "compile", etc. The Maven quickstarts are in good shape by now, but I observed SBT and Gradle users to get this wrong quite often. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-9190) YarnResourceManager sometimes does not request new Containers
Gary Yao created FLINK-9190:
---
Summary: YarnResourceManager sometimes does not request new
Containers
Key: FLINK-9190
URL: https://issues.apache.org/jira/browse/FLINK-9190
Project: Flink
Issue Type: Bug
Components: Distributed Coordination, YARN
Affects Versions: 1.5.0
Environment: Hadoop 2.8.3
ZooKeeper 3.4.5
Flink 71c3cd2781d36e0a03d022a38cc4503d343f7ff8
Reporter: Gary Yao
Attachments: yarn-logs
*Description*
The {{YarnResourceManager}} does not request new containers if {{TaskManagers}}
are killed rapidly in succession. After 5 minutes the job is restarted due to
{{NoResourceAvailableException}}, and the job runs normally afterwards. I
suspect that {{TaskManager}} failures are not registered if the failure occurs
before the {{TaskManager}} registers with the master. Logs are attached; I
added additional log statements to
{{YarnResourceManager.onContainersCompleted}} and
YarnResourceManager.onContainersAllocated}}.
*Expected Behavior*
The {{YarnResourceManager}} should recognize that the container is completed
and keep requesting new containers. The job should run as soon as resources are
available.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-9191) Misleading URLs for website source
Sebb created FLINK-9191: --- Summary: Misleading URLs for website source Key: FLINK-9191 URL: https://issues.apache.org/jira/browse/FLINK-9191 Project: Flink Issue Type: Bug Components: Project Website Reporter: Sebb The project website has the following URLs: Website repositories ASF writable: https://git-wip-us.apache.org/repos/asf/flink-web.git ASF read-only: git://git.apache.org/flink-web.git GitHub mirror: https://github.com/apache/flink-web.git However these all link to the master branch, which is not maintained. If the default branch cannot be changed to asf-site, then the URLs could be changed, e.g: https://git-wip-us.apache.org/repos/asf?p=flink-web.git;a=shortlog;h=refs/heads/asf-site etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-9192) Undo parameterization of StateMachine Example
Stephan Ewen created FLINK-9192:
---
Summary: Undo parameterization of StateMachine Example
Key: FLINK-9192
URL: https://issues.apache.org/jira/browse/FLINK-9192
Project: Flink
Issue Type: Improvement
Reporter: Stephan Ewen
The example has been changed to add parametrization and a different sink.
I would vote to undo these changes, the make the example less nice and use
non-recommended sinks:
- For state backend, incremental checkpoints, async checkpoints, etc. having
these settings in the example blows up the parameter list of the example and
distracts from what the example is about.
- If the main reason for this is an end-to-end test, then these settings
should go into the test's Flink config.
- The {{writeAsText}} is a sink that is not recommended to use, because it is
not integrated with checkpoints and has no well defined semantics.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-9193) Deprecate non-well-defined output methods on DataStream
Aljoscha Krettek created FLINK-9193:
---
Summary: Deprecate non-well-defined output methods on DataStream
Key: FLINK-9193
URL: https://issues.apache.org/jira/browse/FLINK-9193
Project: Flink
Issue Type: Improvement
Components: DataStream API
Reporter: Aljoscha Krettek
Assignee: Aljoscha Krettek
Fix For: 1.5.0
Some output methods on {{DataStream}} that write text to files are not safe to
use in a streaming program as they have no consistency guarantees. They are:
- {{writeAsText()}}
- {{writeAsCsv()}}
- {{writeToSocket()}}
- {{writeUsingOutputFormat()}}
Along with those we should also deprecate the {{SinkFunctions}} that they use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: [jira] [Created] (FLINK-9148) when deploying flink on kubernetes, the taskmanager report "java.net.UnknownHostException: flink-jobmanager: Temporary failure in name resolution"
You'll have create the job manager service like mentioned in the link. http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Flink-on-Kubernetes-issue-tp21664.html You should create job manager service before deploying the task manager. http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Flink-on-Kubernetes-issue-tp21664.html On Sun, Apr 8, 2018 at 8:28 PM, You Chu (JIRA) wrote: > You Chu created FLINK-9148: > -- > > Summary: when deploying flink on kubernetes, the taskmanager > report "java.net.UnknownHostException: flink-jobmanager: Temporary > failure in name resolution" > Key: FLINK-9148 > URL: https://issues.apache.org/jira/browse/FLINK-9148 > Project: Flink > Issue Type: Bug > Components: Docker > Affects Versions: 1.4.2 > Environment: kubernetes 1.9 > docker 1.4 > see :https://ci.apache.org/projects/flink/flink-docs- > release-1.4/ops/deployment/kubernetes.html > Reporter: You Chu > > > refer tohttps://ci.apache.org/projects/flink/flink-docs- > release-1.4/ops/deployment/kubernetes.html: > > I deploy flink1.4 on kubernetes 1.9, the jobmanager container is ok, but > the taskmanager contains failed with error: > > > java.net.UnknownHostException: flink-jobmanager: Temporary failure in name > resolution > at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) > at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928) > at java.net.InetAddress.getAddressesFromNameService( > InetAddress.java:1323) > at java.net.InetAddress.getAllByName0(InetAddress.java:1276) > at java.net.InetAddress.getAllByName(InetAddress.java:1192) > at java.net.InetAddress.getAllByName(InetAddress.java:1126) > at java.net.InetAddress.getByName(InetAddress.java:1076) > at org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils. > getRpcUrl(AkkaRpcServiceUtils.java:172) > at org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils. > getRpcUrl(AkkaRpcServiceUtils.java:137) > at org.apache.flink.runtime.highavailability. > HighAvailabilityServicesUtils.createHighAvailabilityServices( > HighAvailabilityServicesUtils.java:79) > at org.apache.flink.runtime.taskmanager.TaskManager$. > selectNetworkInterfaceAndRunTaskManager(TaskManager.scala:1681) > at org.apache.flink.runtime.taskmanager.TaskManager$$anon$ > 2.call(TaskManager.scala:1592) > at org.apache.flink.runtime.taskmanager.TaskManager$$anon$ > 2.call(TaskManager.scala:1590) > at java.security.AccessController.doPrivileged(Native Method) > > > > -- > This message was sent by Atlassian JIRA > (v7.6.3#76005) >
[jira] [Created] (FLINK-9194) Finished jobs are not archived to HistoryServer
Gary Yao created FLINK-9194: --- Summary: Finished jobs are not archived to HistoryServer Key: FLINK-9194 URL: https://issues.apache.org/jira/browse/FLINK-9194 Project: Flink Issue Type: Bug Components: History Server, JobManager Affects Versions: 1.5.0 Environment: Flink 2af481a Reporter: Gary Yao In flip6 mode, jobs are not archived to the HistoryServer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-9195) Delete non-well-defined output methods on DataStream
Aljoscha Krettek created FLINK-9195:
---
Summary: Delete non-well-defined output methods on DataStream
Key: FLINK-9195
URL: https://issues.apache.org/jira/browse/FLINK-9195
Project: Flink
Issue Type: Improvement
Reporter: Aljoscha Krettek
Assignee: Aljoscha Krettek
Fix For: 1.6.0
Some output methods on {{DataStream}} that write text to files are not safe to
use in a streaming program as they have no consistency guarantees. They are:
- {{writeAsText()}}
- {{writeAsCsv()}}
- {{writeToSocket()}}
- {{writeUsingOutputFormat()}}
Along with those we should also delete the {{SinkFunctions}} that they use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-9196) YARN: Flink binaries are not deleted from HDFS after cluster shutdown
Gary Yao created FLINK-9196:
---
Summary: YARN: Flink binaries are not deleted from HDFS after
cluster shutdown
Key: FLINK-9196
URL: https://issues.apache.org/jira/browse/FLINK-9196
Project: Flink
Issue Type: Bug
Components: YARN
Affects Versions: 1.5.0
Reporter: Gary Yao
When deploying on YARN in flip6 mode, the Flink binaries are not deleted from
HDFS after the cluster shuts down.
*Steps to reproduce*
# Submit job in YARN job mode, non-detached:
{noformat} HADOOP_CLASSPATH=`hadoop classpath` bin/flink run -m yarn-cluster
-yjm 2048 -ytm 2048 -yn 2 ./examples/streaming/WordCount.jar {noformat}
# Check contents of {{/user/hadoop/.flink/}} on HDFS after job
is finished:
{noformat}
[hadoop@ip-172-31-43-78 flink-1.5.0]$ hdfs dfs -ls
/user/hadoop/.flink/application_1523966184826_0016
Found 6 items
-rw-r--r-- 1 hadoop hadoop583 2018-04-17 14:54
/user/hadoop/.flink/application_1523966184826_0016/90cf5b3a-039e-4d52-8266-4e9563d74827-taskmanager-conf.yaml
-rw-r--r-- 1 hadoop hadoop332 2018-04-17 14:54
/user/hadoop/.flink/application_1523966184826_0016/application_1523966184826_0016-flink-conf.yaml3818971235442577934.tmp
-rw-r--r-- 1 hadoop hadoop 89779342 2018-04-02 17:08
/user/hadoop/.flink/application_1523966184826_0016/flink-dist_2.11-1.5.0.jar
drwxrwxrwx - hadoop hadoop 0 2018-04-17 14:54
/user/hadoop/.flink/application_1523966184826_0016/lib
-rw-r--r-- 1 hadoop hadoop 1939 2018-04-02 15:37
/user/hadoop/.flink/application_1523966184826_0016/log4j.properties
-rw-r--r-- 1 hadoop hadoop 2331 2018-04-02 15:37
/user/hadoop/.flink/application_1523966184826_0016/logback.xml
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-9197) Improve error message for TypyInformation and TypeHint with generics
Stephan Ewen created FLINK-9197:
---
Summary: Improve error message for TypyInformation and TypeHint
with generics
Key: FLINK-9197
URL: https://issues.apache.org/jira/browse/FLINK-9197
Project: Flink
Issue Type: Improvement
Components: Core
Affects Versions: 1.4.2
Reporter: Stephan Ewen
Assignee: Stephan Ewen
Fix For: 1.5.0
User feedback: When using a {{TypeHint}} with a generic type variable, the
error message could be better. Similarly, when using
{{TypeInformation.of(Tuple2.class)}}, the error message should refer the user
to the TypeHint method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
CaseClassSerializer and/or TraversableSerializer may still not be threadsafe?
Hello all, I am running Flink 1.4.0 on Amazon EMR, and find that asynchronous snapshots fail when using the Filesystem back-end. Synchronous snapshots succeed, and RocksDB snapshots succeed (both async and sync), but async Filesystem snapshots fail with this error: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 at java.util.ArrayList.rangeCheck(ArrayList.java:657) at java.util.ArrayList.set(ArrayList.java:448) at com.esotericsoftware.kryo.util.MapReferenceResolver.setReadObject(MapReferenceResolver.java:56) at com.esotericsoftware.kryo.Kryo.reference(Kryo.java:875) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:710) at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.copy(KryoSerializer.java:189) at org.apache.flink.api.scala.typeutils.CaseClassSerializer.copy(CaseClassSerializer.scala:101) at org.apache.flink.api.scala.typeutils.CaseClassSerializer.copy(CaseClassSerializer.scala:32) at org.apache.flink.api.scala.typeutils.TraversableSerializer$$anonfun$copy$1.apply(TraversableSerializer.scala:69) at org.apache.flink.api.scala.typeutils.TraversableSerializer$$anonfun$copy$1.apply(TraversableSerializer.scala:69) at scala.collection.Iterator$class.foreach(Iterator.scala:891) at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at org.apache.flink.api.scala.typeutils.TraversableSerializer.copy(TraversableSerializer.scala:69) at org.apache.flink.api.scala.typeutils.TraversableSerializer.copy(TraversableSerializer.scala:33) at org.apache.flink.runtime.state.heap.CopyOnWriteStateTable.get(CopyOnWriteStateTable.java:282) at org.apache.flink.runtime.state.heap.CopyOnWriteStateTable.get(CopyOnWriteStateTable.java:306) at org.apache.flink.runtime.state.heap.HeapValueState.value(HeapValueState.java:55) at net.districtm.segmentsync.processing.JoinSegmentMappingWithSegmentAssignments.enqueueSegmentAssignment(JoinSegmentMappingWithSegmentAssignments.scala:102) at net.districtm.segmentsync.processing.JoinSegmentMappingWithSegmentAssignments.processElement2(JoinSegmentMappingWithSegmentAssignments.scala:218) at net.districtm.segmentsync.processing.JoinSegmentMappingWithSegmentAssignments.processElement2(JoinSegmentMappingWithSegmentAssignments.scala:76) at org.apache.flink.streaming.api.operators.co.KeyedCoProcessOperator.processElement2(KeyedCoProcessOperator.java:86) at org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.processInput(StreamTwoInputProcessor.java:270) at org.apache.flink.streaming.runtime.tasks.TwoInputStreamTask.run(TwoInputStreamTask.java:91) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:264) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718) at java.lang.Thread.run(Thread.java:748) This stack trace occurs when I am trying to access the value of a `ValueState[scala.collection.mutable.PriorityQueue[(AJavaObjectThatUsesTwitterChillProtoSerialization, Long, scala.collection.mutable.BitSet)]` while a checkpoint is going on. I have found similar errors occurring in already-fixed tickets like this one: https://issues.apache.org/jira/browse/FLINK-7484 which is part of this umbrella issue: https://issues.apache.org/jira/browse/FLINK-7830 However these tickets are apparently resolved, maybe the bug has not been completely fixed? Or maybe I am making a mistake in programming? When I get the value of the state, I do mutate it, and I also mutate the mutable.BitSet before persisting again. But as far as I know this is perfectly ok by flink yes? Thanks for any help or pointers! -- Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
[jira] [Created] (FLINK-9198) Improve error messages in AbstractDeserializationSchema for type extraction
Stephan Ewen created FLINK-9198:
---
Summary: Improve error messages in AbstractDeserializationSchema
for type extraction
Key: FLINK-9198
URL: https://issues.apache.org/jira/browse/FLINK-9198
Project: Flink
Issue Type: Improvement
Affects Versions: 1.4.2
Reporter: Stephan Ewen
Assignee: Stephan Ewen
Fix For: 1.5.0
User feedback: When type extraction fails in the
{{AbstractDeserializationSchema}}, the error message does not explain fully how
to fix this.
I suggest to improve the error message and add some convenience constructors to
directly pass TypeInformation when needed.
We can also simplify the class a bit, because TypeInformation needs no longer
be dropped during serialization.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[Discuss] FLINK-8533 MasterTriggerRestoreHook state initialization
Hello, We see an issue with the use of `MasterTriggerRestoreHook` to synchronize the state of a source function with that of an external system. I'd like the fix to be considered for 1.5. There's a patch ready: https://github.com/apache/flink/pull/5427 Thanks!
[jira] [Created] (FLINK-9199) SubtaskExecutionAttemptAccumulatorsHeaders & SubtaskExecutionAttemptDetailsHeaders has malfunctioning URL
Rong Rong created FLINK-9199: Summary: SubtaskExecutionAttemptAccumulatorsHeaders & SubtaskExecutionAttemptDetailsHeaders has malfunctioning URL Key: FLINK-9199 URL: https://issues.apache.org/jira/browse/FLINK-9199 Project: Flink Issue Type: Bug Components: REST Reporter: Rong Rong -- This message was sent by Atlassian JIRA (v7.6.3#76005)
