[jira] [Commented] (FLINK-8769) Quickstart job execution in IDE logs contain several exceptions

2018-03-05 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/FLINK-8769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16385968#comment-16385968
 ] 

ASF GitHub Bot commented on FLINK-8769:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5611


> Quickstart job execution in IDE logs contain several exceptions
> ---
>
> Key: FLINK-8769
> URL: https://issues.apache.org/jira/browse/FLINK-8769
> Project: Flink
>  Issue Type: Bug
>  Components: Local Runtime
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.5.0
>
>
> While checking out [the PR for 
> FLINK-8761|https://github.com/apache/flink/pull/5569] and running a job in 
> the IDE several exceptions are being logged. The job still runs properly 
> though.
> {code:java}
> ...
> 14:19:52,546 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint 
>- Failed to load web based job submission extension.
> org.apache.flink.util.FlinkException: Could not load web submission extension.
>   at 
> org.apache.flink.runtime.webmonitor.WebMonitorUtils.loadWebSubmissionExtension(WebMonitorUtils.java:252)
>   at 
> org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint.initializeHandlers(DispatcherRestEndpoint.java:111)
>   at 
> org.apache.flink.runtime.rest.RestServerEndpoint.start(RestServerEndpoint.java:124)
>   at 
> org.apache.flink.runtime.minicluster.MiniCluster.start(MiniCluster.java:320)
>   at 
> org.apache.flink.client.LocalExecutor.createJobExecutorService(LocalExecutor.java:144)
>   at org.apache.flink.client.LocalExecutor.start(LocalExecutor.java:118)
>   at 
> org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:212)
>   at 
> org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:91)
>   at 
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:815)
>   at org.apache.flink.api.java.DataSet.collect(DataSet.java:413)
>   at org.apache.flink.api.java.DataSet.print(DataSet.java:1652)
>   at iqst.BatchJob.main(BatchJob.java:39)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.flink.runtime.webmonitor.WebMonitorUtils.loadWebSubmissionExtension(WebMonitorUtils.java:243)
>   ... 11 more
> Caused by: org.apache.flink.util.ConfigurationException: Config parameter 
> 'Key: 'jobmanager.rpc.address' , default: null (deprecated keys: [])' is 
> missing (hostname/address of JobManager to connect to).
>   at 
> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.getJobManagerAddress(HighAvailabilityServicesUtils.java:137)
>   at 
> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:79)
>   at 
> org.apache.flink.client.program.ClusterClient.(ClusterClient.java:148)
>   at 
> org.apache.flink.client.program.rest.RestClusterClient.(RestClusterClient.java:144)
>   at 
> org.apache.flink.client.program.rest.RestClusterClient.(RestClusterClient.java:135)
>   at 
> org.apache.flink.runtime.webmonitor.WebSubmissionExtension.(WebSubmissionExtension.java:61)
>   ... 16 more
> 14:19:53,140 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint 
>- Rest endpoint listening at 127.0.0.1:64908
> ...
> 14:19:56,546 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor 
>- Close ResourceManager connection b8a2cff59ba07813067a64ebaf7d7889.
> org.apache.flink.util.FlinkException: New ResourceManager leader found under: 
> null(null)
>   at 
> org.apache.flink.runtime.taskexecutor.TaskExecutor.notifyOfNewResourceManagerLeader(TaskExecutor.java:853)
>   at 
> org.apache.flink.runtime.taskexecutor.TaskExecutor.access$900(TaskExecutor.java:127)
>   at 
> org.apache.flink.runtime.taskexecutor.TaskExecutor$ResourceManagerLeaderListener.lambda$notifyLeaderAddress$0(TaskExecutor.java:1359)
>   at 
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:292)
>   at 
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:147)
>   at 
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$onReceive$0(AkkaRpcActor.java:129)
>   at 
> akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.sca

[jira] [Commented] (FLINK-8769) Quickstart job execution in IDE logs contain several exceptions

2018-03-02 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/FLINK-8769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383717#comment-16383717
 ] 

ASF GitHub Bot commented on FLINK-8769:
---

Github user GJL commented on a diff in the pull request:

https://github.com/apache/flink/pull/5611#discussion_r171883413
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/DispatcherRestEndpoint.java
 ---
@@ -120,7 +120,12 @@ public DispatcherRestEndpoint(
// register extension handlers

handlers.addAll(webSubmissionExtension.getHandlers());
} catch (FlinkException e) {
-   log.info("Failed to load web based job 
submission extension.", e);
+   if (log.isDebugEnabled()) {
+   log.debug("Failed to load web based job 
submission extension.", e);
+   } else {
+   log.info("Failed to load web based job 
submission extension. " +
--- End diff --

It still sounds like an error. Maybe `Web based job submission is disabled. 
Probable reason: [...]`


> Quickstart job execution in IDE logs contain several exceptions
> ---
>
> Key: FLINK-8769
> URL: https://issues.apache.org/jira/browse/FLINK-8769
> Project: Flink
>  Issue Type: Bug
>  Components: Local Runtime
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.5.0
>
>
> While checking out [the PR for 
> FLINK-8761|https://github.com/apache/flink/pull/5569] and running a job in 
> the IDE several exceptions are being logged. The job still runs properly 
> though.
> {code:java}
> ...
> 14:19:52,546 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint 
>- Failed to load web based job submission extension.
> org.apache.flink.util.FlinkException: Could not load web submission extension.
>   at 
> org.apache.flink.runtime.webmonitor.WebMonitorUtils.loadWebSubmissionExtension(WebMonitorUtils.java:252)
>   at 
> org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint.initializeHandlers(DispatcherRestEndpoint.java:111)
>   at 
> org.apache.flink.runtime.rest.RestServerEndpoint.start(RestServerEndpoint.java:124)
>   at 
> org.apache.flink.runtime.minicluster.MiniCluster.start(MiniCluster.java:320)
>   at 
> org.apache.flink.client.LocalExecutor.createJobExecutorService(LocalExecutor.java:144)
>   at org.apache.flink.client.LocalExecutor.start(LocalExecutor.java:118)
>   at 
> org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:212)
>   at 
> org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:91)
>   at 
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:815)
>   at org.apache.flink.api.java.DataSet.collect(DataSet.java:413)
>   at org.apache.flink.api.java.DataSet.print(DataSet.java:1652)
>   at iqst.BatchJob.main(BatchJob.java:39)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.flink.runtime.webmonitor.WebMonitorUtils.loadWebSubmissionExtension(WebMonitorUtils.java:243)
>   ... 11 more
> Caused by: org.apache.flink.util.ConfigurationException: Config parameter 
> 'Key: 'jobmanager.rpc.address' , default: null (deprecated keys: [])' is 
> missing (hostname/address of JobManager to connect to).
>   at 
> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.getJobManagerAddress(HighAvailabilityServicesUtils.java:137)
>   at 
> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:79)
>   at 
> org.apache.flink.client.program.ClusterClient.(ClusterClient.java:148)
>   at 
> org.apache.flink.client.program.rest.RestClusterClient.(RestClusterClient.java:144)
>   at 
> org.apache.flink.client.program.rest.RestClusterClient.(RestClusterClient.java:135)
>   at 
> org.apache.flink.runtime.webmonitor.WebSubmissionExtension.(WebSubmissionExtension.java:61)
>   ... 16 more
> 14:19:53,140 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint 
>- Rest endpoint listening at 127.0.0.1:64908
> ...
> 14:19:56,546 INFO  org.apache.flink.runt

[jira] [Commented] (FLINK-8769) Quickstart job execution in IDE logs contain several exceptions

2018-03-01 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/FLINK-8769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381957#comment-16381957
 ] 

ASF GitHub Bot commented on FLINK-8769:
---

GitHub user NicoK opened a pull request:

https://github.com/apache/flink/pull/5611

[FLINK-8769][flip6] do not print error causing exceptions without debugging

## What is the purpose of the change

Quickstart job execution in IDE logs these two exceptions which may mislead 
the user in thinking something went wrong when indeed it did not:

1)
```
10:23:18,413 INFO  
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint- Failed to load 
web based job submission extension.
org.apache.flink.util.FlinkException: The module flink-runtime-web could 
not be found in the class path. Please add this jar in order to enable web 
based job submission.
at 
org.apache.flink.runtime.webmonitor.WebMonitorUtils.loadWebSubmissionExtension(WebMonitorUtils.java:255)
at 
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint.initializeHandlers(DispatcherRestEndpoint.java:111)
at 
org.apache.flink.runtime.rest.RestServerEndpoint.start(RestServerEndpoint.java:127)
at 
org.apache.flink.runtime.minicluster.MiniCluster.start(MiniCluster.java:330)
at 
org.apache.flink.streaming.api.environment.Flip6LocalStreamEnvironment.execute(Flip6LocalStreamEnvironment.java:101)
at flink.test.StreamingJob.main(StreamingJob.java:64)
```
2)
```
10:23:19,099 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor   
 - Close ResourceManager connection 3ebc4874f7a952d9ba4e4fc6ac574b89.
org.apache.flink.util.FlinkException: New ResourceManager leader found 
under: null(null)
at 
org.apache.flink.runtime.taskexecutor.TaskExecutor.notifyOfNewResourceManagerLeader(TaskExecutor.java:851)
at 
org.apache.flink.runtime.taskexecutor.TaskExecutor.access$900(TaskExecutor.java:127)
at 
org.apache.flink.runtime.taskexecutor.TaskExecutor$ResourceManagerLeaderListener.lambda$notifyLeaderAddress$0(TaskExecutor.java:1359)
at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:295)
at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:150)
at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$onReceive$1(AkkaRpcActor.java:132)
at 
akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:544)
at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at 
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at 
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
```

While for debugging they may be useful, for normal operation it is enough 
to print the messages
themselves, especially since some more details were already logged before 
them.

## Brief change log

- only print error causing exceptions in `DispatcherRestEndpoint` and 
`TaskExecutor` when in debug mode similar to how queryable state loading is done

## Verifying this change

This change can be verified as follows:
- started a fresh quickstart project with a simple streaming program like 
`env.fromElements(1,2,3,4,5).print();` in IntelliJ and verified there are no 
exceptions in the logs

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): **no**
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`:  **no**
  - The serializers:  **no**
  - The runtime per-record code paths (performance sensitive):  **no**
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: **yes**
  - The S3 file system connector:  **no**

## Documentation

  - Does this pull request introduce a new feature?  **no**
  - If yes, how is the feature documented? **not applicable**


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/NicoK/flink flink-8769

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5611.patch

To close this pul

[jira] [Commented] (FLINK-8769) Quickstart job execution in IDE logs contain several exceptions

2018-03-01 Thread Nico Kruber (JIRA)

[ 
https://issues.apache.org/jira/browse/FLINK-8769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381945#comment-16381945
 ] 

Nico Kruber commented on FLINK-8769:


Trying to reproduce this, I only found these two:

{code}
10:23:18,413 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint   
 - Failed to load web based job submission extension.
org.apache.flink.util.FlinkException: The module flink-runtime-web could not be 
found in the class path. Please add this jar in order to enable web based job 
submission.
at 
org.apache.flink.runtime.webmonitor.WebMonitorUtils.loadWebSubmissionExtension(WebMonitorUtils.java:255)
at 
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint.initializeHandlers(DispatcherRestEndpoint.java:111)
at 
org.apache.flink.runtime.rest.RestServerEndpoint.start(RestServerEndpoint.java:127)
at 
org.apache.flink.runtime.minicluster.MiniCluster.start(MiniCluster.java:330)
at 
org.apache.flink.streaming.api.environment.Flip6LocalStreamEnvironment.execute(Flip6LocalStreamEnvironment.java:101)
at flink.test.StreamingJob.main(StreamingJob.java:64)
{code}
This is with a default setup not including the {{flink-runtime-web}} module in 
the classpath.

{code}
10:23:19,099 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor   
 - Close ResourceManager connection 3ebc4874f7a952d9ba4e4fc6ac574b89.
org.apache.flink.util.FlinkException: New ResourceManager leader found under: 
null(null)
at 
org.apache.flink.runtime.taskexecutor.TaskExecutor.notifyOfNewResourceManagerLeader(TaskExecutor.java:851)
at 
org.apache.flink.runtime.taskexecutor.TaskExecutor.access$900(TaskExecutor.java:127)
at 
org.apache.flink.runtime.taskexecutor.TaskExecutor$ResourceManagerLeaderListener.lambda$notifyLeaderAddress$0(TaskExecutor.java:1359)
at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:295)
at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:150)
at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$onReceive$1(AkkaRpcActor.java:132)
at 
akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:544)
at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at 
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at 
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
{code}

Let's not print these messages' error cause in non-debug mode similar to how 
queryable state loading is done.

> Quickstart job execution in IDE logs contain several exceptions
> ---
>
> Key: FLINK-8769
> URL: https://issues.apache.org/jira/browse/FLINK-8769
> Project: Flink
>  Issue Type: Bug
>  Components: Local Runtime
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.5.0
>
>
> While checking out [the PR for 
> FLINK-8761|https://github.com/apache/flink/pull/5569] and running a job in 
> the IDE several exceptions are being logged. The job still runs properly 
> though.
> {code:java}
> ...
> 14:19:52,546 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint 
>- Failed to load web based job submission extension.
> org.apache.flink.util.FlinkException: Could not load web submission extension.
>   at 
> org.apache.flink.runtime.webmonitor.WebMonitorUtils.loadWebSubmissionExtension(WebMonitorUtils.java:252)
>   at 
> org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint.initializeHandlers(DispatcherRestEndpoint.java:111)
>   at 
> org.apache.flink.runtime.rest.RestServerEndpoint.start(RestServerEndpoint.java:124)
>   at 
> org.apache.flink.runtime.minicluster.MiniCluster.start(MiniCluster.java:320)
>   at 
> org.apache.flink.client.LocalExecutor.createJobExecutorService(LocalExecutor.java:144)
>   at org.apache.flink.client.LocalExecutor.start(LocalExecutor.java:118)
>   at 
> org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:212)
>   at 
> org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:91)
>   at 
> org.apache.flin