[jira] [Commented] (FLINK-6387) Flink UI support access log

2018-10-16 Thread ASF GitHub Bot (JIRA) 


[ 
https://issues.apache.org/jira/browse/FLINK-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652343#comment-16652343
 ] 

ASF GitHub Bot commented on FLINK-6387:
---

zentol commented on issue #3777: [FLINK-6387] [webfrontend]Flink UI support 
access log
URL: https://github.com/apache/flink/pull/3777#issuecomment-430374982
 
 
   Subsumed in FLINK-7040.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Flink UI support access log
> ---
>
> Key: FLINK-6387
> URL: https://issues.apache.org/jira/browse/FLINK-6387
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Reporter: shijinkui
>Assignee: shijinkui
>Priority: Major
>
> Record the use request to the access log. Append use access to the log file.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-6387) Flink UI support access log

2018-10-16 Thread ASF GitHub Bot (JIRA) 


[ 
https://issues.apache.org/jira/browse/FLINK-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652344#comment-16652344
 ] 

ASF GitHub Bot commented on FLINK-6387:
---

zentol closed pull request #3777: [FLINK-6387] [webfrontend]Flink UI support 
access log
URL: https://github.com/apache/flink/pull/3777
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/flink-core/src/main/java/org/apache/flink/configuration/JobManagerOptions.java
 
b/flink-core/src/main/java/org/apache/flink/configuration/JobManagerOptions.java
index d129405e06a..153423bbbd1 100644
--- 
a/flink-core/src/main/java/org/apache/flink/configuration/JobManagerOptions.java
+++ 
b/flink-core/src/main/java/org/apache/flink/configuration/JobManagerOptions.java
@@ -168,6 +168,10 @@
key("jobmanager.archive.fs.dir")
.noDefaultValue();
 
+   /** Config parameter indicating whether enable the web access log. */
+   public static final ConfigOption 
JOB_MANAGER_WEB_ACCESSLOG_ENABLE =
+   key("jobmanager.web.accesslog.enable")
+   .defaultValue(false);
// 
-
 
private JobManagerOptions() {
diff --git 
a/flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/HttpRequestHandler.java
 
b/flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/HttpRequestHandler.java
index d14b7a22b1d..ddb6da80ab7 100644
--- 
a/flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/HttpRequestHandler.java
+++ 
b/flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/HttpRequestHandler.java
@@ -33,6 +33,7 @@
 import io.netty.handler.codec.http.DefaultFullHttpResponse;
 import io.netty.handler.codec.http.HttpContent;
 import io.netty.handler.codec.http.HttpHeaders;
+import io.netty.handler.codec.http.HttpHeaders.Names;
 import io.netty.handler.codec.http.HttpMethod;
 import io.netty.handler.codec.http.HttpObject;
 import io.netty.handler.codec.http.HttpRequest;
@@ -50,10 +51,13 @@
 import io.netty.handler.codec.http.multipart.InterfaceHttpData.HttpDataType;
 import org.apache.flink.configuration.ConfigConstants;
 import org.apache.flink.util.ExceptionUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 import java.io.File;
 import java.io.IOException;
 import java.nio.charset.Charset;
+import java.util.Date;
 import java.util.UUID;
 
 /**
@@ -66,19 +70,22 @@
 public class HttpRequestHandler extends 
SimpleChannelInboundHandler {
 
private static final Charset ENCODING = ConfigConstants.DEFAULT_CHARSET;
+   private static final Logger LOG = 
LoggerFactory.getLogger(HttpRequestHandler.class);
 
/** A decoder factory that always stores POST chunks on disk */
private static final HttpDataFactory DATA_FACTORY = new 
DefaultHttpDataFactory(true);
 
private final File tmpDir;
+   private final boolean enableAccesslog;
 
private HttpRequest currentRequest;
 
private HttpPostRequestDecoder currentDecoder;
private String currentRequestPath;
 
-   public HttpRequestHandler(File tmpDir) {
+   public HttpRequestHandler(File tmpDir, boolean enableAccesslog) {
this.tmpDir = tmpDir;
+   this.enableAccesslog = enableAccesslog;
}
 
@Override
@@ -100,6 +107,10 @@ public void channelRead0(ChannelHandlerContext ctx, 
HttpObject msg) {
currentDecoder = null;
}
 
+   if (enableAccesslog) {
+   logAccess(ctx, currentRequest);
+   }
+
if (currentRequest.getMethod() == 
HttpMethod.GET || currentRequest.getMethod() == HttpMethod.DELETE) {
// directly delegate to the router
ctx.fireChannelRead(currentRequest);
@@ -183,4 +194,28 @@ else if (currentDecoder != null && msg instanceof 
HttpContent) {
}
}
}
+
+  /**
+   * Record the access log if enable configure of
+   * {@link 
org.apache.flink.configuration.JobManagerOptions#JOB_MANAGER_WEB_ACCESSLOG_ENABLE}.
+   * record format:
+   * remote_addr - [time_local] "request_method URI protocolVersion" 
"http_referer" "http_user_agent"
+   */
+   private void logAccess(ChannelHandlerContext ctx, HttpRequest req) {
+   HttpHeaders headers = req.headers();
+   if (headers != null) {
+   LOG.info("%s - [%s] \"%s %s %s\" \"%s\"

[jira] [Commented] (FLINK-6387) Flink UI support access log

2017-04-28 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989433#comment-15989433
 ] 

ASF GitHub Bot commented on FLINK-6387:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/3777#discussion_r114015255
  
--- Diff: 
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/HttpRequestHandler.java
 ---
@@ -197,17 +197,16 @@ else if (currentDecoder != null && msg instanceof 
HttpContent) {
 
   /**
* Record the access log if enable configure of
-   * {@link 
org.apache.flink.configuration.ConfigConstants#JOB_MANAGER_WEB_ACCESSLOG_ENABLE}.
+   * {@link 
org.apache.flink.configuration.JobManagerOptions#JOB_MANAGER_WEB_ACCESSLOG_ENABLE}.
* record format:
* remote_addr - [time_local] "request_method URI protocolVersion" 
"http_referer" "http_user_agent"
*/
-   private void accesslog(ChannelHandlerContext ctx, HttpRequest req) {
+   private void logAccess(ChannelHandlerContext ctx, HttpRequest req) {
HttpHeaders headers = req.headers();
if (headers != null) {
-   String line = ctx.channel().remoteAddress() + " - [" + 
new Date() + "] \""
+   LOG.info(ctx.channel().remoteAddress() + " - [" + new 
Date() + "] \""
--- End diff --

Please rework this to use placeholders instead.


> Flink UI support access log
> ---
>
> Key: FLINK-6387
> URL: https://issues.apache.org/jira/browse/FLINK-6387
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Reporter: shijinkui
>Assignee: shijinkui
>
> Record the use request to the access log. Append use access to the log file.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6387) Flink UI support access log

2017-04-27 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15986140#comment-15986140
 ] 

ASF GitHub Bot commented on FLINK-6387:
---

Github user shijinkui commented on the issue:

https://github.com/apache/flink/pull/3777
  
@StephanEwen @zentol   The Flink webmonitor is netty inside. In generally, 
Netty is used as a tcp transfer. For a web or API server, jetty or 
Playframework maybe more suitable. Especially we must fill much gaps and holes 
on web security if choose netty as web framework.

I think we should re-consider whether Netty is suitable?  Current Flink UI 
is hard to extend.


> Flink UI support access log
> ---
>
> Key: FLINK-6387
> URL: https://issues.apache.org/jira/browse/FLINK-6387
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Reporter: shijinkui
>Assignee: shijinkui
>
> Record the use request to the access log. Append use access to the log file.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6387) Flink UI support access log

2017-04-26 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985992#comment-15985992
 ] 

ASF GitHub Bot commented on FLINK-6387:
---

Github user shijinkui commented on the issue:

https://github.com/apache/flink/pull/3777
  
> I can see this flooding the logs like crazy, especially with things like 
metrics and watermarks that update often. Some kind of filtering is probably 
necessary here.

You are right. The crazy log can separated into a new log file.

> This seems to be about auditing, so completely different...

The main purpose is record every access, like auditing log.


> Flink UI support access log
> ---
>
> Key: FLINK-6387
> URL: https://issues.apache.org/jira/browse/FLINK-6387
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Reporter: shijinkui
>Assignee: shijinkui
>
> Record the use request to the access log. Append use access to the log file.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6387) Flink UI support access log

2017-04-26 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984522#comment-15984522
 ] 

ASF GitHub Bot commented on FLINK-6387:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/3777
  
@zentol @shijinkui Ah, forget what I said, I only read the description and 
thought it was something different ;-)

I thought this was about forbidding UI users to access the logs (which 
seems valid since the logs may contain sensitive information).

This seems to be about auditing, so completely different...


> Flink UI support access log
> ---
>
> Key: FLINK-6387
> URL: https://issues.apache.org/jira/browse/FLINK-6387
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Reporter: shijinkui
>Assignee: shijinkui
>
> Record the use request to the access log. Append use access to the log file.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6387) Flink UI support access log

2017-04-26 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984463#comment-15984463
 ] 

ASF GitHub Bot commented on FLINK-6387:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/3777#discussion_r113406982
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -645,6 +645,10 @@
@Deprecated
public static final String JOB_MANAGER_WEB_LOG_PATH_KEY = 
"jobmanager.web.log.path";
 
+   /** Config parameter indicating whether jobs can be uploaded and run 
from the web-frontend. */
+   public static final ConfigOption 
JOB_MANAGER_WEB_ACCESSLOG_ENABLE =
+   
key("jobmanager.web.accesslog.enable").defaultValue(false);
--- End diff --

please move the `.defaultValue` call into a new line. It should also be 
moved to the JobManagerOptions class.


> Flink UI support access log
> ---
>
> Key: FLINK-6387
> URL: https://issues.apache.org/jira/browse/FLINK-6387
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Reporter: shijinkui
>Assignee: shijinkui
>
> Record the use request to the access log. Append use access to the log file.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6387) Flink UI support access log

2017-04-26 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984466#comment-15984466
 ] 

ASF GitHub Bot commented on FLINK-6387:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/3777#discussion_r113406843
  
--- Diff: 
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/HttpRequestHandler.java
 ---
@@ -183,4 +194,28 @@ else if (currentDecoder != null && msg instanceof 
HttpContent) {
}
}
}
+
+  /**
+   * Record the access log if enable configure of
+   * {@link 
org.apache.flink.configuration.ConfigConstants#JOB_MANAGER_WEB_ACCESSLOG_ENABLE}.
+   * record format:
+   * remote_addr - [time_local] "request_method URI protocolVersion" 
"http_referer" "http_user_agent"
+   */
+   private void accesslog(ChannelHandlerContext ctx, HttpRequest req) {
+   HttpHeaders headers = req.headers();
+   if (headers != null) {
+   String line = ctx.channel().remoteAddress() + " - [" + 
new Date() + "] \""
--- End diff --

Please merge this string generation into the log statement.


> Flink UI support access log
> ---
>
> Key: FLINK-6387
> URL: https://issues.apache.org/jira/browse/FLINK-6387
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Reporter: shijinkui
>Assignee: shijinkui
>
> Record the use request to the access log. Append use access to the log file.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6387) Flink UI support access log

2017-04-26 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984464#comment-15984464
 ] 

ASF GitHub Bot commented on FLINK-6387:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/3777#discussion_r113406502
  
--- Diff: 
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/HttpRequestHandler.java
 ---
@@ -183,4 +194,28 @@ else if (currentDecoder != null && msg instanceof 
HttpContent) {
}
}
}
+
+  /**
+   * Record the access log if enable configure of
+   * {@link 
org.apache.flink.configuration.ConfigConstants#JOB_MANAGER_WEB_ACCESSLOG_ENABLE}.
+   * record format:
+   * remote_addr - [time_local] "request_method URI protocolVersion" 
"http_referer" "http_user_agent"
+   */
+   private void accesslog(ChannelHandlerContext ctx, HttpRequest req) {
--- End diff --

this should be renamed to `logAccess`.


> Flink UI support access log
> ---
>
> Key: FLINK-6387
> URL: https://issues.apache.org/jira/browse/FLINK-6387
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Reporter: shijinkui
>Assignee: shijinkui
>
> Record the use request to the access log. Append use access to the log file.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6387) Flink UI support access log

2017-04-26 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984465#comment-15984465
 ] 

ASF GitHub Bot commented on FLINK-6387:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/3777#discussion_r113406464
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -645,6 +645,10 @@
@Deprecated
public static final String JOB_MANAGER_WEB_LOG_PATH_KEY = 
"jobmanager.web.log.path";
 
+   /** Config parameter indicating whether jobs can be uploaded and run 
from the web-frontend. */
--- End diff --

This doc isn't in sync with what the implementation actually does. It 
doesn't restrict anything but only logs all requests.


> Flink UI support access log
> ---
>
> Key: FLINK-6387
> URL: https://issues.apache.org/jira/browse/FLINK-6387
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Reporter: shijinkui
>Assignee: shijinkui
>
> Record the use request to the access log. Append use access to the log file.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6387) Flink UI support access log

2017-04-26 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984391#comment-15984391
 ] 

ASF GitHub Bot commented on FLINK-6387:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/3777
  
I think this is a good issue.
I am wondering if we may want to approach this a bit broader even and 
define certain "access levels" to the UI:

  - view jobs only
  - view jobs and logs
  - view jobs, logs, cancel jobs
  - view jobs, logs, cancel jobs, submit new jobs

What do you think?


> Flink UI support access log
> ---
>
> Key: FLINK-6387
> URL: https://issues.apache.org/jira/browse/FLINK-6387
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Reporter: shijinkui
>Assignee: shijinkui
>
> Record the use request to the access log. Append use access to the log file.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6387) Flink UI support access log

2017-04-26 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984332#comment-15984332
 ] 

ASF GitHub Bot commented on FLINK-6387:
---

GitHub user shijinkui opened a pull request:

https://github.com/apache/flink/pull/3777

[FLINK-6387] [webfrontend]Flink UI support access log

Record the use request to the access log. Append use access to the log file.

- [X] General
  - The pull request references the related JIRA issue ("[FLINK-6387] 
[webfrontend]Flink UI support access log")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [X] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [X] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hwstreaming/flink access_log_support

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3777.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3777


commit 0d19fb95072c90125152513c9b2a07b518d16b27
Author: shijinkui 
Date:   2017-02-23T12:06:43Z

[FLINK-6387] [webfrontend]Flink UI support access log




> Flink UI support access log
> ---
>
> Key: FLINK-6387
> URL: https://issues.apache.org/jira/browse/FLINK-6387
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Reporter: shijinkui
>Assignee: shijinkui
>
> Record the use request to the access log. Append use access to the log file.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)