[jira] [Work logged] (BEAM-8625) Implement servlet in Dataflow runner for sdk status query endpoint
[ https://issues.apache.org/jira/browse/BEAM-8625?focusedWorklogId=373373=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373373 ] ASF GitHub Bot logged work on BEAM-8625: Author: ASF GitHub Bot Created on: 17/Jan/20 03:30 Start Date: 17/Jan/20 03:30 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #10553: [BEAM-8625] Implement servlet for exposing sdk harness statuses in Da… URL: https://github.com/apache/beam/pull/10553 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373373) Time Spent: 2h 10m (was: 2h) > Implement servlet in Dataflow runner for sdk status query endpoint > -- > > Key: BEAM-8625 > URL: https://issues.apache.org/jira/browse/BEAM-8625 > Project: Beam > Issue Type: Sub-task > Components: runner-dataflow >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8625) Implement servlet in Dataflow runner for sdk status query endpoint
[
https://issues.apache.org/jira/browse/BEAM-8625?focusedWorklogId=370152=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-370152
]
ASF GitHub Bot logged work on BEAM-8625:
Author: ASF GitHub Bot
Created on: 10/Jan/20 23:58
Start Date: 10/Jan/20 23:58
Worklog Time Spent: 10m
Work Description: y1chi commented on pull request #10553: [BEAM-8625]
Implement servlet for exposing sdk harness statuses in Da…
URL: https://github.com/apache/beam/pull/10553#discussion_r365475496
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/WorkerStatusPages.java
##
@@ -108,7 +113,11 @@ public void addStatusDataProvider(
/** Returns the set of pages than should be captured by DebugCapture. */
public List getDebugCapturePages() {
-return Arrays.asList(threadzServlet, statuszServlet);
+return this.capturePages;
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 370152)
Time Spent: 2h (was: 1h 50m)
> Implement servlet in Dataflow runner for sdk status query endpoint
> --
>
> Key: BEAM-8625
> URL: https://issues.apache.org/jira/browse/BEAM-8625
> Project: Beam
> Issue Type: Sub-task
> Components: runner-dataflow
>Reporter: Yichi Zhang
>Assignee: Yichi Zhang
>Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8625) Implement servlet in Dataflow runner for sdk status query endpoint
[
https://issues.apache.org/jira/browse/BEAM-8625?focusedWorklogId=370151=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-370151
]
ASF GitHub Bot logged work on BEAM-8625:
Author: ASF GitHub Bot
Created on: 10/Jan/20 23:56
Start Date: 10/Jan/20 23:56
Worklog Time Spent: 10m
Work Description: y1chi commented on pull request #10553: [BEAM-8625]
Implement servlet for exposing sdk harness statuses in Da…
URL: https://github.com/apache/beam/pull/10553#discussion_r365474951
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java
##
@@ -871,6 +871,10 @@ public void startStatusPages() {
statusPages.start();
}
+ public WorkerStatusPages getStatusPages() {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 370151)
Time Spent: 1h 50m (was: 1h 40m)
> Implement servlet in Dataflow runner for sdk status query endpoint
> --
>
> Key: BEAM-8625
> URL: https://issues.apache.org/jira/browse/BEAM-8625
> Project: Beam
> Issue Type: Sub-task
> Components: runner-dataflow
>Reporter: Yichi Zhang
>Assignee: Yichi Zhang
>Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8625) Implement servlet in Dataflow runner for sdk status query endpoint
[
https://issues.apache.org/jira/browse/BEAM-8625?focusedWorklogId=370145=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-370145
]
ASF GitHub Bot logged work on BEAM-8625:
Author: ASF GitHub Bot
Created on: 10/Jan/20 23:44
Start Date: 10/Jan/20 23:44
Worklog Time Spent: 10m
Work Description: y1chi commented on pull request #10553: [BEAM-8625]
Implement servlet for exposing sdk harness statuses in Da…
URL: https://github.com/apache/beam/pull/10553#discussion_r365472923
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/WorkerStatusPages.java
##
@@ -108,7 +113,11 @@ public void addStatusDataProvider(
/** Returns the set of pages than should be captured by DebugCapture. */
public List getDebugCapturePages() {
-return Arrays.asList(threadzServlet, statuszServlet);
+return this.capturePages;
Review comment:
This is only used by DebugCapture Manager which takes a List as constructor
parameter.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 370145)
Time Spent: 1h 40m (was: 1.5h)
> Implement servlet in Dataflow runner for sdk status query endpoint
> --
>
> Key: BEAM-8625
> URL: https://issues.apache.org/jira/browse/BEAM-8625
> Project: Beam
> Issue Type: Sub-task
> Components: runner-dataflow
>Reporter: Yichi Zhang
>Assignee: Yichi Zhang
>Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8625) Implement servlet in Dataflow runner for sdk status query endpoint
[
https://issues.apache.org/jira/browse/BEAM-8625?focusedWorklogId=370142=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-370142
]
ASF GitHub Bot logged work on BEAM-8625:
Author: ASF GitHub Bot
Created on: 10/Jan/20 23:24
Start Date: 10/Jan/20 23:24
Worklog Time Spent: 10m
Work Description: y1chi commented on pull request #10553: [BEAM-8625]
Implement servlet for exposing sdk harness statuses in Da…
URL: https://github.com/apache/beam/pull/10553#discussion_r365469125
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/SdkWorkerStatusServlet.java
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.status;
+
+import com.google.common.base.Strings;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import javax.servlet.ServletException;
+import javax.servlet.ServletOutputStream;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import org.apache.beam.runners.dataflow.worker.status.DebugCapture.Capturable;
+import org.apache.beam.runners.fnexecution.status.BeamWorkerStatusGrpcService;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Servlet dedicated to provide live status info retrieved from SDK Harness.
Note this is different
+ * from {@link WorkerStatusPages} which incorporates all info for Dataflow
runner including this
+ * SDKWorkerStatus page.
+ */
+public class SdkWorkerStatusServlet extends BaseStatusServlet implements
Capturable {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(SdkWorkerStatusServlet.class);
+ private final transient BeamWorkerStatusGrpcService statusGrpcService;
+
+ public SdkWorkerStatusServlet(BeamWorkerStatusGrpcService statusGrpcService)
{
+super("sdk_status");
+this.statusGrpcService = statusGrpcService;
+ }
+
+ @Override
+ protected String getPath(String parameters) {
+return super.getPath(parameters);
+ }
+
+ @Override
+ protected void doGet(HttpServletRequest request, HttpServletResponse
response)
+ throws IOException, ServletException {
+String id = request.getParameter("id");
+if (Strings.isNullOrEmpty(id)) {
+ // return all connected sdk statuses if no id provided.
+ response.setStatus(HttpServletResponse.SC_OK);
+ response.setContentType("text/html;charset=utf-8");
+ ServletOutputStream writer = response.getOutputStream();
+ try (PrintWriter out =
+ new PrintWriter(new OutputStreamWriter(writer,
StandardCharsets.UTF_8))) {
+captureData(out);
+response.flushBuffer();
+return;
+ }
+}
+
+response.setContentType("text/plain;charset=utf-8");
+response.setStatus(HttpServletResponse.SC_OK);
+String status = this.statusGrpcService.getSingleWorkerStatus(id, 10,
TimeUnit.SECONDS);
+ServletOutputStream writer = response.getOutputStream();
+writer.println(status);
+response.flushBuffer();
+ }
+
+ @Override
+ public String pageName() {
+return "/sdk_status";
+ }
+
+ @Override
+ public void captureData(PrintWriter writer) {
+Map allStatuses =
+this.statusGrpcService.getAllWorkerStatuses(10, TimeUnit.SECONDS);
Review comment:
done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 370142)
Time Spent: 1.5h (was: 1h 20m)
> Implement servlet in Dataflow runner for sdk status query endpoint
> --
>
> Key: BEAM-8625
> URL:
[jira] [Work logged] (BEAM-8625) Implement servlet in Dataflow runner for sdk status query endpoint
[
https://issues.apache.org/jira/browse/BEAM-8625?focusedWorklogId=370140=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-370140
]
ASF GitHub Bot logged work on BEAM-8625:
Author: ASF GitHub Bot
Created on: 10/Jan/20 23:24
Start Date: 10/Jan/20 23:24
Worklog Time Spent: 10m
Work Description: y1chi commented on pull request #10553: [BEAM-8625]
Implement servlet for exposing sdk harness statuses in Da…
URL: https://github.com/apache/beam/pull/10553#discussion_r365469050
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/SdkWorkerStatusServlet.java
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.status;
+
+import com.google.common.base.Strings;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import javax.servlet.ServletException;
+import javax.servlet.ServletOutputStream;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import org.apache.beam.runners.dataflow.worker.status.DebugCapture.Capturable;
+import org.apache.beam.runners.fnexecution.status.BeamWorkerStatusGrpcService;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Servlet dedicated to provide live status info retrieved from SDK Harness.
Note this is different
+ * from {@link WorkerStatusPages} which incorporates all info for Dataflow
runner including this
+ * SDKWorkerStatus page.
+ */
+public class SdkWorkerStatusServlet extends BaseStatusServlet implements
Capturable {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(SdkWorkerStatusServlet.class);
+ private final transient BeamWorkerStatusGrpcService statusGrpcService;
+
+ public SdkWorkerStatusServlet(BeamWorkerStatusGrpcService statusGrpcService)
{
+super("sdk_status");
+this.statusGrpcService = statusGrpcService;
+ }
+
+ @Override
+ protected String getPath(String parameters) {
+return super.getPath(parameters);
+ }
+
+ @Override
+ protected void doGet(HttpServletRequest request, HttpServletResponse
response)
+ throws IOException, ServletException {
+String id = request.getParameter("id");
+if (Strings.isNullOrEmpty(id)) {
+ // return all connected sdk statuses if no id provided.
+ response.setStatus(HttpServletResponse.SC_OK);
+ response.setContentType("text/html;charset=utf-8");
+ ServletOutputStream writer = response.getOutputStream();
+ try (PrintWriter out =
+ new PrintWriter(new OutputStreamWriter(writer,
StandardCharsets.UTF_8))) {
+captureData(out);
+response.flushBuffer();
+return;
+ }
+}
+
+response.setContentType("text/plain;charset=utf-8");
+response.setStatus(HttpServletResponse.SC_OK);
Review comment:
done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 370140)
Time Spent: 1h 10m (was: 1h)
> Implement servlet in Dataflow runner for sdk status query endpoint
> --
>
> Key: BEAM-8625
> URL: https://issues.apache.org/jira/browse/BEAM-8625
> Project: Beam
> Issue Type: Sub-task
> Components: runner-dataflow
>Reporter: Yichi Zhang
>Assignee: Yichi Zhang
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8625) Implement servlet in Dataflow runner for sdk status query endpoint
[
https://issues.apache.org/jira/browse/BEAM-8625?focusedWorklogId=370141=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-370141
]
ASF GitHub Bot logged work on BEAM-8625:
Author: ASF GitHub Bot
Created on: 10/Jan/20 23:24
Start Date: 10/Jan/20 23:24
Worklog Time Spent: 10m
Work Description: y1chi commented on pull request #10553: [BEAM-8625]
Implement servlet for exposing sdk harness statuses in Da…
URL: https://github.com/apache/beam/pull/10553#discussion_r365469103
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/SdkWorkerStatusServlet.java
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.status;
+
+import com.google.common.base.Strings;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import javax.servlet.ServletException;
+import javax.servlet.ServletOutputStream;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import org.apache.beam.runners.dataflow.worker.status.DebugCapture.Capturable;
+import org.apache.beam.runners.fnexecution.status.BeamWorkerStatusGrpcService;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Servlet dedicated to provide live status info retrieved from SDK Harness.
Note this is different
+ * from {@link WorkerStatusPages} which incorporates all info for Dataflow
runner including this
+ * SDKWorkerStatus page.
+ */
+public class SdkWorkerStatusServlet extends BaseStatusServlet implements
Capturable {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(SdkWorkerStatusServlet.class);
+ private final transient BeamWorkerStatusGrpcService statusGrpcService;
+
+ public SdkWorkerStatusServlet(BeamWorkerStatusGrpcService statusGrpcService)
{
+super("sdk_status");
+this.statusGrpcService = statusGrpcService;
+ }
+
+ @Override
+ protected String getPath(String parameters) {
+return super.getPath(parameters);
+ }
+
+ @Override
+ protected void doGet(HttpServletRequest request, HttpServletResponse
response)
+ throws IOException, ServletException {
+String id = request.getParameter("id");
+if (Strings.isNullOrEmpty(id)) {
+ // return all connected sdk statuses if no id provided.
+ response.setStatus(HttpServletResponse.SC_OK);
+ response.setContentType("text/html;charset=utf-8");
+ ServletOutputStream writer = response.getOutputStream();
+ try (PrintWriter out =
+ new PrintWriter(new OutputStreamWriter(writer,
StandardCharsets.UTF_8))) {
+captureData(out);
+response.flushBuffer();
+return;
+ }
+}
+
+response.setContentType("text/plain;charset=utf-8");
+response.setStatus(HttpServletResponse.SC_OK);
+String status = this.statusGrpcService.getSingleWorkerStatus(id, 10,
TimeUnit.SECONDS);
Review comment:
done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 370141)
Time Spent: 1h 20m (was: 1h 10m)
> Implement servlet in Dataflow runner for sdk status query endpoint
> --
>
> Key: BEAM-8625
> URL: https://issues.apache.org/jira/browse/BEAM-8625
> Project: Beam
> Issue Type: Sub-task
> Components: runner-dataflow
>Reporter: Yichi Zhang
>Assignee: Yichi Zhang
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8625) Implement servlet in Dataflow runner for sdk status query endpoint
[
https://issues.apache.org/jira/browse/BEAM-8625?focusedWorklogId=370137=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-370137
]
ASF GitHub Bot logged work on BEAM-8625:
Author: ASF GitHub Bot
Created on: 10/Jan/20 23:20
Start Date: 10/Jan/20 23:20
Worklog Time Spent: 10m
Work Description: y1chi commented on pull request #10553: [BEAM-8625]
Implement servlet for exposing sdk harness statuses in Da…
URL: https://github.com/apache/beam/pull/10553#discussion_r365468233
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/SdkWorkerStatusServlet.java
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.status;
+
+import com.google.common.base.Strings;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import javax.servlet.ServletException;
+import javax.servlet.ServletOutputStream;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import org.apache.beam.runners.dataflow.worker.status.DebugCapture.Capturable;
+import org.apache.beam.runners.fnexecution.status.BeamWorkerStatusGrpcService;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Servlet dedicated to provide live status info retrieved from SDK Harness.
Note this is different
+ * from {@link WorkerStatusPages} which incorporates all info for Dataflow
runner including this
+ * SDKWorkerStatus page.
+ */
+public class SdkWorkerStatusServlet extends BaseStatusServlet implements
Capturable {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(SdkWorkerStatusServlet.class);
+ private final transient BeamWorkerStatusGrpcService statusGrpcService;
+
+ public SdkWorkerStatusServlet(BeamWorkerStatusGrpcService statusGrpcService)
{
+super("sdk_status");
+this.statusGrpcService = statusGrpcService;
+ }
+
+ @Override
+ protected String getPath(String parameters) {
Review comment:
done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 370137)
Time Spent: 1h (was: 50m)
> Implement servlet in Dataflow runner for sdk status query endpoint
> --
>
> Key: BEAM-8625
> URL: https://issues.apache.org/jira/browse/BEAM-8625
> Project: Beam
> Issue Type: Sub-task
> Components: runner-dataflow
>Reporter: Yichi Zhang
>Assignee: Yichi Zhang
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8625) Implement servlet in Dataflow runner for sdk status query endpoint
[
https://issues.apache.org/jira/browse/BEAM-8625?focusedWorklogId=370110=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-370110
]
ASF GitHub Bot logged work on BEAM-8625:
Author: ASF GitHub Bot
Created on: 10/Jan/20 22:43
Start Date: 10/Jan/20 22:43
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #10553: [BEAM-8625]
Implement servlet for exposing sdk harness statuses in Da…
URL: https://github.com/apache/beam/pull/10553#discussion_r365459252
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/SdkWorkerStatusServlet.java
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.status;
+
+import com.google.common.base.Strings;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import javax.servlet.ServletException;
+import javax.servlet.ServletOutputStream;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import org.apache.beam.runners.dataflow.worker.status.DebugCapture.Capturable;
+import org.apache.beam.runners.fnexecution.status.BeamWorkerStatusGrpcService;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Servlet dedicated to provide live status info retrieved from SDK Harness.
Note this is different
+ * from {@link WorkerStatusPages} which incorporates all info for Dataflow
runner including this
+ * SDKWorkerStatus page.
+ */
+public class SdkWorkerStatusServlet extends BaseStatusServlet implements
Capturable {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(SdkWorkerStatusServlet.class);
+ private final transient BeamWorkerStatusGrpcService statusGrpcService;
+
+ public SdkWorkerStatusServlet(BeamWorkerStatusGrpcService statusGrpcService)
{
+super("sdk_status");
+this.statusGrpcService = statusGrpcService;
+ }
+
+ @Override
+ protected String getPath(String parameters) {
+return super.getPath(parameters);
+ }
+
+ @Override
+ protected void doGet(HttpServletRequest request, HttpServletResponse
response)
+ throws IOException, ServletException {
+String id = request.getParameter("id");
+if (Strings.isNullOrEmpty(id)) {
+ // return all connected sdk statuses if no id provided.
+ response.setStatus(HttpServletResponse.SC_OK);
+ response.setContentType("text/html;charset=utf-8");
+ ServletOutputStream writer = response.getOutputStream();
+ try (PrintWriter out =
+ new PrintWriter(new OutputStreamWriter(writer,
StandardCharsets.UTF_8))) {
+captureData(out);
+response.flushBuffer();
+return;
+ }
+}
+
+response.setContentType("text/plain;charset=utf-8");
+response.setStatus(HttpServletResponse.SC_OK);
+String status = this.statusGrpcService.getSingleWorkerStatus(id, 10,
TimeUnit.SECONDS);
Review comment:
```suggestion
String status = statusGrpcService.getSingleWorkerStatus(id, 10,
TimeUnit.SECONDS);
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 370110)
Time Spent: 40m (was: 0.5h)
> Implement servlet in Dataflow runner for sdk status query endpoint
> --
>
> Key: BEAM-8625
> URL: https://issues.apache.org/jira/browse/BEAM-8625
> Project: Beam
> Issue Type: Sub-task
> Components: runner-dataflow
>Reporter: Yichi Zhang
>Assignee: Yichi Zhang
>Priority: Major
> Time Spent:
[jira] [Work logged] (BEAM-8625) Implement servlet in Dataflow runner for sdk status query endpoint
[
https://issues.apache.org/jira/browse/BEAM-8625?focusedWorklogId=370111=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-370111
]
ASF GitHub Bot logged work on BEAM-8625:
Author: ASF GitHub Bot
Created on: 10/Jan/20 22:43
Start Date: 10/Jan/20 22:43
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #10553: [BEAM-8625]
Implement servlet for exposing sdk harness statuses in Da…
URL: https://github.com/apache/beam/pull/10553#discussion_r365450284
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowRunnerHarness.java
##
@@ -120,24 +127,35 @@ public static void main(String[] unusedArgs) throws
Exception {
loggingServer =
serverFactory.create(ImmutableList.of(beamFnLoggingService),
loggingApiService);
+ // Grpc server for obtaining SDK harness runtime status information.
+ if (statusApiService != null && beamWorkerStatusGrpcService != null) {
+statusServer =
+
serverFactory.create(ImmutableList.of(beamWorkerStatusGrpcService),
statusApiService);
+ }
+
start(
pipeline,
pipelineOptions,
beamFnControlService,
beamFnDataService,
controlApiService,
- beamFnStateService);
- servicesServer.shutdown();
- loggingServer.shutdown();
+ beamFnStateService,
+ beamWorkerStatusGrpcService);
+ servicesServer.shutdown().awaitTermination(30, TimeUnit.SECONDS);
+ loggingServer.shutdown().awaitTermination(30, TimeUnit.SECONDS);
Review comment:
Lets shutdown the logging server at the last.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 370111)
Time Spent: 40m (was: 0.5h)
> Implement servlet in Dataflow runner for sdk status query endpoint
> --
>
> Key: BEAM-8625
> URL: https://issues.apache.org/jira/browse/BEAM-8625
> Project: Beam
> Issue Type: Sub-task
> Components: runner-dataflow
>Reporter: Yichi Zhang
>Assignee: Yichi Zhang
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8625) Implement servlet in Dataflow runner for sdk status query endpoint
[
https://issues.apache.org/jira/browse/BEAM-8625?focusedWorklogId=370113=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-370113
]
ASF GitHub Bot logged work on BEAM-8625:
Author: ASF GitHub Bot
Created on: 10/Jan/20 22:43
Start Date: 10/Jan/20 22:43
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #10553: [BEAM-8625]
Implement servlet for exposing sdk harness statuses in Da…
URL: https://github.com/apache/beam/pull/10553#discussion_r365459487
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/SdkWorkerStatusServlet.java
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.status;
+
+import com.google.common.base.Strings;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import javax.servlet.ServletException;
+import javax.servlet.ServletOutputStream;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import org.apache.beam.runners.dataflow.worker.status.DebugCapture.Capturable;
+import org.apache.beam.runners.fnexecution.status.BeamWorkerStatusGrpcService;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Servlet dedicated to provide live status info retrieved from SDK Harness.
Note this is different
+ * from {@link WorkerStatusPages} which incorporates all info for Dataflow
runner including this
+ * SDKWorkerStatus page.
+ */
+public class SdkWorkerStatusServlet extends BaseStatusServlet implements
Capturable {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(SdkWorkerStatusServlet.class);
+ private final transient BeamWorkerStatusGrpcService statusGrpcService;
+
+ public SdkWorkerStatusServlet(BeamWorkerStatusGrpcService statusGrpcService)
{
+super("sdk_status");
+this.statusGrpcService = statusGrpcService;
+ }
+
+ @Override
+ protected String getPath(String parameters) {
+return super.getPath(parameters);
+ }
+
+ @Override
+ protected void doGet(HttpServletRequest request, HttpServletResponse
response)
+ throws IOException, ServletException {
+String id = request.getParameter("id");
+if (Strings.isNullOrEmpty(id)) {
+ // return all connected sdk statuses if no id provided.
+ response.setStatus(HttpServletResponse.SC_OK);
+ response.setContentType("text/html;charset=utf-8");
+ ServletOutputStream writer = response.getOutputStream();
+ try (PrintWriter out =
+ new PrintWriter(new OutputStreamWriter(writer,
StandardCharsets.UTF_8))) {
+captureData(out);
+response.flushBuffer();
+return;
+ }
+}
+
+response.setContentType("text/plain;charset=utf-8");
+response.setStatus(HttpServletResponse.SC_OK);
Review comment:
We can do this before the if block so that it is not duplicated.
```
protected void doGet(HttpServletRequest request, HttpServletResponse
response)
throws IOException, ServletException {
String id = request.getParameter("id");
if (Strings.isNullOrEmpty(id)) {
// return all connected sdk statuses if no id provided.
response.setContentType("text/html;charset=utf-8");
ServletOutputStream writer = response.getOutputStream();
try (PrintWriter out =
new PrintWriter(new OutputStreamWriter(writer,
StandardCharsets.UTF_8))) {
captureData(out);
}
} else {
response.setContentType("text/plain;charset=utf-8");
ServletOutputStream writer = response.getOutputStream();
writer.println(statusGrpcService.getSingleWorkerStatus(id, 10,
TimeUnit.SECONDS););
}
response.setStatus(HttpServletResponse.SC_OK);
response.flushBuffer();
}
```
This is an automated message from the
[jira] [Work logged] (BEAM-8625) Implement servlet in Dataflow runner for sdk status query endpoint
[
https://issues.apache.org/jira/browse/BEAM-8625?focusedWorklogId=370107=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-370107
]
ASF GitHub Bot logged work on BEAM-8625:
Author: ASF GitHub Bot
Created on: 10/Jan/20 22:43
Start Date: 10/Jan/20 22:43
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #10553: [BEAM-8625]
Implement servlet for exposing sdk harness statuses in Da…
URL: https://github.com/apache/beam/pull/10553#discussion_r365447360
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowRunnerHarness.java
##
@@ -120,24 +127,35 @@ public static void main(String[] unusedArgs) throws
Exception {
loggingServer =
serverFactory.create(ImmutableList.of(beamFnLoggingService),
loggingApiService);
+ // Grpc server for obtaining SDK harness runtime status information.
+ if (statusApiService != null && beamWorkerStatusGrpcService != null) {
Review comment:
This check seems to be redundant here.
If we expect `statusApiService` to be null, then we should check it before
`beamWorkerStatusGrpcService` creation.
Also, I expect `BeamWorkerStatusGrpcService.create` to throw exception if
the creation fails so it will never be null.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 370107)
Time Spent: 0.5h (was: 20m)
> Implement servlet in Dataflow runner for sdk status query endpoint
> --
>
> Key: BEAM-8625
> URL: https://issues.apache.org/jira/browse/BEAM-8625
> Project: Beam
> Issue Type: Sub-task
> Components: runner-dataflow
>Reporter: Yichi Zhang
>Assignee: Yichi Zhang
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8625) Implement servlet in Dataflow runner for sdk status query endpoint
[
https://issues.apache.org/jira/browse/BEAM-8625?focusedWorklogId=370109=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-370109
]
ASF GitHub Bot logged work on BEAM-8625:
Author: ASF GitHub Bot
Created on: 10/Jan/20 22:43
Start Date: 10/Jan/20 22:43
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #10553: [BEAM-8625]
Implement servlet for exposing sdk harness statuses in Da…
URL: https://github.com/apache/beam/pull/10553#discussion_r365445145
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/WorkerStatusPages.java
##
@@ -108,7 +113,11 @@ public void addStatusDataProvider(
/** Returns the set of pages than should be captured by DebugCapture. */
public List getDebugCapturePages() {
-return Arrays.asList(threadzServlet, statuszServlet);
+return this.capturePages;
Review comment:
Lets make a copy of this list as it can potentially be modified by the
consumer or change the return type to collection.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 370109)
> Implement servlet in Dataflow runner for sdk status query endpoint
> --
>
> Key: BEAM-8625
> URL: https://issues.apache.org/jira/browse/BEAM-8625
> Project: Beam
> Issue Type: Sub-task
> Components: runner-dataflow
>Reporter: Yichi Zhang
>Assignee: Yichi Zhang
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8625) Implement servlet in Dataflow runner for sdk status query endpoint
[
https://issues.apache.org/jira/browse/BEAM-8625?focusedWorklogId=370114=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-370114
]
ASF GitHub Bot logged work on BEAM-8625:
Author: ASF GitHub Bot
Created on: 10/Jan/20 22:43
Start Date: 10/Jan/20 22:43
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #10553: [BEAM-8625]
Implement servlet for exposing sdk harness statuses in Da…
URL: https://github.com/apache/beam/pull/10553#discussion_r365454596
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/SdkWorkerStatusServlet.java
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.status;
+
+import com.google.common.base.Strings;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import javax.servlet.ServletException;
+import javax.servlet.ServletOutputStream;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import org.apache.beam.runners.dataflow.worker.status.DebugCapture.Capturable;
+import org.apache.beam.runners.fnexecution.status.BeamWorkerStatusGrpcService;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Servlet dedicated to provide live status info retrieved from SDK Harness.
Note this is different
+ * from {@link WorkerStatusPages} which incorporates all info for Dataflow
runner including this
+ * SDKWorkerStatus page.
+ */
+public class SdkWorkerStatusServlet extends BaseStatusServlet implements
Capturable {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(SdkWorkerStatusServlet.class);
+ private final transient BeamWorkerStatusGrpcService statusGrpcService;
+
+ public SdkWorkerStatusServlet(BeamWorkerStatusGrpcService statusGrpcService)
{
+super("sdk_status");
+this.statusGrpcService = statusGrpcService;
+ }
+
+ @Override
+ protected String getPath(String parameters) {
Review comment:
We can skip reimplmenting this method.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 370114)
> Implement servlet in Dataflow runner for sdk status query endpoint
> --
>
> Key: BEAM-8625
> URL: https://issues.apache.org/jira/browse/BEAM-8625
> Project: Beam
> Issue Type: Sub-task
> Components: runner-dataflow
>Reporter: Yichi Zhang
>Assignee: Yichi Zhang
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8625) Implement servlet in Dataflow runner for sdk status query endpoint
[
https://issues.apache.org/jira/browse/BEAM-8625?focusedWorklogId=370108=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-370108
]
ASF GitHub Bot logged work on BEAM-8625:
Author: ASF GitHub Bot
Created on: 10/Jan/20 22:43
Start Date: 10/Jan/20 22:43
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #10553: [BEAM-8625]
Implement servlet for exposing sdk harness statuses in Da…
URL: https://github.com/apache/beam/pull/10553#discussion_r365451013
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowRunnerHarness.java
##
@@ -120,24 +127,35 @@ public static void main(String[] unusedArgs) throws
Exception {
loggingServer =
serverFactory.create(ImmutableList.of(beamFnLoggingService),
loggingApiService);
+ // Grpc server for obtaining SDK harness runtime status information.
+ if (statusApiService != null && beamWorkerStatusGrpcService != null) {
+statusServer =
+
serverFactory.create(ImmutableList.of(beamWorkerStatusGrpcService),
statusApiService);
+ }
+
start(
pipeline,
pipelineOptions,
beamFnControlService,
beamFnDataService,
controlApiService,
- beamFnStateService);
- servicesServer.shutdown();
- loggingServer.shutdown();
+ beamFnStateService,
+ beamWorkerStatusGrpcService);
+ servicesServer.shutdown().awaitTermination(30, TimeUnit.SECONDS);
Review comment:
Can we shut them down in parallel?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 370108)
> Implement servlet in Dataflow runner for sdk status query endpoint
> --
>
> Key: BEAM-8625
> URL: https://issues.apache.org/jira/browse/BEAM-8625
> Project: Beam
> Issue Type: Sub-task
> Components: runner-dataflow
>Reporter: Yichi Zhang
>Assignee: Yichi Zhang
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8625) Implement servlet in Dataflow runner for sdk status query endpoint
[
https://issues.apache.org/jira/browse/BEAM-8625?focusedWorklogId=370115=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-370115
]
ASF GitHub Bot logged work on BEAM-8625:
Author: ASF GitHub Bot
Created on: 10/Jan/20 22:43
Start Date: 10/Jan/20 22:43
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #10553: [BEAM-8625]
Implement servlet for exposing sdk harness statuses in Da…
URL: https://github.com/apache/beam/pull/10553#discussion_r365459302
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/SdkWorkerStatusServlet.java
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.status;
+
+import com.google.common.base.Strings;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import javax.servlet.ServletException;
+import javax.servlet.ServletOutputStream;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import org.apache.beam.runners.dataflow.worker.status.DebugCapture.Capturable;
+import org.apache.beam.runners.fnexecution.status.BeamWorkerStatusGrpcService;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Servlet dedicated to provide live status info retrieved from SDK Harness.
Note this is different
+ * from {@link WorkerStatusPages} which incorporates all info for Dataflow
runner including this
+ * SDKWorkerStatus page.
+ */
+public class SdkWorkerStatusServlet extends BaseStatusServlet implements
Capturable {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(SdkWorkerStatusServlet.class);
+ private final transient BeamWorkerStatusGrpcService statusGrpcService;
+
+ public SdkWorkerStatusServlet(BeamWorkerStatusGrpcService statusGrpcService)
{
+super("sdk_status");
+this.statusGrpcService = statusGrpcService;
+ }
+
+ @Override
+ protected String getPath(String parameters) {
+return super.getPath(parameters);
+ }
+
+ @Override
+ protected void doGet(HttpServletRequest request, HttpServletResponse
response)
+ throws IOException, ServletException {
+String id = request.getParameter("id");
+if (Strings.isNullOrEmpty(id)) {
+ // return all connected sdk statuses if no id provided.
+ response.setStatus(HttpServletResponse.SC_OK);
+ response.setContentType("text/html;charset=utf-8");
+ ServletOutputStream writer = response.getOutputStream();
+ try (PrintWriter out =
+ new PrintWriter(new OutputStreamWriter(writer,
StandardCharsets.UTF_8))) {
+captureData(out);
+response.flushBuffer();
+return;
+ }
+}
+
+response.setContentType("text/plain;charset=utf-8");
+response.setStatus(HttpServletResponse.SC_OK);
+String status = this.statusGrpcService.getSingleWorkerStatus(id, 10,
TimeUnit.SECONDS);
+ServletOutputStream writer = response.getOutputStream();
+writer.println(status);
+response.flushBuffer();
+ }
+
+ @Override
+ public String pageName() {
+return "/sdk_status";
+ }
+
+ @Override
+ public void captureData(PrintWriter writer) {
+Map allStatuses =
+this.statusGrpcService.getAllWorkerStatuses(10, TimeUnit.SECONDS);
Review comment:
```suggestion
statusGrpcService.getAllWorkerStatuses(10, TimeUnit.SECONDS);
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 370115)
Time Spent: 50m (was: 40m)
> Implement servlet in Dataflow runner for sdk status query endpoint
>
[jira] [Work logged] (BEAM-8625) Implement servlet in Dataflow runner for sdk status query endpoint
[
https://issues.apache.org/jira/browse/BEAM-8625?focusedWorklogId=370112=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-370112
]
ASF GitHub Bot logged work on BEAM-8625:
Author: ASF GitHub Bot
Created on: 10/Jan/20 22:43
Start Date: 10/Jan/20 22:43
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #10553: [BEAM-8625]
Implement servlet for exposing sdk harness statuses in Da…
URL: https://github.com/apache/beam/pull/10553#discussion_r365452599
##
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java
##
@@ -871,6 +871,10 @@ public void startStatusPages() {
statusPages.start();
}
+ public WorkerStatusPages getStatusPages() {
Review comment:
Let's expose addStatusPage in the worker. Which will add the page as both
servlet and capturePage.
Alternatively, we can pass `beamWorkerStatusGrpcService` to
`StreamingDataflowWorker` where it can create the `sdkWorkerStatusServlet` and
use it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 370112)
Time Spent: 40m (was: 0.5h)
> Implement servlet in Dataflow runner for sdk status query endpoint
> --
>
> Key: BEAM-8625
> URL: https://issues.apache.org/jira/browse/BEAM-8625
> Project: Beam
> Issue Type: Sub-task
> Components: runner-dataflow
>Reporter: Yichi Zhang
>Assignee: Yichi Zhang
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8625) Implement servlet in Dataflow runner for sdk status query endpoint
[ https://issues.apache.org/jira/browse/BEAM-8625?focusedWorklogId=370040=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-370040 ] ASF GitHub Bot logged work on BEAM-8625: Author: ASF GitHub Bot Created on: 10/Jan/20 19:57 Start Date: 10/Jan/20 19:57 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10553: [BEAM-8625] Implement servlet for exposing sdk harness statuses in Da… URL: https://github.com/apache/beam/pull/10553#issuecomment-573183192 R: @angoenka Ankur could you help review this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 370040) Time Spent: 20m (was: 10m) > Implement servlet in Dataflow runner for sdk status query endpoint > -- > > Key: BEAM-8625 > URL: https://issues.apache.org/jira/browse/BEAM-8625 > Project: Beam > Issue Type: Sub-task > Components: runner-dataflow >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8625) Implement servlet in Dataflow runner for sdk status query endpoint
[ https://issues.apache.org/jira/browse/BEAM-8625?focusedWorklogId=370030=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-370030 ] ASF GitHub Bot logged work on BEAM-8625: Author: ASF GitHub Bot Created on: 10/Jan/20 19:46 Start Date: 10/Jan/20 19:46 Worklog Time Spent: 10m Work Description: y1chi commented on pull request #10553: [BEAM-8625] Implement servlet for exposing sdk harness statuses in Da… URL: https://github.com/apache/beam/pull/10553 …taflow runner **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[