[
https://issues.apache.org/jira/browse/BEAM-6666?focusedWorklogId=206164&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-206164
]
ASF GitHub Bot logged work on BEAM-6666:
----------------------------------------
Author: ASF GitHub Bot
Created on: 01/Mar/19 00:59
Start Date: 01/Mar/19 00:59
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #7845: [BEAM-6666]
subprocess.Popen hangs after use of gRPC channel
URL: https://github.com/apache/beam/pull/7845#discussion_r261447318
##########
File path: sdks/python/build.gradle
##########
@@ -456,10 +453,39 @@ project.task('crossLanguagePythonJava') {
dependsOn 'setupVirtualenv'
dependsOn ':beam-sdks-java-container:docker'
dependsOn ':beam-runners-core-construction-java:buildTestExpansionServiceJar'
+
doLast {
+ def testServiceExpansionJar =
project(":beam-runners-core-construction-java:").buildTestExpansionServiceJar.archivePath
+ exec {
+ executable 'sh'
+ args '-c', ". ${project.ext.envdir}/bin/activate && pip install -e
.[test] && python -m apache_beam.transforms.external_test
--expansion_service_jar=${testServiceExpansionJar}"
+ }
+ }
+}
+
+project.task('crossLanguagePortableWordCount') {
+ dependsOn 'setupVirtualenv'
+ dependsOn ':beam-runners-flink_2.11-job-server-container:docker'
+ dependsOn ':beam-sdks-python-container:docker'
+ dependsOn ':beam-sdks-java-container:docker'
+ dependsOn ':beam-runners-core-construction-java:buildTestExpansionServiceJar'
+
+ doLast {
+ def testServiceExpansionJar =
project(":beam-runners-core-construction-java:").buildTestExpansionServiceJar.archivePath
Review comment:
Thanks for your comment. IMHO, `core-construction-java` is the better place
for `ExpansionService` since `ExpansionService` mainly works during the
pipeline construction phase. And, because `java-fn-execution` depends on
`core-construction-java`, we can still bundle `ExpansionService` with
`JobServer` without moving it. When we put `ExpansionService` in
`java-fn-execution`, it will block us from creating a class in
`core-construction-java` which uses `ExpansionService` class (e.g., external
transform for Java SDK and its test).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 206164)
Time Spent: 2h (was: 1h 50m)
> subprocess.Popen hangs after use of gRPC channel
> ------------------------------------------------
>
> Key: BEAM-6666
> URL: https://issues.apache.org/jira/browse/BEAM-6666
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Reporter: Heejong Lee
> Assignee: Heejong Lee
> Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
> subprocess.Popen randomly hangs after use of gRPC channel. This makes
> cross-language wordcount test fail because the test uses Popen to launch
> Dockerized Flink job server in `PortableRunner.run_pipeline` after use of
> gRPC channel for the expansion service in `ExternalTransform.expand`. Few
> symptoms are listed below:
> * Hanging at `docker_path = check_output(['which', 'docker']).strip()`
> * Hanging at `self.docker_process = Popen(cmd)`
> * Crashing with `assertion failed: pthread_mutex_lock(mu) == 0` message
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
