[GitHub] [beam] jaketf edited a comment on pull request #11339: [BEAM-9468] Fhir io
jaketf edited a comment on pull request #11339: URL: https://github.com/apache/beam/pull/11339#issuecomment-631265943 I think that something about `TestPubsubSignal` does not play well with the way this integration test suite runs on Dataflow. Following in suit with `PubsubReadIT` (which also uses `TestPububSignal`) `BigQueryIOReadIT` and `BigQueryIOStorageReadTableRowIT` I've added this to the exclude of this project's build file. This test can still be run (and passes) using DirectRunner and the evidence in the [above comment](https://github.com/apache/beam/pull/11339#issuecomment-631203002) points to this being a red herring of an issue with test pubsub signal and dataflow runner. Potentially related issue: [BEAM-6804](https://issues.apache.org/jira/browse/BEAM-6804?jql=text%20~%20%22TestPubsubSignal%22) I've filed an issue for this instance [BEAM-10040](https://issues.apache.org/jira/browse/BEAM-10040). @pabloem please let me know if this is acceptable. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] jaketf edited a comment on pull request #11339: [BEAM-9468] Fhir io
jaketf edited a comment on pull request #11339: URL: https://github.com/apache/beam/pull/11339#issuecomment-631203002 The issue with FhirIOReadIT seems to be some misuse of [TestPubsubSignal](https://beam.apache.org/releases/javadoc/2.20.0/org/apache/beam/sdk/io/gcp/pubsub/TestPubsubSignal.html) [Example Job](https://pantheon.corp.google.com/dataflow/jobs/us-central1/2020-05-19_19_05_19-1057064265214622054?project=apache-beam-testing) clearly has the expected >2000 elements added to the "waitForAnyMessage" task but the success signal never gets published to the results topic. Notably there are job level warnings about metric descriptors and [warnings in shuffle logs](https://pantheon.corp.google.com/logs/viewer?dateRangeEnd=2020-05-20T02:18:25.642Z=2020-05-20T02:05:21.347Z=false=CUSTOM=apache-beam-testing=dataflow_step%2Fjob_id%2F2020-05-19_19_05_19-1057064265214622054=2020-05-20T02:57:46.18700Z=projects%2Fapache-beam-testing%2Flogs%2Fdataflow.googleapis.com%252Fshuffler=0==true=2020-05-20T02:15:27.999165390Z) which warns: ``` "Update range task returned 'invalid argument'. Assuming lost lease for work with id 5061980071068333770 (expiration time: 1589940982000, now: 1589940923590, full status: INVALID_ARGUMENT: Http(400) Bad Request). For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors.; ``` the [docs](https://cloud.google.com/dataflow/docs/guides/common-errors#bad-request-shuffler-logs) say this can be ignored but smells suspicious here. This is orthogonal to the behavior being tested. Investigating other means of performing this test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] jaketf edited a comment on pull request #11339: [BEAM-9468] Fhir io
jaketf edited a comment on pull request #11339: URL: https://github.com/apache/beam/pull/11339#issuecomment-631203002 The issue with FhirIOReadIT seems to be some misuse of [TestPubsubSignal](https://beam.apache.org/releases/javadoc/2.20.0/org/apache/beam/sdk/io/gcp/pubsub/TestPubsubSignal.html) [Example Job](https://pantheon.corp.google.com/dataflow/jobs/us-central1/2020-05-19_19_05_19-1057064265214622054?project=apache-beam-testing) clearly has elements added to the "waitForAnyMessage" task but the success signal never gets published to the results topic. Notably there are job level warnings about metric descriptors and [warnings in shuffle logs](https://pantheon.corp.google.com/logs/viewer?dateRangeEnd=2020-05-20T02:18:25.642Z=2020-05-20T02:05:21.347Z=false=CUSTOM=apache-beam-testing=dataflow_step%2Fjob_id%2F2020-05-19_19_05_19-1057064265214622054=2020-05-20T02:57:46.18700Z=projects%2Fapache-beam-testing%2Flogs%2Fdataflow.googleapis.com%252Fshuffler=0==true=2020-05-20T02:15:27.999165390Z) which warns: ``` "Update range task returned 'invalid argument'. Assuming lost lease for work with id 5061980071068333770 (expiration time: 1589940982000, now: 1589940923590, full status: INVALID_ARGUMENT: Http(400) Bad Request). For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors.; ``` the [docs](https://cloud.google.com/dataflow/docs/guides/common-errors#bad-request-shuffler-logs) say this can be ignored but smells suspicious here. This is orthogonal to the behavior being tested. Investigating other means of performing this test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] jaketf edited a comment on pull request #11339: [BEAM-9468] Fhir io
jaketf edited a comment on pull request #11339: URL: https://github.com/apache/beam/pull/11339#issuecomment-631203002 The issue with FhirIOReadIT seems to be some misuse of [TestPubsubSignal](https://beam.apache.org/releases/javadoc/2.20.0/org/apache/beam/sdk/io/gcp/pubsub/TestPubsubSignal.html) [Example Job](https://pantheon.corp.google.com/dataflow/jobs/us-central1/2020-05-19_19_05_19-1057064265214622054?project=apache-beam-testing) clearly has elements added to the "waitForAnyMessage" task but the success signal never gets published to the results topic. Notably there are [warnings in shuffle logs](https://pantheon.corp.google.com/logs/viewer?dateRangeEnd=2020-05-20T02:18:25.642Z=2020-05-20T02:05:21.347Z=false=CUSTOM=apache-beam-testing=dataflow_step%2Fjob_id%2F2020-05-19_19_05_19-1057064265214622054=2020-05-20T02:57:46.18700Z=projects%2Fapache-beam-testing%2Flogs%2Fdataflow.googleapis.com%252Fshuffler=0==true=2020-05-20T02:15:27.999165390Z) which warns: ``` "Update range task returned 'invalid argument'. Assuming lost lease for work with id 5061980071068333770 (expiration time: 1589940982000, now: 1589940923590, full status: INVALID_ARGUMENT: Http(400) Bad Request). For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors.; ``` the [docs](https://cloud.google.com/dataflow/docs/guides/common-errors#bad-request-shuffler-logs) say this can be ignored but smells suspicious here. This is orthogonal to the behavior being tested. Investigating other means of performing this test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] jaketf edited a comment on pull request #11339: [BEAM-9468] Fhir io
jaketf edited a comment on pull request #11339: URL: https://github.com/apache/beam/pull/11339#issuecomment-631203002 The issue with FhirIOReadIT seems to be some misuse of [TestPubsubSignal](https://beam.apache.org/releases/javadoc/2.20.0/org/apache/beam/sdk/io/gcp/pubsub/TestPubsubSignal.html) [Example Job](https://pantheon.corp.google.com/dataflow/jobs/us-central1/2020-05-19_19_05_19-1057064265214622054?project=apache-beam-testing) clearly has elements added to the "waitForAnyMessage" task but the success signal never gets published to the results topic. This is orthogonal to the behavior being tested. Investigating other means of performing this test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] jaketf edited a comment on pull request #11339: [BEAM-9468] Fhir io
jaketf edited a comment on pull request #11339:
URL: https://github.com/apache/beam/pull/11339#issuecomment-628842736
Yeah all the FhirIO read tests are parameterized tests that are all failing
like this:
```
WARNING: No terminal state was returned within allotted timeout. State value
RUNNING
May 14, 2020 2:37:23 AM org.apache.beam.runners.dataflow.TestDataflowRunner
waitForStreamingJobTermination
INFO: Dataflow job 2020-05-13_19_27_22-9842849986096969383 took longer than
600 seconds to complete, cancelling.
May 14, 2020 2:37:23 AM org.apache.beam.runners.dataflow.TestDataflowRunner
run
WARNING: Dataflow job 2020-05-13_19_27_22-9842849986096969383 did not output
a success or failure metric.
May 14, 2020 2:37:24 AM
io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue
SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=15,
target=pubsub.googleapis.com:443} was not shutdown properly!!! ~*~*~*
Make sure to call shutdown()/shutdownNow() and wait until
awaitTermination() returns true.
java.lang.RuntimeException: ManagedChannel allocation site
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [beam] jaketf edited a comment on pull request #11339: [BEAM-9468] Fhir io
jaketf edited a comment on pull request #11339: URL: https://github.com/apache/beam/pull/11339#issuecomment-628833485 > I'm unsure if the failures are due to an issue with this PR. Feel free to inspect: > https://scans.gradle.com/s/c4wwwmeidugrc/failure#top=0 There's not much information in that scan. Running `./gradlew checkstyleMain checkstyleTest` locally there was an issue in FhirIO ``` > Task :sdks:java:io:google-cloud-platform:checkstyleMain FAILED [ant:checkstyle] [ERROR] /usr/local/google/home/jferriero/VersionControl/beam/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/FhirIO.java:846:42: Name 'LOG' must match pattern '^[a-z][a-zA-Z0-9]*$'. [MemberName] ``` I'll clean this up. However there were many failures due to FlinkState internals [full log](https://gist.github.com/jaketf/4c784c6e9831caac1a9aaf4d25926b69) > Fhir postcomits are failing: https://builds.apache.org/job/beam_PostCommit_Java_PR/362/ ACK looking into this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] jaketf edited a comment on pull request #11339: [BEAM-9468] Fhir io
jaketf edited a comment on pull request #11339: URL: https://github.com/apache/beam/pull/11339#issuecomment-621525975 @pabloem @lastomato the refactor to remove unnecessary use of `HttpBody` model (and associated coder) in favor of `String` is complete, plus an adding of HTTP status code similar to the one recently added in #11538 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
