[GitHub] [beam] jaketf commented on a change in pull request #11702: [BEAM-9990] Add Conditional Update and Conditional Create to FhirIO

2020-06-09 Thread GitBox


jaketf commented on a change in pull request #11702:
URL: https://github.com/apache/beam/pull/11702#discussion_r437590913



##
File path: 
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/FhirIO.java
##
@@ -155,17 +168,53 @@
  * FhirIO.Write.Result writeResult =
  * output.apply("Execute FHIR Bundles", 
FhirIO.executeBundles(options.getExistingFhirStore()));
  *
+ * // Alternatively you could use import for high throughput to a new store.
+ * FhirIO.Write.Result writeResult =
+ * output.apply("Import FHIR Resources", 
FhirIO.executeBundles(options.getNewFhirStore()));
+ * // [End Writing ]
+ *
  * PCollection> failedBundles = 
writeResult.getFailedInsertsWithErr();
  *
+ * // [Begin Writing to Dead Letter Queue]
  * failedBundles.apply("Write failed bundles to BigQuery",
  * BigQueryIO
  * .write()
  * .to(option.getBQFhirExecuteBundlesDeadLetterTable())
  * .withFormatFunction(new HealthcareIOErrorToTableRow()));
+ * // [End Writing to Dead Letter Queue]
+ *
+ * // Alternatively you may want to handle DeadLetter with conditional update
+ * // [Begin Reconciliation with Conditional Update]
+ * failedBundles
+ * .apply("Reconcile with Conditional Update",
+ * FhirIO.ConditionalUpdate(fhirStore)
+ * 
.withFormatBodyFunction(HealthcareIOError::getDataResource)
+ * .withTypeFunction((HealthcareIOError err) -> {
+ *   String body = err.getDataResource();
+ *   // TODO(user) insert logic to exctract type.
+ *   return params;
+ * })
+ * .withSearchParametersFunction((HealthcareIOError err) 
-> {

Review comment:
   Hmm on second thought the best way to do this if for the user to use 
HAPI library search builder which will be better maintained. 
   I think we can add this as a suggestion in the docs rather than making this 
a beam dependency or duplicating the functionality. Thoughts?
   https://hapifhir.io/hapi-fhir/docs/server_plain/rest_operations_search.html





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [beam] jaketf commented on a change in pull request #11702: [BEAM-9990] Add Conditional Update and Conditional Create to FhirIO

2020-06-09 Thread GitBox


jaketf commented on a change in pull request #11702:
URL: https://github.com/apache/beam/pull/11702#discussion_r436837505



##
File path: 
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/FhirIO.java
##
@@ -155,17 +168,53 @@
  * FhirIO.Write.Result writeResult =
  * output.apply("Execute FHIR Bundles", 
FhirIO.executeBundles(options.getExistingFhirStore()));
  *
+ * // Alternatively you could use import for high throughput to a new store.
+ * FhirIO.Write.Result writeResult =
+ * output.apply("Import FHIR Resources", 
FhirIO.executeBundles(options.getNewFhirStore()));
+ * // [End Writing ]
+ *
  * PCollection> failedBundles = 
writeResult.getFailedInsertsWithErr();
  *
+ * // [Begin Writing to Dead Letter Queue]
  * failedBundles.apply("Write failed bundles to BigQuery",
  * BigQueryIO
  * .write()
  * .to(option.getBQFhirExecuteBundlesDeadLetterTable())
  * .withFormatFunction(new HealthcareIOErrorToTableRow()));
+ * // [End Writing to Dead Letter Queue]
+ *
+ * // Alternatively you may want to handle DeadLetter with conditional update
+ * // [Begin Reconciliation with Conditional Update]
+ * failedBundles
+ * .apply("Reconcile with Conditional Update",
+ * FhirIO.ConditionalUpdate(fhirStore)
+ * 
.withFormatBodyFunction(HealthcareIOError::getDataResource)
+ * .withTypeFunction((HealthcareIOError err) -> {
+ *   String body = err.getDataResource();
+ *   // TODO(user) insert logic to exctract type.
+ *   return params;
+ * })
+ * .withSearchParametersFunction((HealthcareIOError err) 
-> {

Review comment:
   this is a great suggestion will add a FhirSearchParameter Builder.

##
File path: 
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/FhirIO.java
##
@@ -155,17 +168,53 @@
  * FhirIO.Write.Result writeResult =
  * output.apply("Execute FHIR Bundles", 
FhirIO.executeBundles(options.getExistingFhirStore()));
  *
+ * // Alternatively you could use import for high throughput to a new store.
+ * FhirIO.Write.Result writeResult =
+ * output.apply("Import FHIR Resources", 
FhirIO.executeBundles(options.getNewFhirStore()));
+ * // [End Writing ]
+ *
  * PCollection> failedBundles = 
writeResult.getFailedInsertsWithErr();
  *
+ * // [Begin Writing to Dead Letter Queue]
  * failedBundles.apply("Write failed bundles to BigQuery",
  * BigQueryIO
  * .write()
  * .to(option.getBQFhirExecuteBundlesDeadLetterTable())
  * .withFormatFunction(new HealthcareIOErrorToTableRow()));
+ * // [End Writing to Dead Letter Queue]
+ *
+ * // Alternatively you may want to handle DeadLetter with conditional update
+ * // [Begin Reconciliation with Conditional Update]
+ * failedBundles
+ * .apply("Reconcile with Conditional Update",
+ * FhirIO.ConditionalUpdate(fhirStore)
+ * 
.withFormatBodyFunction(HealthcareIOError::getDataResource)
+ * .withTypeFunction((HealthcareIOError err) -> {
+ *   String body = err.getDataResource();
+ *   // TODO(user) insert logic to exctract type.
+ *   return params;

Review comment:
   These are just examples. For R4 the exhaustive list is 
[here](https://www.hl7.org/fhir/valueset-resource-types.html).
   
   I don't want to add an enum because I think different versions of FHIR may 
provide different resource types and I want FhirIO implementation be pretty 
version agnostic. Though we test against all versions, the code path is 
identical.
   
   If we were to add an enum I wouldn't want to maintain it as part of beam. 
   
   If we want to institute stronger typing and domain sanity checks, we could 
consider adding a dependency on the open source [HAPI 
library](https://hapifhir.io/hapi-fhir/apidocs/hapi-fhir-structures-r4/org/hl7/fhir/r4/model/package-summary.html)
 which is maintained by the HL7 community and is pretty popular.

##
File path: 
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/FhirIO.java
##
@@ -1173,4 +1276,339 @@ public void executeBundles(ProcessContext context) {
   }
 }
   }
+
+  /**
+   * Create resources fhir io . create resources.
+   *
+   * @param  the type parameter
+   * @param fhirStore the fhir store
+   * @return the fhir io . create resources
+   */
+  public static  FhirIO.CreateResources 
createResources(ValueProvider fhirStore) {
+return new CreateResources(fhirStore);
+  }
+
+  /**
+   * Create resources fhir io . create resources.
+   *
+   * @param  the type parameter
+   * @param fhirStore the fhir store
+   * @return the fhir io . create resources
+   */
+  public static  FhirIO.CreateResources createResources(String 
fhirStore) {
+return new CreateResources(fhirStore);
+  }
+  /**
+   * {@link PTransform} for Creating FHIR resources.
+