Hi all, I'm planning to use Apache beam to extract and load part of the ETL pipeline and run the jobs on Dataflow. I will have to do the REST API ingestion on our platform. I can opt to make sync API calls from DoFn. With that pipelines will stall while REST requests are made over the network.
Is it best practice to run the REST ingestion job on Dataflow? Is there any best practice I can follow to accomplish this? Just as a reference I'm adding this <https://stackoverflow.com/questions/50335521/best-practices-in-http-calls-in-cloud-dataflow-java> StackOverflow thread here too. Also, I notice that Rest I/O transform <https://beam.apache.org/documentation/io/built-in/> built-in connector is in progress for Java. Let me know if this is the right group to ask this question. I can also ask d...@beam.apache.org if needed. -- Thanks, Shree
