Hi all, We’d like to implement a set of PTransforms that would allow users to use some of the Google Cloud AI services in Beam pipelines.
Here's the full list of services and functionalities we’d like to integrate Beam with: * Video Intelligence [1] * Cloud Natural Language [2] * Cloud AI Platform Prediction [3] * Data Masking/Tokenization [4] * Inspecting image data for sensitive information using Cloud Vision [5] However, we're not sure whether to put those transforms directly into Beam, because they would require some additional GCP dependencies. One of our ideas is a separate library, that depends on Beam and that can be installed optionally, stored somewhere in the beam repository (e.g. in the BEAM_ROOT/extras directory). Do you think it is a reasonable approach? Or maybe it is totally fine to put them into SDKs, just like other IOs? If you have any other thoughts, do not hesitate to let us know. Best, Kamil [1] https://cloud.google.com/video-intelligence/ [2] https://cloud.google.com/natural-language/ [3] https://cloud.google.com/ml-engine/docs/prediction-overview [4] https://cloud.google.com/dataflow/docs/guides/templates/provided-streaming#dlptexttobigquerystreaming [5] https://cloud.google.com/vision/
