damccorm commented on issue #35693: URL: https://github.com/apache/beam/issues/35693#issuecomment-3119503722
> CC [@chamikaramj](https://github.com/chamikaramj) > > The problem comes from here https://github.com/apache/beam/blob/master/sdks/python/apache_beam/yaml/yaml_ml.py#L36. We're only registering transforms that uses TFT, but not text embedding transforms. `SentenceTransformerEmbeddings` transform can be found here https://github.com/apache/beam/blob/master/sdks/python/apache_beam/ml/transforms/embeddings/huggingface.py#L110. > > Not sure if this will turn out to be an easy fix or not... This should be a pretty easy fix, we just need to import some more objects. Specifically, we should probably include `apache_beam.ml.transforms.embeddings.hugging_face.__all__`, `apache_beam.ml.transforms.embeddings.open_ai.__all__`, `apache_beam.ml.transforms.embeddings.tensorflow_hub.__all__`, and `apache_beam.ml.transforms.embeddings.vertex_ai.__all__` (maybe there's an easier way to iterate through all submodules of `apache_beam.ml.transforms.embeddings` as well) > Also a more general issue with MLTransform (related but not specific to this issue) is the underlying `tensorflow-transform` library that MLTransform uses cannot be installed on Apple Silicon [tensorflow/transform#298](https://github.com/tensorflow/transform/issues/298). Idk if this is mentioned somewhere on Beam's site/documentation? In general, tensorflow has lots of Apple Silicon issues unfortunately, there's not much we can do here -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
