On Fri, Apr 17, 2020 at 2:32 PM Ahmet Altay <al...@google.com> wrote:
> Hi Holden, nice to hear from you. Thanks a lot for this email. Adding some > TFX folks as well. +Robert Crowe <robertcr...@google.com> +Irene > Giannoumis <iren...@google.com> +Zhitao Li <zhita...@google.com> +Anusha > Ramesh <anusharam...@google.com> > > Would it be possible for TFX folks to review the TFX section of your book? > Sure. Currently we only cover TFT and TFDV and I can share the draft of that chapter with TFX folks but we might cover more later. > > On Fri, Apr 17, 2020 at 12:27 PM Kyle Weaver <kcwea...@google.com> wrote: > >> Hi Holden, >> >> The note on Flink & Spark support sounds reasonable to me. I am >> optimistic about getting Flink + TFX + Kubeflow working fairly soon, but I >> agree that we don't want to over-promise. >> >> I'm not so sure about the status of Dataflow here, perhaps someone else >> can comment on that. >> > > I believe TFX/KFP works on Dataflow with the same pipeline. (They have an > example on this > https://github.com/tensorflow/tfx/blob/master/docs/tutorials/tfx/template.ipynb > - > step 8) > > So that is only the TFX pipeline, if you want to use Kubeflow pipelines with the TFX components that’s not supported. > >> Looking forward to the book :) >> >> Kyle >> >> On Fri, Apr 17, 2020 at 1:14 PM Holden Karau <hol...@pigscanfly.ca> >> wrote: >> >>> Hi Apache Beam Developers, >>> >>> I'm working on a book about Kubeflow, which naturally has a section on >>> TFX. I want to set users expectations correctly so I wanted to know what >>> y'all thought of this NOTE we were thinking of including in the early >>> release: >>> >>> Apache Beam’s Python support outside of Google cloud's Dataflow is >>> relatively new. TFX is a Python tool, so scaling it depends on Apache >>> Beam's Python support. You can scale your job by using the non-portable >>> dataflow component, but this requires changing your pipeline code and isn't >>> supported by Kubeflow's current TFX components. As Apache Beam's support >>> for Apache Flink & Spark improves support may be added for scaling the TFX >>> components in a portable manner. >>> >>> Does this sound reasonable to folks? I don't want to over-promise but I >>> also don't want to scare people away given all of the progress that is >>> being made in supporting the open-source runners with language portability. >>> >>> Cheers, >>> >>> Holden :) >>> >>> -- >>> Twitter: https://twitter.com/holdenkarau >>> Books (Learning Spark, High Performance Spark, etc.): >>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>> >> -- Twitter: https://twitter.com/holdenkarau Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
