I have been using tensorflow for a long time, it's not hard to implement a distributed training job at all, either by model parallelization or data parallelization. I don't think there is much need to develop spark to support tensorflow jobs. Just my thoughts...
On Thu, Feb 24, 2022 at 4:36 PM Gourav Sengupta <gourav.sengu...@gmail.com> wrote: > Hi, > > I do not think that there is any reason for using over engineered > platforms like Petastorm and Ray, except for certain use cases. > > What Ray is doing, except for certain use cases, could have been easily > done by SPARK, I think, had the open source community got that steer. But > maybe I am wrong and someone should be able to explain why the SPARK open > source community cannot develop the capabilities which are so natural to > almost all use cases of data processing in SPARK where the data gets > consumed by deep learning frameworks and we are asked to use Ray or > Petastorm? > > For those of us who are asking what does native integrations means please > try to compare delta between release 2.x and 3.x and koalas before 3.2 and > after 3.2. > > I am sure that the SPARK community can push for extending the dataframes > from SPARK to deep learning and other frameworks by natively integrating > them. > > > Regards, > Gourav Sengupta > > > On Wed, Feb 23, 2022 at 4:42 PM Dennis Suhari <d.suh...@icloud.com.invalid> > wrote: > >> Currently we are trying AnalyticsZoo and Ray >> >> >> Von meinem iPhone gesendet >> >> Am 23.02.2022 um 04:53 schrieb Bitfox <bit...@bitfox.top>: >> >> >> tensorflow itself can implement the distributed computing via a >> parameter server. Why did you want spark here? >> >> regards. >> >> On Wed, Feb 23, 2022 at 11:27 AM Vijayant Kumar >> <vijayant.ku...@mavenir.com.invalid> wrote: >> >>> Thanks Sean for your response. !! >>> >>> >>> >>> Want to add some more background here. >>> >>> >>> >>> I am using Spark3.0+ version with Tensorflow 2.0+. >>> >>> My use case is not for the image data but for the Time-series data where >>> I am using LSTM and transformers to forecast. >>> >>> >>> >>> I evaluated *SparkFlow* and *spark_tensorflow_distributor *libraries, and >>> there has been no major development recently on those libraries. I faced >>> the issue of version dependencies on those and had a hard time fixing the >>> library compatibilities. Hence a couple of below doubts:- >>> >>> >>> >>> - Does *Horovod* have any dependencies? >>> - Any other library which is suitable for my use case.? >>> - Any example code would really be of great help to understand. >>> >>> >>> >>> Thanks, >>> >>> Vijayant >>> >>> >>> >>> *From:* Sean Owen <sro...@gmail.com> >>> *Sent:* Wednesday, February 23, 2022 8:40 AM >>> *To:* Vijayant Kumar <vijayant.ku...@mavenir.com.invalid> >>> *Cc:* user @spark <user@spark.apache.org> >>> *Subject:* [E] COMMERCIAL BULK: Re: TensorFlow on Spark >>> >>> >>> >>> *Email is from a Free Mail Service (Gmail/Yahoo/Hotmail….) *: Beware of >>> Phishing Scams, Report questionable emails to s...@mavenir.com >>> >>> Sure, Horovod is commonly used on Spark for this: >>> >>> https://horovod.readthedocs.io/en/stable/spark_include.html >>> >>> >>> >>> On Tue, Feb 22, 2022 at 8:51 PM Vijayant Kumar < >>> vijayant.ku...@mavenir.com.invalid> wrote: >>> >>> Hi All, >>> >>> >>> >>> Anyone using Apache spark with TensorFlow for building models. My >>> requirement is to use TensorFlow distributed model training across the >>> Spark executors. >>> >>> Please help me with some resources or some sample code. >>> >>> >>> >>> Thanks, >>> >>> Vijayant >>> ------------------------------ >>> >>> This e-mail message may contain confidential or proprietary information >>> of Mavenir Systems, Inc. or its affiliates and is intended solely for the >>> use of the intended recipient(s). If you are not the intended recipient of >>> this message, you are hereby notified that any review, use or distribution >>> of this information is absolutely prohibited and we request that you delete >>> all copies in your control and contact us by e-mailing to >>> secur...@mavenir.com. This message contains the views of its author and >>> may not necessarily reflect the views of Mavenir Systems, Inc. or its >>> affiliates, who employ systems to monitor email messages, but make no >>> representation that such messages are authorized, secure, uncompromised, or >>> free from computer viruses, malware, or other defects. Thank You >>> >>> ------------------------------ >>> >>> This e-mail message may contain confidential or proprietary information >>> of Mavenir Systems, Inc. or its affiliates and is intended solely for the >>> use of the intended recipient(s). If you are not the intended recipient of >>> this message, you are hereby notified that any review, use or distribution >>> of this information is absolutely prohibited and we request that you delete >>> all copies in your control and contact us by e-mailing to >>> secur...@mavenir.com. This message contains the views of its author and >>> may not necessarily reflect the views of Mavenir Systems, Inc. or its >>> affiliates, who employ systems to monitor email messages, but make no >>> representation that such messages are authorized, secure, uncompromised, or >>> free from computer viruses, malware, or other defects. Thank You >>> >>
