RE: [E] COMMERCIAL BULK: Re: TensorFlow on Spark
Thanks Sean for your response. !! Want to add some more background here. I am using Spark3.0+ version with Tensorflow 2.0+. My use case is not for the image data but for the Time-series data where I am using LSTM and transformers to forecast. I evaluated SparkFlow and spark_tensorflow_distributor libraries, and there has been no major development recently on those libraries. I faced the issue of version dependencies on those and had a hard time fixing the library compatibilities. Hence a couple of below doubts:- * Does Horovod have any dependencies? * Any other library which is suitable for my use case.? * Any example code would really be of great help to understand. Thanks, Vijayant From: Sean Owen Sent: Wednesday, February 23, 2022 8:40 AM To: Vijayant Kumar Cc: user @spark Subject: [E] COMMERCIAL BULK: Re: TensorFlow on Spark Email is from a Free Mail Service (Gmail/Yahoo/Hotmail….) : Beware of Phishing Scams, Report questionable emails to s...@mavenir.com<mailto:s...@mavenir.com> Sure, Horovod is commonly used on Spark for this: https://horovod.readthedocs.io/en/stable/spark_include.html On Tue, Feb 22, 2022 at 8:51 PM Vijayant Kumar mailto:vijayant.ku...@mavenir.com.invalid>> wrote: Hi All, Anyone using Apache spark with TensorFlow for building models. My requirement is to use TensorFlow distributed model training across the Spark executors. Please help me with some resources or some sample code. Thanks, Vijayant This e-mail message may contain confidential or proprietary information of Mavenir Systems, Inc. or its affiliates and is intended solely for the use of the intended recipient(s). If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies in your control and contact us by e-mailing to secur...@mavenir.com<mailto:secur...@mavenir.com>. This message contains the views of its author and may not necessarily reflect the views of Mavenir Systems, Inc. or its affiliates, who employ systems to monitor email messages, but make no representation that such messages are authorized, secure, uncompromised, or free from computer viruses, malware, or other defects. Thank You This e-mail message may contain confidential or proprietary information of Mavenir Systems, Inc. or its affiliates and is intended solely for the use of the intended recipient(s). If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies in your control and contact us by e-mailing to secur...@mavenir.com. This message contains the views of its author and may not necessarily reflect the views of Mavenir Systems, Inc. or its affiliates, who employ systems to monitor email messages, but make no representation that such messages are authorized, secure, uncompromised, or free from computer viruses, malware, or other defects. Thank You
TensorFlow on Spark
Hi All, Anyone using Apache spark with TensorFlow for building models. My requirement is to use TensorFlow distributed model training across the Spark executors. Please help me with some resources or some sample code. Thanks, Vijayant This e-mail message may contain confidential or proprietary information of Mavenir Systems, Inc. or its affiliates and is intended solely for the use of the intended recipient(s). If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies in your control and contact us by e-mailing to secur...@mavenir.com. This message contains the views of its author and may not necessarily reflect the views of Mavenir Systems, Inc. or its affiliates, who employ systems to monitor email messages, but make no representation that such messages are authorized, secure, uncompromised, or free from computer viruses, malware, or other defects. Thank You
RE: [E] Re: Pyspark Kafka Structured Stream not working.
Hi Jungtek,
Thanks for the response. It appears to be #1.
I will appreciate if you can share some sample command to submit the Spark
application.?
From: Jungtaek Lim [mailto:kabhwan.opensou...@gmail.com]
Sent: Wednesday, May 06, 2020 8:24 PM
To: Vijayant Kumar
Cc: user@spark.apache.org
Subject: [E] Re: Pyspark Kafka Structured Stream not working.
[EXTERNAL EMAIL] DO NOT CLICK links or attachments unless you recognize the
sender and know the content is safe.
Hi,
1. You seem to use DStream (Spark Streaming), not Structured Streaming.
2. I'm not familiar with pyspark, but looks like the error message is very
clear - Kafka doesn't allow such name for "client.id<http://client.id>". The
error message guides the naming rule, so you may need to be adopted with such
convention. (e.g. no space)
Hope this helps,
Thanks,
Jungtaek Lim (HeartSaVioR)
On Wed, May 6, 2020 at 5:36 PM Vijayant Kumar
mailto:vijayant.ku...@mavenir.com.invalid>>
wrote:
Hi All,
I am getting the below error while using Pyspark Structured Streaming from
Kafka Producer.
20/05/06 11:51:16 ERROR ReceiverTracker: Deregistered receiver for stream 0:
Error starting receiver 0 - kafka.common.InvalidConfigException:
client.id<http://client.id> Python Kafka streamer is illegal, contains a
character other than ASCII alphanumerics, '.', '_' and '-'
I am using the below code to get the messages:
broker='vm105:2181'
topic='Hello-Kafka'
print 'broker topic is ',broker,topic
kvs = KafkaUtils.createStream(ssc, \
broker, \
"Python Kafka streamer",{topic:1})
And my Submit command is like below :-
spark-submit --jars spark-streaming-kafka-0-8-assembly_2.11-2.4.4.jar
test_kafka.py vm105:2181 Hello-Kafka
Can any one help me what am I missing. ?
Thanks,
Vijayant
This e-mail message may contain confidential or proprietary information of
Mavenir Systems, Inc. or its affiliates and is intended solely for the use of
the intended recipient(s). If you are not the intended recipient of this
message, you are hereby notified that any review, use or distribution of this
information is absolutely prohibited and we request that you delete all copies
in your control and contact us by e-mailing to
secur...@mavenir.com<mailto:secur...@mavenir.com>. This message contains the
views of its author and may not necessarily reflect the views of Mavenir
Systems, Inc. or its affiliates, who employ systems to monitor email messages,
but make no representation that such messages are authorized, secure,
uncompromised, or free from computer viruses, malware, or other defects. Thank
You
This e-mail message may contain confidential or proprietary information of
Mavenir Systems, Inc. or its affiliates and is intended solely for the use of
the intended recipient(s). If you are not the intended recipient of this
message, you are hereby notified that any review, use or distribution of this
information is absolutely prohibited and we request that you delete all copies
in your control and contact us by e-mailing to secur...@mavenir.com. This
message contains the views of its author and may not necessarily reflect the
views of Mavenir Systems, Inc. or its affiliates, who employ systems to monitor
email messages, but make no representation that such messages are authorized,
secure, uncompromised, or free from computer viruses, malware, or other
defects. Thank You
Pyspark Kafka Structured Stream not working.
Hi All,
I am getting the below error while using Pyspark Structured Streaming from
Kafka Producer.
20/05/06 11:51:16 ERROR ReceiverTracker: Deregistered receiver for stream 0:
Error starting receiver 0 - kafka.common.InvalidConfigException: client.id
Python Kafka streamer is illegal, contains a character other than ASCII
alphanumerics, '.', '_' and '-'
I am using the below code to get the messages:
broker='vm105:2181'
topic='Hello-Kafka'
print 'broker topic is ',broker,topic
kvs = KafkaUtils.createStream(ssc, \
broker, \
"Python Kafka streamer",{topic:1})
And my Submit command is like below :-
spark-submit --jars spark-streaming-kafka-0-8-assembly_2.11-2.4.4.jar
test_kafka.py vm105:2181 Hello-Kafka
Can any one help me what am I missing. ?
Thanks,
Vijayant
This e-mail message may contain confidential or proprietary information of
Mavenir Systems, Inc. or its affiliates and is intended solely for the use of
the intended recipient(s). If you are not the intended recipient of this
message, you are hereby notified that any review, use or distribution of this
information is absolutely prohibited and we request that you delete all copies
in your control and contact us by e-mailing to secur...@mavenir.com. This
message contains the views of its author and may not necessarily reflect the
views of Mavenir Systems, Inc. or its affiliates, who employ systems to monitor
email messages, but make no representation that such messages are authorized,
secure, uncompromised, or free from computer viruses, malware, or other
defects. Thank You
Spark Application Dynamic IP and Path assign.
Hi Team , I want to keep my IP and file path as configurable for my spark Application job. Can anyone help me with some sample code references.? I am using Spark 2.4.4 . Thanks, Vijayant This e-mail message may contain confidential or proprietary information of Mavenir Systems, Inc. or its affiliates and is intended solely for the use of the intended recipient(s). If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies in your control and contact us by e-mailing to secur...@mavenir.com. This message contains the views of its author and may not necessarily reflect the views of Mavenir Systems, Inc. or its affiliates, who employ systems to monitor email messages, but make no representation that such messages are authorized, secure, uncompromised, or free from computer viruses, malware, or other defects. Thank You
