spark git commit: [SPARK-9812] [STREAMING] Fix Python 3 compatibility issue in PySpark Streaming and some docs

2015-08-19 Thread tdas 
Repository: spark
Updated Branches:
  refs/heads/master 2f2686a73 -> 1f29d502e


[SPARK-9812] [STREAMING] Fix Python 3 compatibility issue in PySpark Streaming 
and some docs

This PR includes the following fixes:
1. Use `range` instead of `xrange` in `queue_stream.py` to support Python 3.
2. Fix the issue that `utf8_decoder` will return `bytes` rather than `str` when 
receiving an empty `bytes` in Python 3.
3. Fix the commands in docs so that the user can copy them directly to the 
command line. The previous commands was broken in the middle of a path, so when 
copying to the command line, the path would be split to two parts by the extra 
spaces, which forces the user to fix it manually.

Author: zsxwing 

Closes #8315 from zsxwing/SPARK-9812.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/1f29d502
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/1f29d502
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/1f29d502

Branch: refs/heads/master
Commit: 1f29d502e7ecd6faa185d70dc714f9ea3922fb6d
Parents: 2f2686a
Author: zsxwing 
Authored: Wed Aug 19 18:36:01 2015 -0700
Committer: Tathagata Das 
Committed: Wed Aug 19 18:36:01 2015 -0700

--
 examples/src/main/python/streaming/direct_kafka_wordcount.py | 6 +++---
 examples/src/main/python/streaming/flume_wordcount.py| 5 +++--
 examples/src/main/python/streaming/kafka_wordcount.py| 5 +++--
 examples/src/main/python/streaming/mqtt_wordcount.py | 5 +++--
 examples/src/main/python/streaming/queue_stream.py   | 4 ++--
 python/pyspark/streaming/flume.py| 4 +++-
 python/pyspark/streaming/kafka.py| 4 +++-
 python/pyspark/streaming/kinesis.py  | 4 +++-
 8 files changed, 23 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/1f29d502/examples/src/main/python/streaming/direct_kafka_wordcount.py
--
diff --git a/examples/src/main/python/streaming/direct_kafka_wordcount.py 
b/examples/src/main/python/streaming/direct_kafka_wordcount.py
index 6ef188a..ea20678 100644
--- a/examples/src/main/python/streaming/direct_kafka_wordcount.py
+++ b/examples/src/main/python/streaming/direct_kafka_wordcount.py
@@ -23,8 +23,8 @@
  http://kafka.apache.org/documentation.html#quickstart
 
  and then run the example
-`$ bin/spark-submit --jars external/kafka-assembly/target/scala-*/\
-  spark-streaming-kafka-assembly-*.jar \
+`$ bin/spark-submit --jars \
+  
external/kafka-assembly/target/scala-*/spark-streaming-kafka-assembly-*.jar \
   examples/src/main/python/streaming/direct_kafka_wordcount.py \
   localhost:9092 test`
 """
@@ -37,7 +37,7 @@ from pyspark.streaming.kafka import KafkaUtils
 
 if __name__ == "__main__":
 if len(sys.argv) != 3:
-print >> sys.stderr, "Usage: direct_kafka_wordcount.py  
"
+print("Usage: direct_kafka_wordcount.py  ", 
file=sys.stderr)
 exit(-1)
 
 sc = SparkContext(appName="PythonStreamingDirectKafkaWordCount")

http://git-wip-us.apache.org/repos/asf/spark/blob/1f29d502/examples/src/main/python/streaming/flume_wordcount.py
--
diff --git a/examples/src/main/python/streaming/flume_wordcount.py 
b/examples/src/main/python/streaming/flume_wordcount.py
index 091b64d..d75bc6d 100644
--- a/examples/src/main/python/streaming/flume_wordcount.py
+++ b/examples/src/main/python/streaming/flume_wordcount.py
@@ -23,8 +23,9 @@
  https://flume.apache.org/documentation.html
 
  and then run the example
-`$ bin/spark-submit --jars external/flume-assembly/target/scala-*/\
-  spark-streaming-flume-assembly-*.jar 
examples/src/main/python/streaming/flume_wordcount.py \
+`$ bin/spark-submit --jars \
+  
external/flume-assembly/target/scala-*/spark-streaming-flume-assembly-*.jar \
+  examples/src/main/python/streaming/flume_wordcount.py \
   localhost 12345
 """
 from __future__ import print_function

http://git-wip-us.apache.org/repos/asf/spark/blob/1f29d502/examples/src/main/python/streaming/kafka_wordcount.py
--
diff --git a/examples/src/main/python/streaming/kafka_wordcount.py 
b/examples/src/main/python/streaming/kafka_wordcount.py
index b178e78..8d697f6 100644
--- a/examples/src/main/python/streaming/kafka_wordcount.py
+++ b/examples/src/main/python/streaming/kafka_wordcount.py
@@ -23,8 +23,9 @@
  http://kafka.apache.org/documentation.html#quickstart
 
  and then run the example
-`$ bin/spark-submit --jars external/kafka-assembly/target/scala-*/\
-  spark-streaming-kafka-assembly-*.jar 
examples/src/main/python/streaming/kafka_wordcou

spark git commit: [SPARK-9812] [STREAMING] Fix Python 3 compatibility issue in PySpark Streaming and some docs

2015-08-19 Thread tdas 
Repository: spark
Updated Branches:
  refs/heads/branch-1.5 321cb99ca -> 16414dae0


[SPARK-9812] [STREAMING] Fix Python 3 compatibility issue in PySpark Streaming 
and some docs

This PR includes the following fixes:
1. Use `range` instead of `xrange` in `queue_stream.py` to support Python 3.
2. Fix the issue that `utf8_decoder` will return `bytes` rather than `str` when 
receiving an empty `bytes` in Python 3.
3. Fix the commands in docs so that the user can copy them directly to the 
command line. The previous commands was broken in the middle of a path, so when 
copying to the command line, the path would be split to two parts by the extra 
spaces, which forces the user to fix it manually.

Author: zsxwing 

Closes #8315 from zsxwing/SPARK-9812.

(cherry picked from commit 1f29d502e7ecd6faa185d70dc714f9ea3922fb6d)
Signed-off-by: Tathagata Das 


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/16414dae
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/16414dae
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/16414dae

Branch: refs/heads/branch-1.5
Commit: 16414dae03b427506b2a1ebb7d405e6fa3bdad17
Parents: 321cb99
Author: zsxwing 
Authored: Wed Aug 19 18:36:01 2015 -0700
Committer: Tathagata Das 
Committed: Wed Aug 19 18:36:10 2015 -0700

--
 examples/src/main/python/streaming/direct_kafka_wordcount.py | 6 +++---
 examples/src/main/python/streaming/flume_wordcount.py| 5 +++--
 examples/src/main/python/streaming/kafka_wordcount.py| 5 +++--
 examples/src/main/python/streaming/mqtt_wordcount.py | 5 +++--
 examples/src/main/python/streaming/queue_stream.py   | 4 ++--
 python/pyspark/streaming/flume.py| 4 +++-
 python/pyspark/streaming/kafka.py| 4 +++-
 python/pyspark/streaming/kinesis.py  | 4 +++-
 8 files changed, 23 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/16414dae/examples/src/main/python/streaming/direct_kafka_wordcount.py
--
diff --git a/examples/src/main/python/streaming/direct_kafka_wordcount.py 
b/examples/src/main/python/streaming/direct_kafka_wordcount.py
index 6ef188a..ea20678 100644
--- a/examples/src/main/python/streaming/direct_kafka_wordcount.py
+++ b/examples/src/main/python/streaming/direct_kafka_wordcount.py
@@ -23,8 +23,8 @@
  http://kafka.apache.org/documentation.html#quickstart
 
  and then run the example
-`$ bin/spark-submit --jars external/kafka-assembly/target/scala-*/\
-  spark-streaming-kafka-assembly-*.jar \
+`$ bin/spark-submit --jars \
+  
external/kafka-assembly/target/scala-*/spark-streaming-kafka-assembly-*.jar \
   examples/src/main/python/streaming/direct_kafka_wordcount.py \
   localhost:9092 test`
 """
@@ -37,7 +37,7 @@ from pyspark.streaming.kafka import KafkaUtils
 
 if __name__ == "__main__":
 if len(sys.argv) != 3:
-print >> sys.stderr, "Usage: direct_kafka_wordcount.py  
"
+print("Usage: direct_kafka_wordcount.py  ", 
file=sys.stderr)
 exit(-1)
 
 sc = SparkContext(appName="PythonStreamingDirectKafkaWordCount")

http://git-wip-us.apache.org/repos/asf/spark/blob/16414dae/examples/src/main/python/streaming/flume_wordcount.py
--
diff --git a/examples/src/main/python/streaming/flume_wordcount.py 
b/examples/src/main/python/streaming/flume_wordcount.py
index 091b64d..d75bc6d 100644
--- a/examples/src/main/python/streaming/flume_wordcount.py
+++ b/examples/src/main/python/streaming/flume_wordcount.py
@@ -23,8 +23,9 @@
  https://flume.apache.org/documentation.html
 
  and then run the example
-`$ bin/spark-submit --jars external/flume-assembly/target/scala-*/\
-  spark-streaming-flume-assembly-*.jar 
examples/src/main/python/streaming/flume_wordcount.py \
+`$ bin/spark-submit --jars \
+  
external/flume-assembly/target/scala-*/spark-streaming-flume-assembly-*.jar \
+  examples/src/main/python/streaming/flume_wordcount.py \
   localhost 12345
 """
 from __future__ import print_function

http://git-wip-us.apache.org/repos/asf/spark/blob/16414dae/examples/src/main/python/streaming/kafka_wordcount.py
--
diff --git a/examples/src/main/python/streaming/kafka_wordcount.py 
b/examples/src/main/python/streaming/kafka_wordcount.py
index b178e78..8d697f6 100644
--- a/examples/src/main/python/streaming/kafka_wordcount.py
+++ b/examples/src/main/python/streaming/kafka_wordcount.py
@@ -23,8 +23,9 @@
  http://kafka.apache.org/documentation.html#quickstart
 
  and then run the example
-`$ bin/spark-submit --jars external/kafka-assembly/ta