[jira] [Commented] (PIG-5157) Upgrade to Spark 2.0

2017-07-16 Thread liyunzhang_intel (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089381#comment-16089381
 ] 

liyunzhang_intel commented on PIG-5157:
---

[~nkollar]: left comment on review board, just a small fix! meanwhile , please 
help review PIG-5246, thanks!

> Upgrade to Spark 2.0
> 
>
> Key: PIG-5157
> URL: https://issues.apache.org/jira/browse/PIG-5157
> Project: Pig
>  Issue Type: Improvement
>  Components: spark
>Reporter: Nandor Kollar
>Assignee: Nandor Kollar
> Fix For: 0.18.0
>
> Attachments: PIG-5157.patch, SkewedJoinInput1.txt, 
> SkewedJoinInput2.txt
>
>
> Upgrade to Spark 2.0 (or latest)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-07-16 Thread kelly zhang 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review180657
---



Fix it then Ship it


src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
Line 32 (original), 32 (patched)


There are two JobMetricsListener.


org.apache.pig.backend.hadoop.executionengine.spark.Spark1Shims.JobMetricsListener
org.apache.pig.backend.hadoop.executionengine.spark.JobMetricsListener

Can you rename to make it more understandable?


- kelly zhang


On June 28, 2017, 8:55 a.m., Nandor Kollar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> ---
> 
> (Updated June 28, 2017, 8:55 a.m.)
> 
> 
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
> 
> 
> Repository: pig-git
> 
> 
> Description
> ---
> 
> Upgrade to Spark 2.1 API using shims.
> 
> 
> Diffs
> -
> 
>   build.xml d274af9be3805f5167f11afed9d85f19e72916d8 
>   ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588 
>   ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
>  PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
>  f81341233447203abc4800cc7b22a4f419e10262 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
>  PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
> 237fd9431a16226234d91059088f91aab346b83c 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java 
> PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
>  83311dfa5bb25209a5366c2db7e8d483c31d94cd 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
>  382258e7ff9105aa397c5a2888df0c11e9562ec9 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
>  b58415e7e18ca4cf1331beef06e9214600a51424 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
>  130c8b9a747b176ce2b649ca6d5260527595fb76 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
>  fe1b54c8f128661d7d19c276d3bb2de7874d3086 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
>  adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
>  d1c43b1e06adc4c9fe45a83b8110402e3756 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
>  e003bbd95763b2d189ff9ec540c89abe52592420 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
>  00d29b44848546ed16dde2baa8c61b36939971b2 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
>  c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
>  baabfa090323e3bef087e259ce19df2e4c34dd63 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
>  3166fdc31745c013380492e089c83f3e853a3e6e 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
>  3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
>   src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
> c8cc03194b223d2ee181d73c6b651a6872cac6b6 
>   src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 
> 61ccbcc9fd723f6e2e578a8476230c42d5587dfe 
>   test/org/apache/pig/test/TestPigRunner.java 
> ec08417f2f71deec514cab5cfb9d2f99520ad641 
> 
> 
> Diff: https://reviews.apache.org/r/59530/diff/13/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Nandor Kollar
> 
>

[jira] Subscription: PIG patch available

2017-07-16 Thread jira 
Issue Subscription
Filter: PIG patch available (35 issues)

Subscriber: pigdaily

Key Summary
PIG-5268Review of org.apache.pig.backend.hadoop.datastorage.HDataStorage
https://issues-test.apache.org/jira/browse/PIG-5268
PIG-5267Review of org.apache.pig.impl.io.BufferedPositionedInputStream
https://issues-test.apache.org/jira/browse/PIG-5267
PIG-5264Remove deprecated keys from PigConfiguration
https://issues-test.apache.org/jira/browse/PIG-5264
PIG-5246Modify bin/pig about SPARK_HOME, SPARK_ASSEMBLY_JAR after upgrading 
spark to 2
https://issues-test.apache.org/jira/browse/PIG-5246
PIG-5160SchemaTupleFrontend.java is not thread safe, cause PigServer thrown 
NPE in multithread env
https://issues-test.apache.org/jira/browse/PIG-5160
PIG-5157Upgrade to Spark 2.0
https://issues-test.apache.org/jira/browse/PIG-5157
PIG-5115Builtin AvroStorage generates incorrect avro schema when the same 
pig field name appears in the alias
https://issues-test.apache.org/jira/browse/PIG-5115
PIG-5106Optimize when mapreduce.input.fileinputformat.input.dir.recursive 
set to true
https://issues-test.apache.org/jira/browse/PIG-5106
PIG-5081Can not run pig on spark source code distribution
https://issues-test.apache.org/jira/browse/PIG-5081
PIG-5080Support store alias as spark table
https://issues-test.apache.org/jira/browse/PIG-5080
PIG-5057IndexOutOfBoundsException when pig reducer processOnePackageOutput
https://issues-test.apache.org/jira/browse/PIG-5057
PIG-5029Optimize sort case when data is skewed
https://issues-test.apache.org/jira/browse/PIG-5029
PIG-4926Modify the content of start.xml for spark mode
https://issues-test.apache.org/jira/browse/PIG-4926
PIG-4913Reduce jython function initiation during compilation
https://issues-test.apache.org/jira/browse/PIG-4913
PIG-4849pig on tez will cause tez-ui to crash,because the content from 
timeline server is too long. 
https://issues-test.apache.org/jira/browse/PIG-4849
PIG-4750REPLACE_MULTI should compile Pattern once and reuse it
https://issues-test.apache.org/jira/browse/PIG-4750
PIG-4684Exception should be changed to warning when job diagnostics cannot 
be fetched
https://issues-test.apache.org/jira/browse/PIG-4684
PIG-4656Improve String serialization and comparator performance in 
BinInterSedes
https://issues-test.apache.org/jira/browse/PIG-4656
PIG-4598Allow user defined plan optimizer rules
https://issues-test.apache.org/jira/browse/PIG-4598
PIG-4551Partition filter is not pushed down in case of SPLIT
https://issues-test.apache.org/jira/browse/PIG-4551
PIG-4539New PigUnit
https://issues-test.apache.org/jira/browse/PIG-4539
PIG-4515org.apache.pig.builtin.Distinct throws ClassCastException
https://issues-test.apache.org/jira/browse/PIG-4515
PIG-4323PackageConverter hanging in Spark
https://issues-test.apache.org/jira/browse/PIG-4323
PIG-4313StackOverflowError in LIMIT operation on Spark
https://issues-test.apache.org/jira/browse/PIG-4313
PIG-4251Pig on Storm
https://issues-test.apache.org/jira/browse/PIG-4251
PIG-4002Disable combiner when map-side aggregation is used
https://issues-test.apache.org/jira/browse/PIG-4002
PIG-3952PigStorage accepts '-tagSplit' to return full split information
https://issues-test.apache.org/jira/browse/PIG-3952
PIG-3911Define unique fields with @OutputSchema
https://issues-test.apache.org/jira/browse/PIG-3911
PIG-3877Getting Geo Latitude/Longitude from Address Lines
https://issues-test.apache.org/jira/browse/PIG-3877
PIG-3873Geo distance calculation using Haversine
https://issues-test.apache.org/jira/browse/PIG-3873
PIG-3864ToDate(userstring, format, timezone) computes DateTime with strange 
handling of Daylight Saving Time with location based timezones
https://issues-test.apache.org/jira/browse/PIG-3864
PIG-3668COR built-in function when atleast one of the coefficient values is 
NaN
https://issues-test.apache.org/jira/browse/PIG-3668
PIG-3655BinStorage and InterStorage approach to record markers is broken
https://issues-test.apache.org/jira/browse/PIG-3655
PIG-3587add functionality for rolling over dates
https://issues-test.apache.org/jira/browse/PIG-3587
PIG-1804Alow Jython function to implement Algebraic and/or Accumulator 
interfaces
https://issues-test.apache.org/jira/browse/PIG-1804

You may edit this subscription at:
https://issues-test.apache.org/jira/secure/FilterSubscription!default.jspa?subId=16328&filterId=12322384

[jira] Subscription: PIG patch available

2017-07-16 Thread jira 
Issue Subscription
Filter: PIG patch available (35 issues)

Subscriber: pigdaily

Key Summary
PIG-5268Review of org.apache.pig.backend.hadoop.datastorage.HDataStorage
https://issues.apache.org/jira/browse/PIG-5268
PIG-5267Review of org.apache.pig.impl.io.BufferedPositionedInputStream
https://issues.apache.org/jira/browse/PIG-5267
PIG-5264Remove deprecated keys from PigConfiguration
https://issues.apache.org/jira/browse/PIG-5264
PIG-5246Modify bin/pig about SPARK_HOME, SPARK_ASSEMBLY_JAR after upgrading 
spark to 2
https://issues.apache.org/jira/browse/PIG-5246
PIG-5160SchemaTupleFrontend.java is not thread safe, cause PigServer thrown 
NPE in multithread env
https://issues.apache.org/jira/browse/PIG-5160
PIG-5157Upgrade to Spark 2.0
https://issues.apache.org/jira/browse/PIG-5157
PIG-5115Builtin AvroStorage generates incorrect avro schema when the same 
pig field name appears in the alias
https://issues.apache.org/jira/browse/PIG-5115
PIG-5106Optimize when mapreduce.input.fileinputformat.input.dir.recursive 
set to true
https://issues.apache.org/jira/browse/PIG-5106
PIG-5081Can not run pig on spark source code distribution
https://issues.apache.org/jira/browse/PIG-5081
PIG-5080Support store alias as spark table
https://issues.apache.org/jira/browse/PIG-5080
PIG-5057IndexOutOfBoundsException when pig reducer processOnePackageOutput
https://issues.apache.org/jira/browse/PIG-5057
PIG-5029Optimize sort case when data is skewed
https://issues.apache.org/jira/browse/PIG-5029
PIG-4926Modify the content of start.xml for spark mode
https://issues.apache.org/jira/browse/PIG-4926
PIG-4913Reduce jython function initiation during compilation
https://issues.apache.org/jira/browse/PIG-4913
PIG-4849pig on tez will cause tez-ui to crash,because the content from 
timeline server is too long. 
https://issues.apache.org/jira/browse/PIG-4849
PIG-4750REPLACE_MULTI should compile Pattern once and reuse it
https://issues.apache.org/jira/browse/PIG-4750
PIG-4684Exception should be changed to warning when job diagnostics cannot 
be fetched
https://issues.apache.org/jira/browse/PIG-4684
PIG-4656Improve String serialization and comparator performance in 
BinInterSedes
https://issues.apache.org/jira/browse/PIG-4656
PIG-4598Allow user defined plan optimizer rules
https://issues.apache.org/jira/browse/PIG-4598
PIG-4551Partition filter is not pushed down in case of SPLIT
https://issues.apache.org/jira/browse/PIG-4551
PIG-4539New PigUnit
https://issues.apache.org/jira/browse/PIG-4539
PIG-4515org.apache.pig.builtin.Distinct throws ClassCastException
https://issues.apache.org/jira/browse/PIG-4515
PIG-4323PackageConverter hanging in Spark
https://issues.apache.org/jira/browse/PIG-4323
PIG-4313StackOverflowError in LIMIT operation on Spark
https://issues.apache.org/jira/browse/PIG-4313
PIG-4251Pig on Storm
https://issues.apache.org/jira/browse/PIG-4251
PIG-4002Disable combiner when map-side aggregation is used
https://issues.apache.org/jira/browse/PIG-4002
PIG-3952PigStorage accepts '-tagSplit' to return full split information
https://issues.apache.org/jira/browse/PIG-3952
PIG-3911Define unique fields with @OutputSchema
https://issues.apache.org/jira/browse/PIG-3911
PIG-3877Getting Geo Latitude/Longitude from Address Lines
https://issues.apache.org/jira/browse/PIG-3877
PIG-3873Geo distance calculation using Haversine
https://issues.apache.org/jira/browse/PIG-3873
PIG-3864ToDate(userstring, format, timezone) computes DateTime with strange 
handling of Daylight Saving Time with location based timezones
https://issues.apache.org/jira/browse/PIG-3864
PIG-3668COR built-in function when atleast one of the coefficient values is 
NaN
https://issues.apache.org/jira/browse/PIG-3668
PIG-3655BinStorage and InterStorage approach to record markers is broken
https://issues.apache.org/jira/browse/PIG-3655
PIG-3587add functionality for rolling over dates
https://issues.apache.org/jira/browse/PIG-3587
PIG-1804Alow Jython function to implement Algebraic and/or Accumulator 
interfaces
https://issues.apache.org/jira/browse/PIG-1804

You may edit this subscription at:
https://issues.apache.org/jira/secure/FilterSubscription!default.jspa?subId=16328&filterId=12322384

[jira] [Updated] (PIG-5080) Support store alias as spark table

2017-07-16 Thread Jeff Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-5080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Zhang updated PIG-5080:

Status: Patch Available  (was: Open)

> Support store alias as spark table
> --
>
> Key: PIG-5080
> URL: https://issues.apache.org/jira/browse/PIG-5080
> Project: Pig
>  Issue Type: New Feature
>  Components: spark
>Affects Versions: 0.16.0
>Reporter: Jeff Zhang
>Assignee: Jeff Zhang
> Fix For: 0.17.1
>
> Attachments: PIG-5080-1.patch, PIG-5080-2.patch
>
>
> The purpose is that I'd like to take advantage of both pig and hive. 
> Pig-latin has powerful data flow expression ability which is useful for ETL 
> while hive is good at query. 
> The scenario is that I'd like to store pig alias as spark temporary table 
> (cache can be optional). And I have an another spark engine which share the 
> same SparkContext (in the same JVM) to query the table.
> Please close this ticket if it is already supported. I didn't go through all 
> the features of pig-spark.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PIG-5080) Support store alias as spark table

2017-07-16 Thread Jeff Zhang (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-5080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089237#comment-16089237
 ] 

Jeff Zhang commented on PIG-5080:
-

[~kellyzly] [~nkollar] Mind to review this ? Thanks

> Support store alias as spark table
> --
>
> Key: PIG-5080
> URL: https://issues.apache.org/jira/browse/PIG-5080
> Project: Pig
>  Issue Type: New Feature
>  Components: spark
>Affects Versions: 0.16.0
>Reporter: Jeff Zhang
>Assignee: Jeff Zhang
> Fix For: 0.17.1
>
> Attachments: PIG-5080-1.patch, PIG-5080-2.patch
>
>
> The purpose is that I'd like to take advantage of both pig and hive. 
> Pig-latin has powerful data flow expression ability which is useful for ETL 
> while hive is good at query. 
> The scenario is that I'd like to store pig alias as spark temporary table 
> (cache can be optional). And I have an another spark engine which share the 
> same SparkContext (in the same JVM) to query the table.
> Please close this ticket if it is already supported. I didn't go through all 
> the features of pig-spark.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PIG-5080) Support store alias as spark table

2017-07-16 Thread Jeff Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-5080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Zhang updated PIG-5080:

Status: Open  (was: Patch Available)

> Support store alias as spark table
> --
>
> Key: PIG-5080
> URL: https://issues.apache.org/jira/browse/PIG-5080
> Project: Pig
>  Issue Type: New Feature
>  Components: spark
>Affects Versions: 0.16.0
>Reporter: Jeff Zhang
>Assignee: Jeff Zhang
> Attachments: PIG-5080-1.patch, PIG-5080-2.patch
>
>
> The purpose is that I'd like to take advantage of both pig and hive. 
> Pig-latin has powerful data flow expression ability which is useful for ETL 
> while hive is good at query. 
> The scenario is that I'd like to store pig alias as spark temporary table 
> (cache can be optional). And I have an another spark engine which share the 
> same SparkContext (in the same JVM) to query the table.
> Please close this ticket if it is already supported. I didn't go through all 
> the features of pig-spark.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PIG-5080) Support store alias as spark table

2017-07-16 Thread Jeff Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-5080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Zhang updated PIG-5080:

Fix Version/s: 0.17.1

> Support store alias as spark table
> --
>
> Key: PIG-5080
> URL: https://issues.apache.org/jira/browse/PIG-5080
> Project: Pig
>  Issue Type: New Feature
>  Components: spark
>Affects Versions: 0.16.0
>Reporter: Jeff Zhang
>Assignee: Jeff Zhang
> Fix For: 0.17.1
>
> Attachments: PIG-5080-1.patch, PIG-5080-2.patch
>
>
> The purpose is that I'd like to take advantage of both pig and hive. 
> Pig-latin has powerful data flow expression ability which is useful for ETL 
> while hive is good at query. 
> The scenario is that I'd like to store pig alias as spark temporary table 
> (cache can be optional). And I have an another spark engine which share the 
> same SparkContext (in the same JVM) to query the table.
> Please close this ticket if it is already supported. I didn't go through all 
> the features of pig-spark.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PIG-5080) Support store alias as spark table

2017-07-16 Thread Jeff Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-5080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Zhang updated PIG-5080:

Attachment: PIG-5080-2.patch

> Support store alias as spark table
> --
>
> Key: PIG-5080
> URL: https://issues.apache.org/jira/browse/PIG-5080
> Project: Pig
>  Issue Type: New Feature
>  Components: spark
>Affects Versions: 0.16.0
>Reporter: Jeff Zhang
>Assignee: Jeff Zhang
> Fix For: 0.17.1
>
> Attachments: PIG-5080-1.patch, PIG-5080-2.patch
>
>
> The purpose is that I'd like to take advantage of both pig and hive. 
> Pig-latin has powerful data flow expression ability which is useful for ETL 
> while hive is good at query. 
> The scenario is that I'd like to store pig alias as spark temporary table 
> (cache can be optional). And I have an another spark engine which share the 
> same SparkContext (in the same JVM) to query the table.
> Please close this ticket if it is already supported. I didn't go through all 
> the features of pig-spark.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PIG-5246) Modify bin/pig about SPARK_HOME, SPARK_ASSEMBLY_JAR after upgrading spark to 2

2017-07-16 Thread liyunzhang_intel (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089217#comment-16089217
 ] 

liyunzhang_intel commented on PIG-5246:
---

[~nkollar]: the problem about basic script passed on yarn-client mode. now 
running all unit tests in local mode. 

> Modify bin/pig about SPARK_HOME, SPARK_ASSEMBLY_JAR after upgrading spark to 2
> --
>
> Key: PIG-5246
> URL: https://issues.apache.org/jira/browse/PIG-5246
> Project: Pig
>  Issue Type: Bug
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HBase9498.patch, PIG-5246.1.patch, PIG-5246_2.patch, 
> PIG-5246.3.patch, PIG-5246_4.patch, PIG-5246.patch
>
>
> in bin/pig.
> we copy assembly jar to pig's classpath in spark1.6.
> {code}
> # For spark mode:
> # Please specify SPARK_HOME first so that we can locate 
> $SPARK_HOME/lib/spark-assembly*.jar,
> # we will add spark-assembly*.jar to the classpath.
> if [ "$isSparkMode"  == "true" ]; then
> if [ -z "$SPARK_HOME" ]; then
>echo "Error: SPARK_HOME is not set!"
>exit 1
> fi
> # Please specify SPARK_JAR which is the hdfs path of spark-assembly*.jar 
> to allow YARN to cache spark-assembly*.jar on nodes so that it doesn't need 
> to be distributed each time an application runs.
> if [ -z "$SPARK_JAR" ]; then
>echo "Error: SPARK_JAR is not set, SPARK_JAR stands for the hdfs 
> location of spark-assembly*.jar. This allows YARN to cache 
> spark-assembly*.jar on nodes so that it doesn't need to be distributed each 
> time an application runs."
>exit 1
> fi
> if [ -n "$SPARK_HOME" ]; then
> echo "Using Spark Home: " ${SPARK_HOME}
> SPARK_ASSEMBLY_JAR=`ls ${SPARK_HOME}/lib/spark-assembly*`
> CLASSPATH=${CLASSPATH}:$SPARK_ASSEMBLY_JAR
> fi
> fi
> {code}
> after upgrade to spark2.0, we may modify it



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)