[jira] [Updated] (SPARK-21126) The configuration which named "spark.core.connection.auth.wait.timeout" hasn't been used in spark

[Sean Owen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Owen updated SPARK-21126:
--
   Priority: Trivial  (was: Major)
Component/s: Documentation

[~liuzhaokun] please fill out JIRAs more carefully. This can't be a "Major Bug".

> The configuration which named "spark.core.connection.auth.wait.timeout" 
> hasn't been used in spark
> -
>
> Key: SPARK-21126
> URL: https://issues.apache.org/jira/browse/SPARK-21126
> Project: Spark
>  Issue Type: Bug
>  Components: Documentation, Spark Core
>Affects Versions: 2.1.1
>Reporter: liuzhaokun
>Priority: Trivial
>
> The configuration which named "spark.core.connection.auth.wait.timeout" 
> hasn't been used in spark,so I think it should be removed from 
> configuration.md.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-19490) Hive partition columns are case-sensitive

[cen yuhai (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-19490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16052633#comment-16052633
 ] 

cen yuhai commented on SPARK-19490:
---

I don't know, what is your spark version?

> Hive partition columns are case-sensitive
> -
>
> Key: SPARK-19490
> URL: https://issues.apache.org/jira/browse/SPARK-19490
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.1.0
>Reporter: cen yuhai
>
> The real partitions columns are lower case (year, month, day)
> {code}
> Caused by: java.lang.RuntimeException: Expected only partition pruning 
> predicates: (concat(YEAR#22, MONTH#23, DAY#24) = 20170202)
>   at scala.sys.package$.error(package.scala:27)
>   at 
> org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$listPartitionsByFilter$1.apply(HiveExternalCatalog.scala:985)
>   at 
> org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$listPartitionsByFilter$1.apply(HiveExternalCatalog.scala:976)
>   at 
> org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:95)
>   at 
> org.apache.spark.sql.hive.HiveExternalCatalog.listPartitionsByFilter(HiveExternalCatalog.scala:976)
>   at 
> org.apache.spark.sql.hive.MetastoreRelation.getHiveQlPartitions(MetastoreRelation.scala:161)
>   at 
> org.apache.spark.sql.hive.execution.HiveTableScanExec$$anonfun$10.apply(HiveTableScanExec.scala:151)
>   at 
> org.apache.spark.sql.hive.execution.HiveTableScanExec$$anonfun$10.apply(HiveTableScanExec.scala:150)
>   at org.apache.spark.util.Utils$.withDummyCallSite(Utils.scala:2472)
>   at 
> org.apache.spark.sql.hive.execution.HiveTableScanExec.doExecute(HiveTableScanExec.scala:149)
>   at 
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
>   at 
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
>   at 
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135)
>   at 
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>   at 
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)
>   at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)
>   at 
> org.apache.spark.sql.execution.InputAdapter.inputRDDs(WholeStageCodegenExec.scala:235)
>   at 
> org.apache.spark.sql.execution.FilterExec.inputRDDs(basicPhysicalOperators.scala:124)
>   at 
> org.apache.spark.sql.execution.ProjectExec.inputRDDs(basicPhysicalOperators.scala:42)
>   at 
> org.apache.spark.sql.execution.aggregate.HashAggregateExec.inputRDDs(HashAggregateExec.scala:141)
>   at 
> org.apache.spark.sql.execution.WholeStageCodegenExec.doExecute(WholeStageCodegenExec.scala:368)
>   at 
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
>   at 
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
>   at 
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135)
>   at 
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>   at 
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)
>   at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)
>   at 
> org.apache.spark.sql.execution.exchange.ShuffleExchange.prepareShuffleDependency(ShuffleExchange.scala:85)
>   at 
> org.apache.spark.sql.execution.exchange.ExchangeCoordinator.doEstimationIfNecessary(ExchangeCoordinator.scala:213)
>   at 
> org.apache.spark.sql.execution.exchange.ExchangeCoordinator.postShuffleRDD(ExchangeCoordinator.scala:261)
>   at 
> org.apache.spark.sql.execution.exchange.ShuffleExchange$$anonfun$doExecute$1.apply(ShuffleExchange.scala:117)
>   at 
> org.apache.spark.sql.execution.exchange.ShuffleExchange$$anonfun$doExecute$1.apply(ShuffleExchange.scala:112)
>   at 
> org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:52)
> {code}
> Use these sql can reproduce this bug:
> CREATE TABLE partition_test (key Int) partitioned by (date string)
> SELECT * FROM partition_test where DATE = '20170101'



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-21126) The configuration which named "spark.core.connection.auth.wait.timeout" hasn't been used in spark

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-21126:


Assignee: (was: Apache Spark)

> The configuration which named "spark.core.connection.auth.wait.timeout" 
> hasn't been used in spark
> -
>
> Key: SPARK-21126
> URL: https://issues.apache.org/jira/browse/SPARK-21126
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Affects Versions: 2.1.1
>Reporter: liuzhaokun
>
> The configuration which named "spark.core.connection.auth.wait.timeout" 
> hasn't been used in spark,so I think it should be removed from 
> configuration.md.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-21126) The configuration which named "spark.core.connection.auth.wait.timeout" hasn't been used in spark

[Apache Spark (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-21126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16052626#comment-16052626
 ] 

Apache Spark commented on SPARK-21126:
--

User 'liu-zhaokun' has created a pull request for this issue:
https://github.com/apache/spark/pull/18333

> The configuration which named "spark.core.connection.auth.wait.timeout" 
> hasn't been used in spark
> -
>
> Key: SPARK-21126
> URL: https://issues.apache.org/jira/browse/SPARK-21126
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Affects Versions: 2.1.1
>Reporter: liuzhaokun
>
> The configuration which named "spark.core.connection.auth.wait.timeout" 
> hasn't been used in spark,so I think it should be removed from 
> configuration.md.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-21126) The configuration which named "spark.core.connection.auth.wait.timeout" hasn't been used in spark

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-21126:


Assignee: Apache Spark

> The configuration which named "spark.core.connection.auth.wait.timeout" 
> hasn't been used in spark
> -
>
> Key: SPARK-21126
> URL: https://issues.apache.org/jira/browse/SPARK-21126
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Affects Versions: 2.1.1
>Reporter: liuzhaokun
>Assignee: Apache Spark
>
> The configuration which named "spark.core.connection.auth.wait.timeout" 
> hasn't been used in spark,so I think it should be removed from 
> configuration.md.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-21126) The configuration which named "spark.core.connection.auth.wait.timeout" hasn't been used in spark

[liuzhaokun (JIRA)]

liuzhaokun created SPARK-21126:
--

 Summary: The configuration which named 
"spark.core.connection.auth.wait.timeout" hasn't been used in spark
 Key: SPARK-21126
 URL: https://issues.apache.org/jira/browse/SPARK-21126
 Project: Spark
  Issue Type: Bug
  Components: Spark Core
Affects Versions: 2.1.1
Reporter: liuzhaokun


The configuration which named "spark.core.connection.auth.wait.timeout" hasn't 
been used in spark,so I think it should be removed from configuration.md.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Resolved] (SPARK-2971) Orphaned YARN ApplicationMaster lingers forever

[Marcelo Vanzin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-2971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcelo Vanzin resolved SPARK-2971.
---
Resolution: Not A Problem

Pretty sure this has been fixed some point after 1.0; there's both the code 
above and also there's a timeout in {{ApplicationMaster.waitForSparkDriver}}.

> Orphaned YARN ApplicationMaster lingers forever
> ---
>
> Key: SPARK-2971
> URL: https://issues.apache.org/jira/browse/SPARK-2971
> Project: Spark
>  Issue Type: Bug
>  Components: YARN
>Affects Versions: 1.0.2
> Environment: Python yarn client mode, Cloudera 5.1.0 on Ubuntu precise
>Reporter: Shay Rojansky
>
> We have cases where if CTRL-C is hit during a Spark job startup, a YARN 
> ApplicationMaster is created but cannot connect to the driver (presumably 
> because the driver has terminated). Once an AM enters this state it never 
> exits it, and has to be manually killed in YARN.
> Here's an excerpt from the AM logs:
> {noformat}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/yarn/nm/usercache/roji/filecache/40/spark-assembly-1.0.2-hadoop2.2.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 14/08/11 16:29:39 WARN NativeCodeLoader: Unable to load native-hadoop library 
> for your platform... using builtin-java classes where applicable
> 14/08/11 16:29:39 INFO SecurityManager: Changing view acls to: roji
> 14/08/11 16:29:39 INFO SecurityManager: SecurityManager: authentication 
> disabled; ui acls disabled; users with view permissions: Set(roji)
> 14/08/11 16:29:40 INFO Slf4jLogger: Slf4jLogger started
> 14/08/11 16:29:40 INFO Remoting: Starting remoting
> 14/08/11 16:29:40 INFO Remoting: Remoting started; listening on addresses 
> :[akka.tcp://sparkyar...@g024.grid.eaglerd.local:34075]
> 14/08/11 16:29:40 INFO Remoting: Remoting now listens on addresses: 
> [akka.tcp://sparkyar...@g024.grid.eaglerd.local:34075]
> 14/08/11 16:29:40 INFO RMProxy: Connecting to ResourceManager at 
> master.grid.eaglerd.local/192.168.41.100:8030
> 14/08/11 16:29:40 INFO ExecutorLauncher: ApplicationAttemptId: 
> appattempt_1407759736957_0014_01
> 14/08/11 16:29:40 INFO ExecutorLauncher: Registering the ApplicationMaster
> 14/08/11 16:29:40 INFO ExecutorLauncher: Waiting for Spark driver to be 
> reachable.
> 14/08/11 16:29:40 ERROR ExecutorLauncher: Failed to connect to driver at 
> master.grid.eaglerd.local:44911, retrying ...
> 14/08/11 16:29:40 ERROR ExecutorLauncher: Failed to connect to driver at 
> master.grid.eaglerd.local:44911, retrying ...
> 14/08/11 16:29:40 ERROR ExecutorLauncher: Failed to connect to driver at 
> master.grid.eaglerd.local:44911, retrying ...
> 14/08/11 16:29:40 ERROR ExecutorLauncher: Failed to connect to driver at 
> master.grid.eaglerd.local:44911, retrying ...
> 14/08/11 16:29:40 ERROR ExecutorLauncher: Failed to connect to driver at 
> master.grid.eaglerd.local:44911, retrying ...
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Resolved] (SPARK-6393) Extra RPC to the AM during killExecutor invocation

[Marcelo Vanzin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-6393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcelo Vanzin resolved SPARK-6393.
---
Resolution: Won't Fix

I read the PR again and I'm not really sure what this is talking about, but 
it's probably not that important given nobody has really updated this bug in 
forever...

> Extra RPC to the AM during killExecutor invocation
> --
>
> Key: SPARK-6393
> URL: https://issues.apache.org/jira/browse/SPARK-6393
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core, YARN
>Affects Versions: 1.3.1
>Reporter: Sandy Ryza
>
> This was introduced by SPARK-6325



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Resolved] (SPARK-7252) Add support for creating new Hive and HBase delegation tokens

[Marcelo Vanzin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-7252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcelo Vanzin resolved SPARK-7252.
---
Resolution: Duplicate

> Add support for creating new Hive and HBase delegation tokens
> -
>
> Key: SPARK-7252
> URL: https://issues.apache.org/jira/browse/SPARK-7252
> Project: Spark
>  Issue Type: Improvement
>  Components: YARN
>Affects Versions: 1.5.0
>Reporter: Hari Shreedharan
>
> In SPARK-5342, support is being added for long running apps to be able to 
> write to HDFS, but this does not work for Hive and HBase. We need to add the 
> same support for these too.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Resolved] (SPARK-8929) [Windows] Application with Appname including whiteSpace fails in Yarn-client mode

[Marcelo Vanzin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-8929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcelo Vanzin resolved SPARK-8929.
---
Resolution: Unresolved

This doesn't even have the Spark version that's affected... since this is 
pretty old and Windows support has seen several fixes since this bug was filed, 
I'll close this for now, but please re-open if it's still an issue.

> [Windows] Application with Appname including whiteSpace fails in Yarn-client 
> mode
> -
>
> Key: SPARK-8929
> URL: https://issues.apache.org/jira/browse/SPARK-8929
> Project: Spark
>  Issue Type: Bug
>  Components: Windows, YARN
>Reporter: Yesha Vora
>
> Some MachineLearning examples has space in App Name. These applications fail 
> to start AM with yarn-client mode in Windows environment.
> Affected Test example list:
> * BinaryClassification
> * Correlations
> * DecisionTreeRunner
> * DenseKmeans
> * GradientBoostedTreesRunner
> * LinearRegression
> * MovieLensALS
> * MultivariateSummarizer
> * SampledRDDs
> * SparseNaiveBayes
> {code:title=SampledRDDs}
> RUNNING: call spark-submit  --class 
> org.apache.spark.examples.mllib.SampledRDDs --master yarn-client 
> --properties-file c:\windows\temp\spark-defaults.conf --jars  
> spark-examples-*.jar  --input 
> /tmp/sparkMLLInput/sample_binary_classification_data.txt 
> {code}
> {code:title=Fails to Start AM}
> 2015-06-13 22:07:27,526|beaver.machine|INFO||7600|MainThread|Container 
> id: container_e02_1434177640451_0029_02_01
> 2015-06-13 22:07:27,526|beaver.machine|INFO||7600|MainThread|Exit code: 
> 9009
> 2015-06-13 22:07:27,528|beaver.machine|INFO||7600|MainThread|Exception 
> message: Usage: java [-options] class [args...]
> 2015-06-13 22:07:27,529|beaver.machine|INFO||7600|MainThread|(to execute 
> a class)
> 2015-06-13 22:07:27,529|beaver.machine|INFO||7600|MainThread|or  java 
> [-options] -jar jarfile [args...]
> 2015-06-13 22:07:27,529|beaver.machine|INFO||7600|MainThread|(to execute 
> a jar file)
> 2015-06-13 22:07:27,529|beaver.machine|INFO||7600|MainThread|where 
> options include:
> 2015-06-13 22:07:27,530|beaver.machine|INFO||7600|MainThread|-d32   use a 
> 32-bit data model if available
> 2015-06-13 22:07:27,530|beaver.machine|INFO||7600|MainThread|-d64   use a 
> 64-bit data model if available
> 2015-06-13 22:07:27,532|beaver.machine|INFO||7600|MainThread|-server  
>   to select the "server" VM
> 2015-06-13 22:07:27,532|beaver.machine|INFO||7600|MainThread|The default 
> VM is server.
> 2015-06-13 22:07:27,532|beaver.machine|INFO||7600|MainThread|
> 2015-06-13 22:07:27,532|beaver.machine|INFO||7600|MainThread|-cp  search path of directories and zip/jar files>
> 2015-06-13 22:07:27,533|beaver.machine|INFO||7600|MainThread|-classpath 
> 
> 2015-06-13 22:07:27,533|beaver.machine|INFO||7600|MainThread|A ; 
> separated list of directories, JAR archives,
> 2015-06-13 22:07:27,535|beaver.machine|INFO||7600|MainThread|and ZIP 
> archives to search for class files.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-21125) PySpark context missing function to set Job Description.

[Shane Jarvie (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shane Jarvie updated SPARK-21125:
-
Shepherd: Alex Bozarth

> PySpark context missing function to set Job Description.
> 
>
> Key: SPARK-21125
> URL: https://issues.apache.org/jira/browse/SPARK-21125
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.1.1
>Reporter: Shane Jarvie
>Priority: Trivial
>  Labels: beginner
> Fix For: 2.2.0
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> The PySpark API is missing a convienient function currently found in the 
> Scala API, which sets the Job Description for display in the Spark UI.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Reopened] (SPARK-11058) failed spark job reports on YARN as successful

[Marcelo Vanzin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-11058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcelo Vanzin reopened SPARK-11058:


> failed spark job reports on YARN as successful
> --
>
> Key: SPARK-11058
> URL: https://issues.apache.org/jira/browse/SPARK-11058
> Project: Spark
>  Issue Type: Bug
>  Components: YARN
>Affects Versions: 1.3.0
> Environment: CDH 5.4
>Reporter: Lan Jiang
>Priority: Minor
>
> I have a spark batch job running on CDH5.4 + Spark 1.3.0. Job is submitted in 
> “yarn-client” mode. The job itself failed due to YARN kills several executor 
> containers because the containers exceeded the memory limit posed by YARN. 
> However, when I went to the YARN resource manager site, it displayed the job 
> as successful. I found there was an issue reported in JIRA 
> https://issues.apache.org/jira/browse/SPARK-3627, but it says it was fixed in 
> Spark 1.2. On Spark history server, it shows the job as “Incomplete”. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Resolved] (SPARK-11058) failed spark job reports on YARN as successful

[Marcelo Vanzin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-11058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcelo Vanzin resolved SPARK-11058.

Resolution: Won't Fix

> failed spark job reports on YARN as successful
> --
>
> Key: SPARK-11058
> URL: https://issues.apache.org/jira/browse/SPARK-11058
> Project: Spark
>  Issue Type: Bug
>  Components: YARN
>Affects Versions: 1.3.0
> Environment: CDH 5.4
>Reporter: Lan Jiang
>Priority: Minor
>
> I have a spark batch job running on CDH5.4 + Spark 1.3.0. Job is submitted in 
> “yarn-client” mode. The job itself failed due to YARN kills several executor 
> containers because the containers exceeded the memory limit posed by YARN. 
> However, when I went to the YARN resource manager site, it displayed the job 
> as successful. I found there was an issue reported in JIRA 
> https://issues.apache.org/jira/browse/SPARK-3627, but it says it was fixed in 
> Spark 1.2. On Spark history server, it shows the job as “Incomplete”. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Resolved] (SPARK-11058) failed spark job reports on YARN as successful

[Marcelo Vanzin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-11058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcelo Vanzin resolved SPARK-11058.

Resolution: Not A Problem

This is a limitation of client mode in Spark. The YARN application always shows 
up as successful. The {{SparkContext}} API is not rich enough for the driver to 
tell the AM whether to succeed or fail the YARN application.

> failed spark job reports on YARN as successful
> --
>
> Key: SPARK-11058
> URL: https://issues.apache.org/jira/browse/SPARK-11058
> Project: Spark
>  Issue Type: Bug
>  Components: YARN
>Affects Versions: 1.3.0
> Environment: CDH 5.4
>Reporter: Lan Jiang
>Priority: Minor
>
> I have a spark batch job running on CDH5.4 + Spark 1.3.0. Job is submitted in 
> “yarn-client” mode. The job itself failed due to YARN kills several executor 
> containers because the containers exceeded the memory limit posed by YARN. 
> However, when I went to the YARN resource manager site, it displayed the job 
> as successful. I found there was an issue reported in JIRA 
> https://issues.apache.org/jira/browse/SPARK-3627, but it says it was fixed in 
> Spark 1.2. On Spark history server, it shows the job as “Incomplete”. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-11170) ​ EOFException on History server reading in progress lz4

[Marcelo Vanzin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-11170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcelo Vanzin updated SPARK-11170:
---
Component/s: (was: YARN)

> ​ EOFException on History server reading in progress lz4
> 
>
> Key: SPARK-11170
> URL: https://issues.apache.org/jira/browse/SPARK-11170
> Project: Spark
>  Issue Type: Bug
>  Components: Web UI
>Affects Versions: 1.5.1
> Environment: HDP: 2.3.2.0-2950 (Hadoop 2.7.1.2.3.2.0-2950)
> Spark: 1.5.x (c27e1904)
>Reporter: Sebastian YEPES FERNANDEZ
>
> The Spark​ ​History server is not able to read/save the jobs history if Spark 
> is configured to use 
> "spark.io.compression.codec=org.apache.spark.io.LZ4CompressionCodec", it 
> continuously generated the following error:
> {code}
> ERROR 2015-10-16 16:21:39 org.apache.spark.deploy.history.FsHistoryProvider: 
> Exception encountered when attempting to load application log 
> hdfs://DATA/user/spark/his
> tory/application_1444297190346_0073_1.lz4.inprogress
> java.io.EOFException: Stream ended prematurely
> at 
> net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:218)
> at 
> net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:150)
> at 
> net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:117)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at 
> scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:67)
> at 
> org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:55)
> at 
> org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$replay(FsHistoryProvider.scala:457)
> at 
> org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$10.apply(FsHistoryProvider.scala:292)
> at 
> org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$10.apply(FsHistoryProvider.scala:289)
> at 
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
> at 
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
> at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> at 
> scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
> at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
> at 
> org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$mergeApplicationListing(FsHistoryProvider.scala:289)
> at 
> org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$checkForLogs$1$$anon$2.run(FsHistoryProvider.scala:210)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> INFO 2015-10-16 16:21:39 org.apache.spark.deploy.history.FsHistoryProvider: 
> Replaying log path: 
> hdfs://DATA/user/spark/history/application_1444297190346_0072_1.lz4.i
> nprogress
> {code}
> As a workaround setting 
> "spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec​"​ 
> makes the​ ​History server work correctly



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Resolved] (SPARK-12279) Requesting a HBase table with kerberos is not working

[Marcelo Vanzin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-12279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcelo Vanzin resolved SPARK-12279.

Resolution: Cannot Reproduce

I'm going to close this because I'm pretty sure this works (we have tests for 
it internally). The main things you have to look at are:

- make sure Spark can see your hbase-site.xml file and any other needed config 
file (e.g. by placing it in {{$SPARK_CONF_DIR}})
- make sure HBase jars are in Spark's classpath.

I believe in 2.2 HBase jars can even just be provided with {{\-\-jars}}. 
Cluster mode might be a little trickier, but it also works (we use 
{{SPARK_DIST_CLASSPATH}} for that case).

> Requesting a HBase table with kerberos is not working
> -
>
> Key: SPARK-12279
> URL: https://issues.apache.org/jira/browse/SPARK-12279
> Project: Spark
>  Issue Type: Bug
>  Components: YARN
>Affects Versions: 1.5.2, 1.6.0
> Environment: Spark 1.6.0 / HBase 1.1.2 / Hadoop 2.7.1 / Zookeeper 
> 3.4.5 / Authentication done through Kerberos
>Reporter: Pierre Beauvois
>
> I can't read a HBase table with Spark 1.5.2. 
> I added the option "spark.driver.extraClassPath" in the spark-defaults.conf 
> which contains the HBASE_CONF_DIR as below:
> spark.driver.extraClassPath = /opt/application/Hbase/current/conf/
> On the driver, I started spark-shell (I was running it in yarn-client mode) 
> {code}
> [my_user@uabigspark01 ~]$ spark-shell -v --name HBaseTest --jars 
> /opt/application/Hbase/current/lib/hbase-common-1.1.2.jar,/opt/application/Hbase/current/lib/hbase-server-1.1.2.jar,/opt/application/Hbase/current/lib/hbase-client-1.1.2.jar,/opt/application/Hbase/current/lib/hbase-protocol-1.1.2.jar,/opt/application/Hbase/current/lib/protobuf-java-2.5.0.jar,/opt/application/Hbase/current/lib/htrace-core-3.1.0-incubating.jar,/opt/application/Hbase/current/lib/hbase-annotations-1.1.2.jar,/opt/application/Hbase/current/lib/guava-12.0.1.jar
> {code}
> Then I ran the following lines:
> {code}
> scala> import org.apache.spark._
> import org.apache.spark._
> scala> import org.apache.spark.rdd.NewHadoopRDD
> import org.apache.spark.rdd.NewHadoopRDD
> scala> import org.apache.hadoop.fs.Path
> import org.apache.hadoop.fs.Path
> scala> import org.apache.hadoop.hbase.util.Bytes
> import org.apache.hadoop.hbase.util.Bytes
> scala> import org.apache.hadoop.hbase.HColumnDescriptor
> import org.apache.hadoop.hbase.HColumnDescriptor
> scala> import org.apache.hadoop.hbase.{HBaseConfiguration, HTableDescriptor}
> import org.apache.hadoop.hbase.{HBaseConfiguration, HTableDescriptor}
> scala> import org.apache.hadoop.hbase.client.{HBaseAdmin, Put, HTable, Result}
> import org.apache.hadoop.hbase.client.{HBaseAdmin, Put, HTable, Result}
> scala> import org.apache.hadoop.hbase.mapreduce.TableInputFormat
> import org.apache.hadoop.hbase.mapreduce.TableInputFormat
> scala> import org.apache.hadoop.hbase.io.ImmutableBytesWritable
> import org.apache.hadoop.hbase.io.ImmutableBytesWritable
> scala> val conf = HBaseConfiguration.create()
> conf: org.apache.hadoop.conf.Configuration = Configuration: core-default.xml, 
> core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, 
> yarn-site.xml, hdfs-default.xml, hdfs-site.xml, hbase-default.xml, 
> hbase-site.xml
> scala> conf.addResource(new 
> Path("/opt/application/Hbase/current/conf/hbase-site.xml"))
> scala> conf.set("hbase.zookeeper.quorum", "FQDN1:2181,FQDN2:2181,FQDN3:2181")
> scala> conf.set(TableInputFormat.INPUT_TABLE, "user:noheader")
> scala> val hBaseRDD = sc.newAPIHadoopRDD(conf, classOf[TableInputFormat], 
> classOf[org.apache.hadoop.hbase.io.ImmutableBytesWritable], 
> classOf[org.apache.hadoop.hbase.client.Result])
> 2015-12-09 15:17:58,890 INFO  [main] storage.MemoryStore: 
> ensureFreeSpace(266248) called with curMem=0, maxMem=556038881
> 2015-12-09 15:17:58,892 INFO  [main] storage.MemoryStore: Block broadcast_0 
> stored as values in memory (estimated size 260.0 KB, free 530.0 MB)
> 2015-12-09 15:17:59,196 INFO  [main] storage.MemoryStore: 
> ensureFreeSpace(32808) called with curMem=266248, maxMem=556038881
> 2015-12-09 15:17:59,197 INFO  [main] storage.MemoryStore: Block 
> broadcast_0_piece0 stored as bytes in memory (estimated size 32.0 KB, free 
> 530.0 MB)
> 2015-12-09 15:17:59,199 INFO  [sparkDriver-akka.actor.default-dispatcher-2] 
> storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on 
> 192.168.200.208:60217 (size: 32.0 KB, free: 530.2 MB)
> 2015-12-09 15:17:59,203 INFO  [main] spark.SparkContext: Created broadcast 0 
> from newAPIHadoopRDD at :34
> hBaseRDD: 
> org.apache.spark.rdd.RDD[(org.apache.hadoop.hbase.io.ImmutableBytesWritable, 
> org.apache.hadoop.hbase.client.Result)] = NewHadoopRDD[0] at newAPIHadoopRDD 
> at :34
> scala> hBaseRDD.count()
> 2015-12-09 15:18:09,441 INFO  

[jira] [Assigned] (SPARK-21125) PySpark context missing function to set Job Description.

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-21125:


Assignee: (was: Apache Spark)

> PySpark context missing function to set Job Description.
> 
>
> Key: SPARK-21125
> URL: https://issues.apache.org/jira/browse/SPARK-21125
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.1.1
>Reporter: Shane Jarvie
>Priority: Trivial
>  Labels: beginner
> Fix For: 2.2.0
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> The PySpark API is missing a convienient function currently found in the 
> Scala API, which sets the Job Description for display in the Spark UI.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-21125) PySpark context missing function to set Job Description.

[Apache Spark (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-21125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16052516#comment-16052516
 ] 

Apache Spark commented on SPARK-21125:
--

User 'sjarvie' has created a pull request for this issue:
https://github.com/apache/spark/pull/18332

> PySpark context missing function to set Job Description.
> 
>
> Key: SPARK-21125
> URL: https://issues.apache.org/jira/browse/SPARK-21125
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.1.1
>Reporter: Shane Jarvie
>Priority: Trivial
>  Labels: beginner
> Fix For: 2.2.0
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> The PySpark API is missing a convienient function currently found in the 
> Scala API, which sets the Job Description for display in the Spark UI.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-21125) PySpark context missing function to set Job Description.

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-21125:


Assignee: Apache Spark

> PySpark context missing function to set Job Description.
> 
>
> Key: SPARK-21125
> URL: https://issues.apache.org/jira/browse/SPARK-21125
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.1.1
>Reporter: Shane Jarvie
>Assignee: Apache Spark
>Priority: Trivial
>  Labels: beginner
> Fix For: 2.2.0
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> The PySpark API is missing a convienient function currently found in the 
> Scala API, which sets the Job Description for display in the Spark UI.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-21125) PySpark context missing function to set Job Description.

[Shane Jarvie (JIRA)]

Shane Jarvie created SPARK-21125:


 Summary: PySpark context missing function to set Job Description.
 Key: SPARK-21125
 URL: https://issues.apache.org/jira/browse/SPARK-21125
 Project: Spark
  Issue Type: Bug
  Components: PySpark
Affects Versions: 2.1.1
Reporter: Shane Jarvie
Priority: Trivial
 Fix For: 2.2.0


The PySpark API is missing a convienient function currently found in the Scala 
API, which sets the Job Description for display in the Spark UI.





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Resolved] (SPARK-20458) support getting Yarn Tracking URL in code

[Marcelo Vanzin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-20458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcelo Vanzin resolved SPARK-20458.

Resolution: Won't Fix

> support getting Yarn Tracking URL in code
> -
>
> Key: SPARK-20458
> URL: https://issues.apache.org/jira/browse/SPARK-20458
> Project: Spark
>  Issue Type: Improvement
>  Components: YARN
>Affects Versions: 2.1.0
>Reporter: PJ Fanning
>
> org.apache.spark.deploy.yarn.Client logs the Yarn tracking URL but it would 
> be useful to be able to access this in code, as opposed to mining log output.
> I have an application where I monitor the health of the SparkContext and 
> associated Executors using the Spark REST API.
> Would it be feasible to add a listener API to listen for new 
> ApplicationReports in org.apache.spark.deploy.yarn.Client? Alternatively, 
> this URL could be exposed as a property associated with the SparkContext.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-20458) support getting Yarn Tracking URL in code

[Marcelo Vanzin (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-20458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16052504#comment-16052504
 ] 

Marcelo Vanzin commented on SPARK-20458:


You can do this by reading the {{yarn.resourcemanager.webapp.address}} value 
from YARN's config and the application ID (which is exposed both in an event 
sent on the listener bus, and an existing {{SparkContext}} method.

> support getting Yarn Tracking URL in code
> -
>
> Key: SPARK-20458
> URL: https://issues.apache.org/jira/browse/SPARK-20458
> Project: Spark
>  Issue Type: Improvement
>  Components: YARN
>Affects Versions: 2.1.0
>Reporter: PJ Fanning
>
> org.apache.spark.deploy.yarn.Client logs the Yarn tracking URL but it would 
> be useful to be able to access this in code, as opposed to mining log output.
> I have an application where I monitor the health of the SparkContext and 
> associated Executors using the Spark REST API.
> Would it be feasible to add a listener API to listen for new 
> ApplicationReports in org.apache.spark.deploy.yarn.Client? Alternatively, 
> this URL could be exposed as a property associated with the SparkContext.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-21124) Wrong user shown in UI when using kerberos

[Apache Spark (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-21124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16052441#comment-16052441
 ] 

Apache Spark commented on SPARK-21124:
--

User 'vanzin' has created a pull request for this issue:
https://github.com/apache/spark/pull/18331

> Wrong user shown in UI when using kerberos
> --
>
> Key: SPARK-21124
> URL: https://issues.apache.org/jira/browse/SPARK-21124
> Project: Spark
>  Issue Type: Bug
>  Components: Web UI
>Affects Versions: 2.2.0
>Reporter: Marcelo Vanzin
>Priority: Minor
>
> When submitting an app to a kerberos-secured cluster, the OS user and the 
> user running the application may differ. Although it may also happen in 
> cluster mode depending on the cluster manager's configuration, it's more 
> common in client mode.
> The UI should show enough information about user running the application to 
> correctly identify the actual user. The "app user" can be easily retrieved 
> via {{Utils.getCurrentUserName()}}, so it's mostly a matter of how to record 
> this information (for showing in replayed applications) and how to present it 
> in the UI.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-21124) Wrong user shown in UI when using kerberos

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-21124:


Assignee: Apache Spark

> Wrong user shown in UI when using kerberos
> --
>
> Key: SPARK-21124
> URL: https://issues.apache.org/jira/browse/SPARK-21124
> Project: Spark
>  Issue Type: Bug
>  Components: Web UI
>Affects Versions: 2.2.0
>Reporter: Marcelo Vanzin
>Assignee: Apache Spark
>Priority: Minor
>
> When submitting an app to a kerberos-secured cluster, the OS user and the 
> user running the application may differ. Although it may also happen in 
> cluster mode depending on the cluster manager's configuration, it's more 
> common in client mode.
> The UI should show enough information about user running the application to 
> correctly identify the actual user. The "app user" can be easily retrieved 
> via {{Utils.getCurrentUserName()}}, so it's mostly a matter of how to record 
> this information (for showing in replayed applications) and how to present it 
> in the UI.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-21124) Wrong user shown in UI when using kerberos

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-21124:


Assignee: (was: Apache Spark)

> Wrong user shown in UI when using kerberos
> --
>
> Key: SPARK-21124
> URL: https://issues.apache.org/jira/browse/SPARK-21124
> Project: Spark
>  Issue Type: Bug
>  Components: Web UI
>Affects Versions: 2.2.0
>Reporter: Marcelo Vanzin
>Priority: Minor
>
> When submitting an app to a kerberos-secured cluster, the OS user and the 
> user running the application may differ. Although it may also happen in 
> cluster mode depending on the cluster manager's configuration, it's more 
> common in client mode.
> The UI should show enough information about user running the application to 
> correctly identify the actual user. The "app user" can be easily retrieved 
> via {{Utils.getCurrentUserName()}}, so it's mostly a matter of how to record 
> this information (for showing in replayed applications) and how to present it 
> in the UI.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-21124) Wrong user shown in UI when using kerberos

[Marcelo Vanzin (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-21124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16052409#comment-16052409
 ] 

Marcelo Vanzin edited comment on SPARK-21124 at 6/16/17 9:35 PM:
-

Also, the changes in SPARK-14483 also mask which is the actual user of the 
application running in the cluster (i.e. who actually is reading / writing 
files).

The thrift server doesn't do per-job impersonation, as far as I know, so it 
would still be useful to show the kerberos user the application is running as 
somewhere.


was (Author: vanzin):
Also, the changes in SPARK-14483 also mask which is the actual user the 
application running in the cluster (i.e. who actually is reading / writing 
files).

The thrift server doesn't do per-job impersonation, as far as I know, so it 
would still be useful to show the kerberos user the application is running as 
somewhere.

> Wrong user shown in UI when using kerberos
> --
>
> Key: SPARK-21124
> URL: https://issues.apache.org/jira/browse/SPARK-21124
> Project: Spark
>  Issue Type: Bug
>  Components: Web UI
>Affects Versions: 2.2.0
>Reporter: Marcelo Vanzin
>Priority: Minor
>
> When submitting an app to a kerberos-secured cluster, the OS user and the 
> user running the application may differ. Although it may also happen in 
> cluster mode depending on the cluster manager's configuration, it's more 
> common in client mode.
> The UI should show enough information about user running the application to 
> correctly identify the actual user. The "app user" can be easily retrieved 
> via {{Utils.getCurrentUserName()}}, so it's mostly a matter of how to record 
> this information (for showing in replayed applications) and how to present it 
> in the UI.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-21124) Wrong user shown in UI when using kerberos

[Marcelo Vanzin (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-21124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16052409#comment-16052409
 ] 

Marcelo Vanzin commented on SPARK-21124:


Also, the SPARK-14483 also masks which is the actual user the application 
running in the cluster is (i.e. who actually is writing files).

The thrift server doesn't do per-job impersonation, as far as I know, so it 
would still be useful to show the kerberos user the application is running as 
somewhere.

> Wrong user shown in UI when using kerberos
> --
>
> Key: SPARK-21124
> URL: https://issues.apache.org/jira/browse/SPARK-21124
> Project: Spark
>  Issue Type: Bug
>  Components: Web UI
>Affects Versions: 2.2.0
>Reporter: Marcelo Vanzin
>Priority: Minor
>
> When submitting an app to a kerberos-secured cluster, the OS user and the 
> user running the application may differ. Although it may also happen in 
> cluster mode depending on the cluster manager's configuration, it's more 
> common in client mode.
> The UI should show enough information about user running the application to 
> correctly identify the actual user. The "app user" can be easily retrieved 
> via {{Utils.getCurrentUserName()}}, so it's mostly a matter of how to record 
> this information (for showing in replayed applications) and how to present it 
> in the UI.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-21124) Wrong user shown in UI when using kerberos

[Marcelo Vanzin (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-21124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16052409#comment-16052409
 ] 

Marcelo Vanzin edited comment on SPARK-21124 at 6/16/17 9:21 PM:
-

Also, the changes in SPARK-14483 also mask which is the actual user the 
application running in the cluster (i.e. who actually is reading / writing 
files).

The thrift server doesn't do per-job impersonation, as far as I know, so it 
would still be useful to show the kerberos user the application is running as 
somewhere.


was (Author: vanzin):
Also, the SPARK-14483 also masks which is the actual user the application 
running in the cluster is (i.e. who actually is writing files).

The thrift server doesn't do per-job impersonation, as far as I know, so it 
would still be useful to show the kerberos user the application is running as 
somewhere.

> Wrong user shown in UI when using kerberos
> --
>
> Key: SPARK-21124
> URL: https://issues.apache.org/jira/browse/SPARK-21124
> Project: Spark
>  Issue Type: Bug
>  Components: Web UI
>Affects Versions: 2.2.0
>Reporter: Marcelo Vanzin
>Priority: Minor
>
> When submitting an app to a kerberos-secured cluster, the OS user and the 
> user running the application may differ. Although it may also happen in 
> cluster mode depending on the cluster manager's configuration, it's more 
> common in client mode.
> The UI should show enough information about user running the application to 
> correctly identify the actual user. The "app user" can be easily retrieved 
> via {{Utils.getCurrentUserName()}}, so it's mostly a matter of how to record 
> this information (for showing in replayed applications) and how to present it 
> in the UI.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-21124) Wrong user shown in UI when using kerberos

[Marcelo Vanzin (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-21124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16052404#comment-16052404
 ] 

Marcelo Vanzin commented on SPARK-21124:


Hmm, yeah, it seems like that code would solve this, but I have the same 
concern as you - in a normal application the user information all over the 
place will just clutter up the UI.

Anyway, I'll link both bugs and if that goes in I'll close this.

> Wrong user shown in UI when using kerberos
> --
>
> Key: SPARK-21124
> URL: https://issues.apache.org/jira/browse/SPARK-21124
> Project: Spark
>  Issue Type: Bug
>  Components: Web UI
>Affects Versions: 2.2.0
>Reporter: Marcelo Vanzin
>Priority: Minor
>
> When submitting an app to a kerberos-secured cluster, the OS user and the 
> user running the application may differ. Although it may also happen in 
> cluster mode depending on the cluster manager's configuration, it's more 
> common in client mode.
> The UI should show enough information about user running the application to 
> correctly identify the actual user. The "app user" can be easily retrieved 
> via {{Utils.getCurrentUserName()}}, so it's mostly a matter of how to record 
> this information (for showing in replayed applications) and how to present it 
> in the UI.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-21124) Wrong user shown in UI when using kerberos

[Alex Bozarth (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-21124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16052394#comment-16052394
 ] 

Alex Bozarth commented on SPARK-21124:
--

Though a separate issue, I believe the open PR for SPARK-14483 actually fixes 
this as well

> Wrong user shown in UI when using kerberos
> --
>
> Key: SPARK-21124
> URL: https://issues.apache.org/jira/browse/SPARK-21124
> Project: Spark
>  Issue Type: Bug
>  Components: Web UI
>Affects Versions: 2.2.0
>Reporter: Marcelo Vanzin
>Priority: Minor
>
> When submitting an app to a kerberos-secured cluster, the OS user and the 
> user running the application may differ. Although it may also happen in 
> cluster mode depending on the cluster manager's configuration, it's more 
> common in client mode.
> The UI should show enough information about user running the application to 
> correctly identify the actual user. The "app user" can be easily retrieved 
> via {{Utils.getCurrentUserName()}}, so it's mostly a matter of how to record 
> this information (for showing in replayed applications) and how to present it 
> in the UI.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-21124) Wrong user shown in UI when using kerberos

[Marcelo Vanzin (JIRA)]

Marcelo Vanzin created SPARK-21124:
--

 Summary: Wrong user shown in UI when using kerberos
 Key: SPARK-21124
 URL: https://issues.apache.org/jira/browse/SPARK-21124
 Project: Spark
  Issue Type: Bug
  Components: Web UI
Affects Versions: 2.2.0
Reporter: Marcelo Vanzin
Priority: Minor


When submitting an app to a kerberos-secured cluster, the OS user and the user 
running the application may differ. Although it may also happen in cluster mode 
depending on the cluster manager's configuration, it's more common in client 
mode.

The UI should show enough information about user running the application to 
correctly identify the actual user. The "app user" can be easily retrieved via 
{{Utils.getCurrentUserName()}}, so it's mostly a matter of how to record this 
information (for showing in replayed applications) and how to present it in the 
UI.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-21122) Address starvation issues when dynamic allocation is enabled

[Craig Ingram (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-21122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16052362#comment-16052362
 ] 

Craig Ingram commented on SPARK-21122:
--

Thanks for the feedback, Sean. This ticket simply represents one of the issues 
brought up in [SPARK-21084|https://issues.apache.org/jira/browse/SPARK-21084]. 
Since this topic covers a lot of ground, we wanted to create a separate ticket 
for the discussion and work. We are definitely open to alternate approaches as 
well.

I agree that this is something the resource managers should manage, but I don't 
see a way control deallocation at a reasonable level from within a resource 
manager. Even without these changes, you still have to configure things in two 
places. I also don't believe you would want to enable something like YARN's 
preemption while using this feature as the two mechanisms would end up fighting 
each other.

At the very least, the existing policy needs some additional logic to 
preemptively decommission executors when other applications are being starved. 
Couple this with 
[SPARK-21097|https://issues.apache.org/jira/browse/SPARK-21097] and even 
executors with cached data can safely be removed without a huge performance 
impact. I think the improvement here is maximizing cluster utilization while 
also allowing other applications to start and fairly consume resources.

> Address starvation issues when dynamic allocation is enabled
> 
>
> Key: SPARK-21122
> URL: https://issues.apache.org/jira/browse/SPARK-21122
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core, YARN
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Craig Ingram
>
> When dynamic resource allocation is enabled on a cluster, it’s currently 
> possible for one application to consume all the cluster’s resources, 
> effectively starving any other application trying to start. This is 
> particularly painful in a notebook environment where notebooks may be idle 
> for tens of minutes while the user is figuring out what to do next (or eating 
> their lunch). Ideally the application should give resources back to the 
> cluster when monitoring indicates other applications are pending.
> Before delving into the specifics of the solution. There are some workarounds 
> to this problem that are worth mentioning:
> * Set spark.dynamicAllocation.maxExecutors to a small value, so that users 
> are unlikely to use the entire cluster even when many of them are doing work. 
> This approach will hurt cluster utilization.
> * If using YARN, enable preemption and have each application (or 
> organization) run in a separate queue. The downside of this is that when YARN 
> preempts, it doesn't know anything about which executor it's killing. It 
> would just as likely kill a long running executor with cached data as one 
> that just spun up. Moreover, given a feature like 
> https://issues.apache.org/jira/browse/SPARK-21097 (preserving cached data on 
> executor decommission), YARN may not wait long enough between trying to 
> gracefully and forcefully shut down the executor. This would mean the blocks 
> that belonged to that executor would be lost and have to be recomputed.
> * Configure YARN to use the capacity scheduler with multiple scheduler 
> queues. Put high-priority notebook users into a high-priority queue. Prevents 
> high-priority users from being starved out by low-priority notebook users. 
> Does not prevent users in the same priority class from starving each other.
> Obviously any solution to this problem that depends on YARN would leave other 
> resource managers out in the cold. The solution proposed in this ticket will 
> afford spark clusters the flexibly to hook in different resource allocation 
> policies to fulfill their user's needs regardless of resource manager choice. 
> Initially the focus will be on users in a notebook environment. When 
> operating in a notebook environment with many users, the goal is fair 
> resource allocation. Given that all users will be using the same memory 
> configuration, this solution will focus primarily on fair sharing of cores.
> The fair resource allocation policy should pick executors to remove based on 
> three factors initially: idleness, presence of cached data, and uptime. The 
> policy will favor removing executors that are idle, short-lived, and have no 
> cached data. The policy will only preemptively remove executors if there are 
> pending applications or cores (otherwise the default dynamic allocation 
> timeout/removal process is followed). The policy will also allow an 
> application's resource consumption to expand based on cluster utilization. 
> For example if there are 3 applications running but 2 of them are idle, the 
> policy will allow a busy application with pending tasks to consume 

[jira] [Commented] (SPARK-15689) Data source API v2

[Russell Spitzer (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-15689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16052321#comment-16052321
 ] 

Russell Spitzer commented on SPARK-15689:
-

I've been trying to work with making Catalyst Cassandra partitioning aware. 
There seem to be two major blocks on this.

The first is that DataSourceScanExec is unable to learn what the underlying 
partitioning should be from the BaseRelation it comes from. I'm currently able 
to get around this by using the DataSourceStrategy plan and then transforming 
the resultant DataSourceScanExec.

The second is that the Partitioning trait is sealed. I want to define a new 
partitioning which is Clustered but is not hashed based on certain columns. It 
would look almost identical to the HashPartitioning class except the
expression which returns a valid PartitionID given expressions would be 
different. 

So for V2 I would really like the ability to specify the physical partitioning 
and as well be able to define new custom partitioning. 

> Data source API v2
> --
>
> Key: SPARK-15689
> URL: https://issues.apache.org/jira/browse/SPARK-15689
> Project: Spark
>  Issue Type: New Feature
>  Components: SQL
>Reporter: Reynold Xin
>  Labels: releasenotes
>
> This ticket tracks progress in creating the v2 of data source API. This new 
> API should focus on:
> 1. Have a small surface so it is easy to freeze and maintain compatibility 
> for a long time. Ideally, this API should survive architectural rewrites and 
> user-facing API revamps of Spark.
> 2. Have a well-defined column batch interface for high performance. 
> Convenience methods should exist to convert row-oriented formats into column 
> batches for data source developers.
> 3. Still support filter push down, similar to the existing API.
> 4. Nice-to-have: support additional common operators, including limit and 
> sampling.
> Note that both 1 and 2 are problems that the current data source API (v1) 
> suffers. The current data source API has a wide surface with dependency on 
> DataFrame/SQLContext, making the data source API compatibility depending on 
> the upper level API. The current data source API is also only row oriented 
> and has to go through an expensive external data type conversion to internal 
> data type.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-21065) Spark Streaming concurrentJobs + StreamingJobProgressListener conflict

[Shixiong Zhu (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-21065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16052242#comment-16052242
 ] 

Shixiong Zhu commented on SPARK-21065:
--

Please don't use `spark.streaming.concurrentJobs` if possible. It doesn't work 
with many features, and also can cause data lost.

> Spark Streaming concurrentJobs + StreamingJobProgressListener conflict
> --
>
> Key: SPARK-21065
> URL: https://issues.apache.org/jira/browse/SPARK-21065
> Project: Spark
>  Issue Type: Bug
>  Components: DStreams, Scheduler, Web UI
>Affects Versions: 2.1.0
>Reporter: Dan Dutrow
>
> My streaming application has 200+ output operations, many of them stateful 
> and several of them windowed. In an attempt to reduce the processing times, I 
> set "spark.streaming.concurrentJobs" to 2+. Initial results are very 
> positive, cutting our processing time from ~3 minutes to ~1 minute, but 
> eventually we encounter an exception as follows:
> (Note that 149697756 ms is 2017-06-09 03:06:00, so it's trying to get a 
> batch from 45 minutes before the exception is thrown.)
> 2017-06-09 03:50:28,259 [Spark Listener Bus] ERROR 
> org.apache.spark.streaming.scheduler.StreamingListenerBus - Listener 
> StreamingJobProgressListener threw an exception
> java.util.NoSuchElementException: key not found 149697756 ms
> at scala.collection.MalLike$class.default(MapLike.scala:228)
> at scala.collection.AbstractMap.default(Map.scala:59)
> at scala.collection.mutable.HashMap.apply(HashMap.scala:65)
> at 
> org.apache.spark.streaming.ui.StreamingJobProgressListener.onOutputOperationCompleted(StreamingJobProgressListener.scala:128)
> at 
> org.apache.spark.streaming.scheduler.StreamingListenerBus.doPostEvent(StreamingListenerBus.scala:67)
> at 
> org.apache.spark.streaming.scheduler.StreamingListenerBus.doPostEvent(StreamingListenerBus.scala:29)
> at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:63)
> at 
> org.apache.spark.streaming.scheduler.StreamingListenerBus.postToAll(StreamingListenerBus.scala:29)
> at 
> org.apache.spark.streaming.scheduler.StreamingListenerBus.onOtherEvent(StreamingListenerBus.scala:43)
> ...
> The Spark code causing the exception is here:
> https://github.com/apache/spark/blob/branch-2.1/streaming/src/main/scala/org/apache/spark/streaming/ui/StreamingJobProgressListener.scala#LC125
>   override def onOutputOperationCompleted(
>   outputOperationCompleted: StreamingListenerOutputOperationCompleted): 
> Unit = synchronized {
> // This method is called before onBatchCompleted
> {color:red}runningBatchUIData(outputOperationCompleted.outputOperationInfo.batchTime).{color}
>   updateOutputOperationInfo(outputOperationCompleted.outputOperationInfo)
> }
> It seems to me that it may be caused by that batch being removed earlier.
> https://github.com/apache/spark/blob/branch-2.1/streaming/src/main/scala/org/apache/spark/streaming/ui/StreamingJobProgressListener.scala#LC102
>   override def onBatchCompleted(batchCompleted: 
> StreamingListenerBatchCompleted): Unit = {
> synchronized {
>   waitingBatchUIData.remove(batchCompleted.batchInfo.batchTime)
>   
> {color:red}runningBatchUIData.remove(batchCompleted.batchInfo.batchTime){color}
>   val batchUIData = BatchUIData(batchCompleted.batchInfo)
>   completedBatchUIData.enqueue(batchUIData)
>   if (completedBatchUIData.size > batchUIDataLimit) {
> val removedBatch = completedBatchUIData.dequeue()
> batchTimeToOutputOpIdSparkJobIdPair.remove(removedBatch.batchTime)
>   }
>   totalCompletedBatches += 1L
>   totalProcessedRecords += batchUIData.numRecords
> }
> }
> What is the solution here? Should I make my spark streaming context remember 
> duration a lot longer? ssc.remember(batchDuration * rememberMultiple)
> Otherwise, it seems like there should be some kind of existence check on 
> runningBatchUIData before dereferencing it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Resolved] (SPARK-21065) Spark Streaming concurrentJobs + StreamingJobProgressListener conflict

[Shixiong Zhu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shixiong Zhu resolved SPARK-21065.
--
Resolution: Won't Fix

> Spark Streaming concurrentJobs + StreamingJobProgressListener conflict
> --
>
> Key: SPARK-21065
> URL: https://issues.apache.org/jira/browse/SPARK-21065
> Project: Spark
>  Issue Type: Bug
>  Components: DStreams, Scheduler, Web UI
>Affects Versions: 2.1.0
>Reporter: Dan Dutrow
>
> My streaming application has 200+ output operations, many of them stateful 
> and several of them windowed. In an attempt to reduce the processing times, I 
> set "spark.streaming.concurrentJobs" to 2+. Initial results are very 
> positive, cutting our processing time from ~3 minutes to ~1 minute, but 
> eventually we encounter an exception as follows:
> (Note that 149697756 ms is 2017-06-09 03:06:00, so it's trying to get a 
> batch from 45 minutes before the exception is thrown.)
> 2017-06-09 03:50:28,259 [Spark Listener Bus] ERROR 
> org.apache.spark.streaming.scheduler.StreamingListenerBus - Listener 
> StreamingJobProgressListener threw an exception
> java.util.NoSuchElementException: key not found 149697756 ms
> at scala.collection.MalLike$class.default(MapLike.scala:228)
> at scala.collection.AbstractMap.default(Map.scala:59)
> at scala.collection.mutable.HashMap.apply(HashMap.scala:65)
> at 
> org.apache.spark.streaming.ui.StreamingJobProgressListener.onOutputOperationCompleted(StreamingJobProgressListener.scala:128)
> at 
> org.apache.spark.streaming.scheduler.StreamingListenerBus.doPostEvent(StreamingListenerBus.scala:67)
> at 
> org.apache.spark.streaming.scheduler.StreamingListenerBus.doPostEvent(StreamingListenerBus.scala:29)
> at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:63)
> at 
> org.apache.spark.streaming.scheduler.StreamingListenerBus.postToAll(StreamingListenerBus.scala:29)
> at 
> org.apache.spark.streaming.scheduler.StreamingListenerBus.onOtherEvent(StreamingListenerBus.scala:43)
> ...
> The Spark code causing the exception is here:
> https://github.com/apache/spark/blob/branch-2.1/streaming/src/main/scala/org/apache/spark/streaming/ui/StreamingJobProgressListener.scala#LC125
>   override def onOutputOperationCompleted(
>   outputOperationCompleted: StreamingListenerOutputOperationCompleted): 
> Unit = synchronized {
> // This method is called before onBatchCompleted
> {color:red}runningBatchUIData(outputOperationCompleted.outputOperationInfo.batchTime).{color}
>   updateOutputOperationInfo(outputOperationCompleted.outputOperationInfo)
> }
> It seems to me that it may be caused by that batch being removed earlier.
> https://github.com/apache/spark/blob/branch-2.1/streaming/src/main/scala/org/apache/spark/streaming/ui/StreamingJobProgressListener.scala#LC102
>   override def onBatchCompleted(batchCompleted: 
> StreamingListenerBatchCompleted): Unit = {
> synchronized {
>   waitingBatchUIData.remove(batchCompleted.batchInfo.batchTime)
>   
> {color:red}runningBatchUIData.remove(batchCompleted.batchInfo.batchTime){color}
>   val batchUIData = BatchUIData(batchCompleted.batchInfo)
>   completedBatchUIData.enqueue(batchUIData)
>   if (completedBatchUIData.size > batchUIDataLimit) {
> val removedBatch = completedBatchUIData.dequeue()
> batchTimeToOutputOpIdSparkJobIdPair.remove(removedBatch.batchTime)
>   }
>   totalCompletedBatches += 1L
>   totalProcessedRecords += batchUIData.numRecords
> }
> }
> What is the solution here? Should I make my spark streaming context remember 
> duration a lot longer? ssc.remember(batchDuration * rememberMultiple)
> Otherwise, it seems like there should be some kind of existence check on 
> runningBatchUIData before dereferencing it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-21123) Options for file stream source are in a wrong table

[Shixiong Zhu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shixiong Zhu updated SPARK-21123:
-
Labels: starter  (was: )

> Options for file stream source are in a wrong table
> ---
>
> Key: SPARK-21123
> URL: https://issues.apache.org/jira/browse/SPARK-21123
> Project: Spark
>  Issue Type: Documentation
>  Components: Documentation, Structured Streaming
>Affects Versions: 2.1.1, 2.2.0
>Reporter: Shixiong Zhu
>Priority: Minor
>  Labels: starter
> Attachments: Screen Shot 2017-06-15 at 11.25.49 AM.png
>
>
> Right now options for file stream source are documented with file sink. We 
> should create a table for source options and fix it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-21123) Options for file stream source are in a wrong table

[Shixiong Zhu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shixiong Zhu updated SPARK-21123:
-
Affects Version/s: 2.2.0

> Options for file stream source are in a wrong table
> ---
>
> Key: SPARK-21123
> URL: https://issues.apache.org/jira/browse/SPARK-21123
> Project: Spark
>  Issue Type: Documentation
>  Components: Documentation, Structured Streaming
>Affects Versions: 2.1.1, 2.2.0
>Reporter: Shixiong Zhu
>Priority: Minor
>  Labels: starter
> Attachments: Screen Shot 2017-06-15 at 11.25.49 AM.png
>
>
> Right now options for file stream source are documented with file sink. We 
> should create a table for source options and fix it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-21123) Options for file stream source are in a wrong table

[Shixiong Zhu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shixiong Zhu updated SPARK-21123:
-
Description: 
Right now options for file stream source are documented with file sink. We 
should create a table for source options and fix it.


  was:
!Screen Shot 2017-06-15 at 11.25.49 AM.png|thumbnail!




> Options for file stream source are in a wrong table
> ---
>
> Key: SPARK-21123
> URL: https://issues.apache.org/jira/browse/SPARK-21123
> Project: Spark
>  Issue Type: Documentation
>  Components: Documentation, Structured Streaming
>Affects Versions: 2.1.1
>Reporter: Shixiong Zhu
>Priority: Minor
> Attachments: Screen Shot 2017-06-15 at 11.25.49 AM.png
>
>
> Right now options for file stream source are documented with file sink. We 
> should create a table for source options and fix it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-21123) Options for file stream source are in a wrong table

[Shixiong Zhu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shixiong Zhu updated SPARK-21123:
-
Description: 
!Screen Shot 2017-06-15 at 11.25.49 AM.png|thumbnail!



  was:!attachment-name.jpg|thumbnail!


> Options for file stream source are in a wrong table
> ---
>
> Key: SPARK-21123
> URL: https://issues.apache.org/jira/browse/SPARK-21123
> Project: Spark
>  Issue Type: Documentation
>  Components: Documentation, Structured Streaming
>Affects Versions: 2.1.1
>Reporter: Shixiong Zhu
>Priority: Minor
> Attachments: Screen Shot 2017-06-15 at 11.25.49 AM.png
>
>
> !Screen Shot 2017-06-15 at 11.25.49 AM.png|thumbnail!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-21123) Options for file stream source are in a wrong table

[Shixiong Zhu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shixiong Zhu updated SPARK-21123:
-
Description: !attachment-name.jpg|thumbnail!

> Options for file stream source are in a wrong table
> ---
>
> Key: SPARK-21123
> URL: https://issues.apache.org/jira/browse/SPARK-21123
> Project: Spark
>  Issue Type: Documentation
>  Components: Documentation, Structured Streaming
>Affects Versions: 2.1.1
>Reporter: Shixiong Zhu
>Priority: Minor
> Attachments: Screen Shot 2017-06-15 at 11.25.49 AM.png
>
>
> !attachment-name.jpg|thumbnail!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-21123) Options for file stream source are in a wrong table

[Shixiong Zhu (JIRA)]

Shixiong Zhu created SPARK-21123:


 Summary: Options for file stream source are in a wrong table
 Key: SPARK-21123
 URL: https://issues.apache.org/jira/browse/SPARK-21123
 Project: Spark
  Issue Type: Documentation
  Components: Documentation, Structured Streaming
Affects Versions: 2.1.1
Reporter: Shixiong Zhu
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-21123) Options for file stream source are in a wrong table

[Shixiong Zhu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shixiong Zhu updated SPARK-21123:
-
Attachment: Screen Shot 2017-06-15 at 11.25.49 AM.png

> Options for file stream source are in a wrong table
> ---
>
> Key: SPARK-21123
> URL: https://issues.apache.org/jira/browse/SPARK-21123
> Project: Spark
>  Issue Type: Documentation
>  Components: Documentation, Structured Streaming
>Affects Versions: 2.1.1
>Reporter: Shixiong Zhu
>Priority: Minor
> Attachments: Screen Shot 2017-06-15 at 11.25.49 AM.png
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-21122) Address starvation issues when dynamic allocation is enabled

[Sean Owen (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-21122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16052199#comment-16052199
 ] 

Sean Owen commented on SPARK-21122:
---

How is this different from https://issues.apache.org/jira/browse/SPARK-21084 ?  
Let's not fork the conversation, so I'd prefer this be closed.

Much of this like still sounds like what resource managers manage, and the 
problem you run into immediately is how the overlapping semantics of the two 
interact, or why users would want to configure it in two places.

There's probably an argument for augmenting the built-in standalone resource 
manager with stuff YARN does, but, I think it's intended as a light-weight 
manager anyway.

One thing Spark can reasonably control is when and how to give up resources, 
yes. There's already an existing policy for this, which sounds like what you're 
describing. I'm not clear how it's different or an improvement.

> Address starvation issues when dynamic allocation is enabled
> 
>
> Key: SPARK-21122
> URL: https://issues.apache.org/jira/browse/SPARK-21122
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core, YARN
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Craig Ingram
>
> When dynamic resource allocation is enabled on a cluster, it’s currently 
> possible for one application to consume all the cluster’s resources, 
> effectively starving any other application trying to start. This is 
> particularly painful in a notebook environment where notebooks may be idle 
> for tens of minutes while the user is figuring out what to do next (or eating 
> their lunch). Ideally the application should give resources back to the 
> cluster when monitoring indicates other applications are pending.
> Before delving into the specifics of the solution. There are some workarounds 
> to this problem that are worth mentioning:
> * Set spark.dynamicAllocation.maxExecutors to a small value, so that users 
> are unlikely to use the entire cluster even when many of them are doing work. 
> This approach will hurt cluster utilization.
> * If using YARN, enable preemption and have each application (or 
> organization) run in a separate queue. The downside of this is that when YARN 
> preempts, it doesn't know anything about which executor it's killing. It 
> would just as likely kill a long running executor with cached data as one 
> that just spun up. Moreover, given a feature like 
> https://issues.apache.org/jira/browse/SPARK-21097 (preserving cached data on 
> executor decommission), YARN may not wait long enough between trying to 
> gracefully and forcefully shut down the executor. This would mean the blocks 
> that belonged to that executor would be lost and have to be recomputed.
> * Configure YARN to use the capacity scheduler with multiple scheduler 
> queues. Put high-priority notebook users into a high-priority queue. Prevents 
> high-priority users from being starved out by low-priority notebook users. 
> Does not prevent users in the same priority class from starving each other.
> Obviously any solution to this problem that depends on YARN would leave other 
> resource managers out in the cold. The solution proposed in this ticket will 
> afford spark clusters the flexibly to hook in different resource allocation 
> policies to fulfill their user's needs regardless of resource manager choice. 
> Initially the focus will be on users in a notebook environment. When 
> operating in a notebook environment with many users, the goal is fair 
> resource allocation. Given that all users will be using the same memory 
> configuration, this solution will focus primarily on fair sharing of cores.
> The fair resource allocation policy should pick executors to remove based on 
> three factors initially: idleness, presence of cached data, and uptime. The 
> policy will favor removing executors that are idle, short-lived, and have no 
> cached data. The policy will only preemptively remove executors if there are 
> pending applications or cores (otherwise the default dynamic allocation 
> timeout/removal process is followed). The policy will also allow an 
> application's resource consumption to expand based on cluster utilization. 
> For example if there are 3 applications running but 2 of them are idle, the 
> policy will allow a busy application with pending tasks to consume more than 
> 1/3rd of the the cluster's resources.
> More complexity could be added to take advantage of task/stage metrics, 
> histograms, and heuristics (i.e. favor removing executors running tasks that 
> are quick). The important thing here is to benchmark effectively before 
> adding complexity so we can measure the impact of the changes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (SPARK-20338) Spaces in spark.eventLog.dir are not correctly handled

[Marcelo Vanzin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-20338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcelo Vanzin resolved SPARK-20338.

   Resolution: Fixed
 Assignee: zuotingbing
Fix Version/s: 2.3.0

> Spaces in spark.eventLog.dir are not correctly handled
> --
>
> Key: SPARK-20338
> URL: https://issues.apache.org/jira/browse/SPARK-20338
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Affects Versions: 2.1.0
>Reporter: zuotingbing
>Assignee: zuotingbing
> Fix For: 2.3.0
>
>
> set spark.eventLog.dir=/home/mr/event log and submit an app ,we got error as 
> follows:
> 017-04-14 17:28:40,378 INFO org.apache.spark.SparkContext: Successfully 
> stopped SparkContext
> Exception in thread "main" ExitCodeException exitCode=1: chmod: cannot access 
> `/home/mr/event%20log/app-20170414172839-.inprogress': No such file or 
> directory
>   at org.apache.hadoop.util.Shell.runCommand(Shell.java:561)
>   at org.apache.hadoop.util.Shell.run(Shell.java:478)
>   at 
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:738)
>   at org.apache.hadoop.util.Shell.execCommand(Shell.java:831)
>   at org.apache.hadoop.util.Shell.execCommand(Shell.java:814)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:712)
>   at 
> org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:506)
>   at 
> org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:125)
>   at org.apache.spark.SparkContext.(SparkContext.scala:516)
>   at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2258)
>   at 
> org.apache.spark.sql.SparkSession$Builder$$anonfun$9.apply(SparkSession.scala:879)
>   at 
> org.apache.spark.sql.SparkSession$Builder$$anonfun$9.apply(SparkSession.scala:871)
>   at scala.Option.getOrElse(Option.scala:121)
>   at 
> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:871)
>   at 
> org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:58)
>   at 
> org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.(SparkSQLCLIDriver.scala:288)
>   at 
> org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:137)
>   at 
> org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736)
>   at 
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
>   at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
>   at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
>   at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-21122) Address starvation issues when dynamic allocation is enabled

[Craig Ingram (JIRA)]

Craig Ingram created SPARK-21122:


 Summary: Address starvation issues when dynamic allocation is 
enabled
 Key: SPARK-21122
 URL: https://issues.apache.org/jira/browse/SPARK-21122
 Project: Spark
  Issue Type: Improvement
  Components: Spark Core, YARN
Affects Versions: 2.2.0, 2.3.0
Reporter: Craig Ingram


When dynamic resource allocation is enabled on a cluster, it’s currently 
possible for one application to consume all the cluster’s resources, 
effectively starving any other application trying to start. This is 
particularly painful in a notebook environment where notebooks may be idle for 
tens of minutes while the user is figuring out what to do next (or eating their 
lunch). Ideally the application should give resources back to the cluster when 
monitoring indicates other applications are pending.

Before delving into the specifics of the solution. There are some workarounds 
to this problem that are worth mentioning:
* Set spark.dynamicAllocation.maxExecutors to a small value, so that users are 
unlikely to use the entire cluster even when many of them are doing work. This 
approach will hurt cluster utilization.
* If using YARN, enable preemption and have each application (or organization) 
run in a separate queue. The downside of this is that when YARN preempts, it 
doesn't know anything about which executor it's killing. It would just as 
likely kill a long running executor with cached data as one that just spun up. 
Moreover, given a feature like 
https://issues.apache.org/jira/browse/SPARK-21097 (preserving cached data on 
executor decommission), YARN may not wait long enough between trying to 
gracefully and forcefully shut down the executor. This would mean the blocks 
that belonged to that executor would be lost and have to be recomputed.
* Configure YARN to use the capacity scheduler with multiple scheduler queues. 
Put high-priority notebook users into a high-priority queue. Prevents 
high-priority users from being starved out by low-priority notebook users. Does 
not prevent users in the same priority class from starving each other.

Obviously any solution to this problem that depends on YARN would leave other 
resource managers out in the cold. The solution proposed in this ticket will 
afford spark clusters the flexibly to hook in different resource allocation 
policies to fulfill their user's needs regardless of resource manager choice. 
Initially the focus will be on users in a notebook environment. When operating 
in a notebook environment with many users, the goal is fair resource 
allocation. Given that all users will be using the same memory configuration, 
this solution will focus primarily on fair sharing of cores.

The fair resource allocation policy should pick executors to remove based on 
three factors initially: idleness, presence of cached data, and uptime. The 
policy will favor removing executors that are idle, short-lived, and have no 
cached data. The policy will only preemptively remove executors if there are 
pending applications or cores (otherwise the default dynamic allocation 
timeout/removal process is followed). The policy will also allow an 
application's resource consumption to expand based on cluster utilization. For 
example if there are 3 applications running but 2 of them are idle, the policy 
will allow a busy application with pending tasks to consume more than 1/3rd of 
the the cluster's resources.

More complexity could be added to take advantage of task/stage metrics, 
histograms, and heuristics (i.e. favor removing executors running tasks that 
are quick). The important thing here is to benchmark effectively before adding 
complexity so we can measure the impact of the changes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Resolved] (SPARK-21089) Table properties are not shown in DESC EXTENDED/FORMATTED

[Xiao Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Li resolved SPARK-21089.
-
Resolution: Fixed

> Table properties are not shown in DESC EXTENDED/FORMATTED
> -
>
> Key: SPARK-21089
> URL: https://issues.apache.org/jira/browse/SPARK-21089
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.2.0
>Reporter: Xiao Li
>Assignee: Xiao Li
>Priority: Critical
>
> Since both table properties and storage properties share the same key values, 
> table properties are not shown in the output of DESC EXTENDED/FORMATTED when 
> the storage properties are not empty. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-21089) Table properties are not shown in DESC EXTENDED/FORMATTED

[Xiao Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Li updated SPARK-21089:

Fix Version/s: 2.2.0

> Table properties are not shown in DESC EXTENDED/FORMATTED
> -
>
> Key: SPARK-21089
> URL: https://issues.apache.org/jira/browse/SPARK-21089
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.2.0
>Reporter: Xiao Li
>Assignee: Xiao Li
>Priority: Critical
> Fix For: 2.2.0
>
>
> Since both table properties and storage properties share the same key values, 
> table properties are not shown in the output of DESC EXTENDED/FORMATTED when 
> the storage properties are not empty. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-20977) NPE in CollectionAccumulator

[Leonardo Zanivan (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-20977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16052104#comment-16052104
 ] 

Leonardo Zanivan edited comment on SPARK-20977 at 6/16/17 4:35 PM:
---

I have the exactly same issue Spark 2.0.1 cluster under high load (master + 3 
workers)

{code:java}
17/06/16 13:17:30 ERROR Utils: Uncaught exception in thread 
heartbeat-receiver-event-loop-thread
java.lang.NullPointerException
at 
org.apache.spark.util.CollectionAccumulator.value(AccumulatorV2.scala:461)
at 
org.apache.spark.util.CollectionAccumulator.value(AccumulatorV2.scala:438)
at 
org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$5$$anonfun$6.apply(TaskSchedulerImpl.scala:396)
at 
org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$5$$anonfun$6.apply(TaskSchedulerImpl.scala:396)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at 
org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$5.apply(TaskSchedulerImpl.scala:396)
at 
org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$5.apply(TaskSchedulerImpl.scala:395)
at 
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at 
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at 
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at 
scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:186)
at 
org.apache.spark.scheduler.TaskSchedulerImpl.executorHeartbeatReceived(TaskSchedulerImpl.scala:395)
at 
org.apache.spark.HeartbeatReceiver$$anonfun$receiveAndReply$1$$anon$2$$anonfun$run$2.apply$mcV$sp(HeartbeatReceiver.scala:128)
at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1287)
at 
org.apache.spark.HeartbeatReceiver$$anonfun$receiveAndReply$1$$anon$2.run(HeartbeatReceiver.scala:127)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}



was (Author: panga):
I have the exactly same issue Spark 2.0.1 cluster (master + 3 workers)


{code:java}
17/06/16 13:17:30 ERROR Utils: Uncaught exception in thread 
heartbeat-receiver-event-loop-thread
java.lang.NullPointerException
at 
org.apache.spark.util.CollectionAccumulator.value(AccumulatorV2.scala:461)
at 
org.apache.spark.util.CollectionAccumulator.value(AccumulatorV2.scala:438)
at 
org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$5$$anonfun$6.apply(TaskSchedulerImpl.scala:396)
at 
org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$5$$anonfun$6.apply(TaskSchedulerImpl.scala:396)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at 
org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$5.apply(TaskSchedulerImpl.scala:396)
at 
org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$5.apply(TaskSchedulerImpl.scala:395)
at 
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at 

[jira] [Commented] (SPARK-20977) NPE in CollectionAccumulator

[Leonardo Zanivan (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-20977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16052104#comment-16052104
 ] 

Leonardo Zanivan commented on SPARK-20977:
--

I have the exactly same issue Spark 2.0.1 cluster (master + 3 workers)


{code:java}
17/06/16 13:17:30 ERROR Utils: Uncaught exception in thread 
heartbeat-receiver-event-loop-thread
java.lang.NullPointerException
at 
org.apache.spark.util.CollectionAccumulator.value(AccumulatorV2.scala:461)
at 
org.apache.spark.util.CollectionAccumulator.value(AccumulatorV2.scala:438)
at 
org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$5$$anonfun$6.apply(TaskSchedulerImpl.scala:396)
at 
org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$5$$anonfun$6.apply(TaskSchedulerImpl.scala:396)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at 
org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$5.apply(TaskSchedulerImpl.scala:396)
at 
org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$5.apply(TaskSchedulerImpl.scala:395)
at 
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at 
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at 
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at 
scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:186)
at 
org.apache.spark.scheduler.TaskSchedulerImpl.executorHeartbeatReceived(TaskSchedulerImpl.scala:395)
at 
org.apache.spark.HeartbeatReceiver$$anonfun$receiveAndReply$1$$anon$2$$anonfun$run$2.apply$mcV$sp(HeartbeatReceiver.scala:128)
at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1287)
at 
org.apache.spark.HeartbeatReceiver$$anonfun$receiveAndReply$1$$anon$2.run(HeartbeatReceiver.scala:127)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}


> NPE in CollectionAccumulator
> 
>
> Key: SPARK-20977
> URL: https://issues.apache.org/jira/browse/SPARK-20977
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Affects Versions: 2.1.1
> Environment: OpenJDK 64-Bit Server VM (25.71-b00) for linux-ppc64 JRE 
> (1.8.0-internal-centos_2017_04_25_01_11-b00), built on Apr 25 2017 01:24:21 
> by "centos" with gcc 6.3.1 20170110 (Advance-Toolchain-at10.0) IBM AT 10 
> branch
>Reporter: sharkd tu
>
> 17/06/03 13:39:31 ERROR Utils: Uncaught exception in thread 
> heartbeat-receiver-event-loop-thread
> java.lang.NullPointerException
>   at 
> org.apache.spark.util.CollectionAccumulator.value(AccumulatorV2.scala:464)
>   at 
> org.apache.spark.util.CollectionAccumulator.value(AccumulatorV2.scala:439)
>   at 
> org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$6$$anonfun$7.apply(TaskSchedulerImpl.scala:408)
>   at 
> org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$6$$anonfun$7.apply(TaskSchedulerImpl.scala:408)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>   at scala.collection.Iterator$class.foreach(Iterator.scala:893)
>   at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
>   at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>   at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>   at 

[jira] [Commented] (SPARK-20749) Built-in SQL Function Support - all variants of LEN[GTH]

[Apache Spark (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-20749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16052058#comment-16052058
 ] 

Apache Spark commented on SPARK-20749:
--

User 'wangyum' has created a pull request for this issue:
https://github.com/apache/spark/pull/18330

> Built-in SQL Function Support - all variants of LEN[GTH]
> 
>
> Key: SPARK-20749
> URL: https://issues.apache.org/jira/browse/SPARK-20749
> Project: Spark
>  Issue Type: Sub-task
>  Components: SQL
>Affects Versions: 2.2.0
>Reporter: Xiao Li
>Assignee: Kazuaki Ishizaki
>  Labels: starter
> Fix For: 2.3.0
>
>
> {noformat}
> LEN[GTH]()
> {noformat}
> The SQL 99 standard includes BIT_LENGTH(), CHAR_LENGTH(), and OCTET_LENGTH() 
> functions.
> We need to support all of them.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-18016) Code Generation: Constant Pool Past Limit for Wide/Nested Dataset

[Wenchen Fan (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16052057#comment-16052057
 ] 

Wenchen Fan commented on SPARK-18016:
-

cc [~aeskilson] do you wanna send a new PR to backport it?

> Code Generation: Constant Pool Past Limit for Wide/Nested Dataset
> -
>
> Key: SPARK-18016
> URL: https://issues.apache.org/jira/browse/SPARK-18016
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.1.0
>Reporter: Aleksander Eskilson
>Assignee: Aleksander Eskilson
> Fix For: 2.3.0
>
>
> When attempting to encode collections of large Java objects to Datasets 
> having very wide or deeply nested schemas, code generation can fail, yielding:
> {code}
> Caused by: org.codehaus.janino.JaninoRuntimeException: Constant pool for 
> class 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection
>  has grown past JVM limit of 0x
>   at 
> org.codehaus.janino.util.ClassFile.addToConstantPool(ClassFile.java:499)
>   at 
> org.codehaus.janino.util.ClassFile.addConstantNameAndTypeInfo(ClassFile.java:439)
>   at 
> org.codehaus.janino.util.ClassFile.addConstantMethodrefInfo(ClassFile.java:358)
>   at 
> org.codehaus.janino.UnitCompiler.writeConstantMethodrefInfo(UnitCompiler.java:4)
>   at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:4547)
>   at org.codehaus.janino.UnitCompiler.access$7500(UnitCompiler.java:206)
>   at 
> org.codehaus.janino.UnitCompiler$12.visitMethodInvocation(UnitCompiler.java:3774)
>   at 
> org.codehaus.janino.UnitCompiler$12.visitMethodInvocation(UnitCompiler.java:3762)
>   at org.codehaus.janino.Java$MethodInvocation.accept(Java.java:4328)
>   at org.codehaus.janino.UnitCompiler.compileGet(UnitCompiler.java:3762)
>   at 
> org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:4933)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3180)
>   at org.codehaus.janino.UnitCompiler.access$5000(UnitCompiler.java:206)
>   at 
> org.codehaus.janino.UnitCompiler$9.visitMethodInvocation(UnitCompiler.java:3151)
>   at 
> org.codehaus.janino.UnitCompiler$9.visitMethodInvocation(UnitCompiler.java:3139)
>   at org.codehaus.janino.Java$MethodInvocation.accept(Java.java:4328)
>   at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:3139)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2112)
>   at org.codehaus.janino.UnitCompiler.access$1700(UnitCompiler.java:206)
>   at 
> org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1377)
>   at 
> org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1370)
>   at org.codehaus.janino.Java$ExpressionStatement.accept(Java.java:2558)
>   at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370)
>   at 
> org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1450)
>   at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:2811)
>   at 
> org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1262)
>   at 
> org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1234)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:538)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:890)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:894)
>   at org.codehaus.janino.UnitCompiler.access$600(UnitCompiler.java:206)
>   at 
> org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:377)
>   at 
> org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:369)
>   at 
> org.codehaus.janino.Java$MemberClassDeclaration.accept(Java.java:1128)
>   at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369)
>   at 
> org.codehaus.janino.UnitCompiler.compileDeclaredMemberTypes(UnitCompiler.java:1209)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:564)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:420)
>   at org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:206)
>   at 
> org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:374)
>   at 
> org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:369)
>   at 
> org.codehaus.janino.Java$AbstractPackageMemberClassDeclaration.accept(Java.java:1309)
>   at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369)
>   at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:345)
>   at 
> 

[jira] [Resolved] (SPARK-21119) unset table properties should keep the table comment

[Xiao Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Li resolved SPARK-21119.
-
   Resolution: Fixed
Fix Version/s: 2.3.0

> unset table properties should keep the table comment
> 
>
> Key: SPARK-21119
> URL: https://issues.apache.org/jira/browse/SPARK-21119
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.2.0
>Reporter: Wenchen Fan
>Assignee: Wenchen Fan
> Fix For: 2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-19909) Batches will fail in case that temporary checkpoint dir is on local file system while metadata dir is on HDFS

[Apache Spark (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-19909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051981#comment-16051981
 ] 

Apache Spark commented on SPARK-19909:
--

User 'mgaido91' has created a pull request for this issue:
https://github.com/apache/spark/pull/18329

> Batches will fail in case that temporary checkpoint dir is on local file 
> system while metadata dir is on HDFS
> -
>
> Key: SPARK-19909
> URL: https://issues.apache.org/jira/browse/SPARK-19909
> Project: Spark
>  Issue Type: Bug
>  Components: Structured Streaming
>Affects Versions: 2.2.0
>Reporter: Kousuke Saruta
>Priority: Minor
>
> When we try to run Structured Streaming in local mode but use HDFS for the 
> storage, batches will be fail because of error like as follows.
> {code}
> val handle = stream.writeStream.format("console").start()
> 17/03/09 16:54:45 ERROR StreamMetadata: Error writing stream metadata 
> StreamMetadata(fc07a0b1-5423-483e-a59d-b2206a49491e) to 
> /private/var/folders/4y/tmspvv353y59p3w4lknrf7ccgn/T/temporary-79d4fe05-4301-4b6d-a902-dff642d0ddca/metadata
> org.apache.hadoop.security.AccessControlException: Permission denied: 
> user=kou, access=WRITE, 
> inode="/private/var/folders/4y/tmspvv353y59p3w4lknrf7ccgn/T/temporary-79d4fe05-4301-4b6d-a902-dff642d0ddca/metadata":hdfs:supergroup:drwxr-xr-x
> {code}
> It's because that a temporary checkpoint directory is created on local file 
> system but metadata whose path is based on the checkpoint directory will be 
> created on HDFS.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-19909) Batches will fail in case that temporary checkpoint dir is on local file system while metadata dir is on HDFS

[Marco Gaido (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-19909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051963#comment-16051963
 ] 

Marco Gaido commented on SPARK-19909:
-

IMHO the best option to deal with this problem is to force the setting of the 
{{checkpointLocation}} if the default filesystem (the one the metadata dir is 
written on) is different from the filesystem for the temporary directory.
In this way, if the {{checkpointLocation}}  is not set, we get a much more 
meaningful exception that suggests the proper solution.

I am creating the PR with this implementation.

> Batches will fail in case that temporary checkpoint dir is on local file 
> system while metadata dir is on HDFS
> -
>
> Key: SPARK-19909
> URL: https://issues.apache.org/jira/browse/SPARK-19909
> Project: Spark
>  Issue Type: Bug
>  Components: Structured Streaming
>Affects Versions: 2.2.0
>Reporter: Kousuke Saruta
>Priority: Minor
>
> When we try to run Structured Streaming in local mode but use HDFS for the 
> storage, batches will be fail because of error like as follows.
> {code}
> val handle = stream.writeStream.format("console").start()
> 17/03/09 16:54:45 ERROR StreamMetadata: Error writing stream metadata 
> StreamMetadata(fc07a0b1-5423-483e-a59d-b2206a49491e) to 
> /private/var/folders/4y/tmspvv353y59p3w4lknrf7ccgn/T/temporary-79d4fe05-4301-4b6d-a902-dff642d0ddca/metadata
> org.apache.hadoop.security.AccessControlException: Permission denied: 
> user=kou, access=WRITE, 
> inode="/private/var/folders/4y/tmspvv353y59p3w4lknrf7ccgn/T/temporary-79d4fe05-4301-4b6d-a902-dff642d0ddca/metadata":hdfs:supergroup:drwxr-xr-x
> {code}
> It's because that a temporary checkpoint directory is created on local file 
> system but metadata whose path is based on the checkpoint directory will be 
> created on HDFS.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-18016) Code Generation: Constant Pool Past Limit for Wide/Nested Dataset

[Divya (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051950#comment-16051950
 ] 

Divya commented on SPARK-18016:
---

Is a backport fix available for 2.1.0?

> Code Generation: Constant Pool Past Limit for Wide/Nested Dataset
> -
>
> Key: SPARK-18016
> URL: https://issues.apache.org/jira/browse/SPARK-18016
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.1.0
>Reporter: Aleksander Eskilson
>Assignee: Aleksander Eskilson
> Fix For: 2.3.0
>
>
> When attempting to encode collections of large Java objects to Datasets 
> having very wide or deeply nested schemas, code generation can fail, yielding:
> {code}
> Caused by: org.codehaus.janino.JaninoRuntimeException: Constant pool for 
> class 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection
>  has grown past JVM limit of 0x
>   at 
> org.codehaus.janino.util.ClassFile.addToConstantPool(ClassFile.java:499)
>   at 
> org.codehaus.janino.util.ClassFile.addConstantNameAndTypeInfo(ClassFile.java:439)
>   at 
> org.codehaus.janino.util.ClassFile.addConstantMethodrefInfo(ClassFile.java:358)
>   at 
> org.codehaus.janino.UnitCompiler.writeConstantMethodrefInfo(UnitCompiler.java:4)
>   at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:4547)
>   at org.codehaus.janino.UnitCompiler.access$7500(UnitCompiler.java:206)
>   at 
> org.codehaus.janino.UnitCompiler$12.visitMethodInvocation(UnitCompiler.java:3774)
>   at 
> org.codehaus.janino.UnitCompiler$12.visitMethodInvocation(UnitCompiler.java:3762)
>   at org.codehaus.janino.Java$MethodInvocation.accept(Java.java:4328)
>   at org.codehaus.janino.UnitCompiler.compileGet(UnitCompiler.java:3762)
>   at 
> org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:4933)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3180)
>   at org.codehaus.janino.UnitCompiler.access$5000(UnitCompiler.java:206)
>   at 
> org.codehaus.janino.UnitCompiler$9.visitMethodInvocation(UnitCompiler.java:3151)
>   at 
> org.codehaus.janino.UnitCompiler$9.visitMethodInvocation(UnitCompiler.java:3139)
>   at org.codehaus.janino.Java$MethodInvocation.accept(Java.java:4328)
>   at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:3139)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2112)
>   at org.codehaus.janino.UnitCompiler.access$1700(UnitCompiler.java:206)
>   at 
> org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1377)
>   at 
> org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1370)
>   at org.codehaus.janino.Java$ExpressionStatement.accept(Java.java:2558)
>   at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370)
>   at 
> org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1450)
>   at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:2811)
>   at 
> org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1262)
>   at 
> org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1234)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:538)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:890)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:894)
>   at org.codehaus.janino.UnitCompiler.access$600(UnitCompiler.java:206)
>   at 
> org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:377)
>   at 
> org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:369)
>   at 
> org.codehaus.janino.Java$MemberClassDeclaration.accept(Java.java:1128)
>   at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369)
>   at 
> org.codehaus.janino.UnitCompiler.compileDeclaredMemberTypes(UnitCompiler.java:1209)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:564)
>   at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:420)
>   at org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:206)
>   at 
> org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:374)
>   at 
> org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:369)
>   at 
> org.codehaus.janino.Java$AbstractPackageMemberClassDeclaration.accept(Java.java:1309)
>   at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369)
>   at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:345)
>   at 
> org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:396)
> 

[jira] [Commented] (SPARK-20760) Memory Leak of RDD blocks

[Jose Soltren (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-20760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051942#comment-16051942
 ] 

Jose Soltren commented on SPARK-20760:
--

Hello Ravi - current thinking is that this issue is a duplicate of SPARK-18991. 
I'll have a look at your JIRA and comment there if I have any further 
information. Thanks.

> Memory Leak of RDD blocks 
> --
>
> Key: SPARK-20760
> URL: https://issues.apache.org/jira/browse/SPARK-20760
> Project: Spark
>  Issue Type: Bug
>  Components: Block Manager
>Affects Versions: 2.1.0
> Environment: Spark 2.1.0
>Reporter: Binzi Cao
> Attachments: RDD blocks in spark 2.1.1.png, RDD Blocks .png, Storage 
> in spark 2.1.1.png
>
>
> Memory leak for RDD blocks for a long time running rdd process.
> We  have a long term running application, which is doing computations of 
> RDDs. and we found the RDD blocks are keep increasing in the spark ui page. 
> The rdd blocks and memory usage do not mach the cached rdds and memory. It 
> looks like spark keeps old rdd in memory and never released it or never got a 
> chance to release it. The job will eventually die of out of memory. 
> In addition, I'm not seeing this issue in spark 1.6. We are seeing the same 
> issue in Yarn Cluster mode both in kafka streaming and batch applications. 
> The issue in streaming is similar, however, it seems the rdd blocks grows a 
> bit slower than batch jobs. 
> The below is the sample code and it is reproducible by justing running it in 
> local mode. 
> Scala file:
> {code}
> import scala.concurrent.duration.Duration
> import scala.util.{Try, Failure, Success}
> import org.apache.spark.SparkConf
> import org.apache.spark.SparkContext
> import org.apache.spark.rdd.RDD
> import scala.concurrent._
> import ExecutionContext.Implicits.global
> case class Person(id: String, name: String)
> object RDDApp {
>   def run(sc: SparkContext) = {
> while (true) {
>   val r = scala.util.Random
>   val data = (1 to r.nextInt(100)).toList.map { a =>
> Person(a.toString, a.toString)
>   }
>   val rdd = sc.parallelize(data)
>   rdd.cache
>   println("running")
>   val a = (1 to 100).toList.map { x =>
> Future(rdd.filter(_.id == x.toString).collect)
>   }
>   a.foreach { f =>
> println(Await.ready(f, Duration.Inf).value.get)
>   }
>   rdd.unpersist()
> }
>   }
>   def main(args: Array[String]): Unit = {
>val conf = new SparkConf().setAppName("test")
> val sc   = new SparkContext(conf)
> run(sc)
>   }
> }
> {code}
> build sbt file:
> {code}
> name := "RDDTest"
> version := "0.1.1"
> scalaVersion := "2.11.5"
> libraryDependencies ++= Seq (
> "org.scalaz" %% "scalaz-core" % "7.2.0",
> "org.scalaz" %% "scalaz-concurrent" % "7.2.0",
> "org.apache.spark" % "spark-core_2.11" % "2.1.0" % "provided",
> "org.apache.spark" % "spark-hive_2.11" % "2.1.0" % "provided"
>   )
> addCompilerPlugin("org.spire-math" %% "kind-projector" % "0.7.1")
> mainClass in assembly := Some("RDDApp")
> test in assembly := {}
> {code}
> To reproduce it: 
> Just 
> {code}
> spark-2.1.0-bin-hadoop2.7/bin/spark-submit   --driver-memory 4G \
> --executor-memory 4G \
> --executor-cores 1 \
> --num-executors 1 \
> --class "RDDApp" --master local[4] RDDTest-assembly-0.1.1.jar
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-21121) Set up StorageLevel for CACHE TABLE command

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-21121:


Assignee: (was: Apache Spark)

> Set up StorageLevel for CACHE TABLE command
> ---
>
> Key: SPARK-21121
> URL: https://issues.apache.org/jira/browse/SPARK-21121
> Project: Spark
>  Issue Type: New Feature
>  Components: SQL
>Affects Versions: 2.1.1
>Reporter: Oleg Danilov
>Priority: Minor
>
> Currently, "CACHE TABLE" always uses the default MEMORY_AND_DISK storage 
> level. We can add a possibility to specify it using variable, let say, 
> spark.sql.inMemoryColumnarStorage.level. It will give user a chance to fit 
> data into the memory with using MEMORY_AND_DISK_SER storage level.
> Going to submit PR for this change.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-21121) Set up StorageLevel for CACHE TABLE command

[Apache Spark (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-21121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051910#comment-16051910
 ] 

Apache Spark commented on SPARK-21121:
--

User 'dosoft' has created a pull request for this issue:
https://github.com/apache/spark/pull/18328

> Set up StorageLevel for CACHE TABLE command
> ---
>
> Key: SPARK-21121
> URL: https://issues.apache.org/jira/browse/SPARK-21121
> Project: Spark
>  Issue Type: New Feature
>  Components: SQL
>Affects Versions: 2.1.1
>Reporter: Oleg Danilov
>Priority: Minor
>
> Currently, "CACHE TABLE" always uses the default MEMORY_AND_DISK storage 
> level. We can add a possibility to specify it using variable, let say, 
> spark.sql.inMemoryColumnarStorage.level. It will give user a chance to fit 
> data into the memory with using MEMORY_AND_DISK_SER storage level.
> Going to submit PR for this change.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-21121) Set up StorageLevel for CACHE TABLE command

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-21121:


Assignee: Apache Spark

> Set up StorageLevel for CACHE TABLE command
> ---
>
> Key: SPARK-21121
> URL: https://issues.apache.org/jira/browse/SPARK-21121
> Project: Spark
>  Issue Type: New Feature
>  Components: SQL
>Affects Versions: 2.1.1
>Reporter: Oleg Danilov
>Assignee: Apache Spark
>Priority: Minor
>
> Currently, "CACHE TABLE" always uses the default MEMORY_AND_DISK storage 
> level. We can add a possibility to specify it using variable, let say, 
> spark.sql.inMemoryColumnarStorage.level. It will give user a chance to fit 
> data into the memory with using MEMORY_AND_DISK_SER storage level.
> Going to submit PR for this change.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-11083) insert overwrite table failed when beeline reconnect

[Sean Owen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-11083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Owen updated SPARK-11083:
--
Fix Version/s: (was: 1.6.0)

> insert overwrite table failed when beeline reconnect
> 
>
> Key: SPARK-11083
> URL: https://issues.apache.org/jira/browse/SPARK-11083
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
> Environment: Spark: master branch
> Hadoop: 2.7.1
> JDK: 1.8.0_60
>Reporter: Weizhong
>Assignee: Davies Liu
>
> 1. Start Thriftserver
> 2. Use beeline connect to thriftserver, then execute "insert overwrite 
> table_name ..." clause -- success
> 3. Exit beelin
> 4. Reconnect to thriftserver, and then execute "insert overwrite table_name 
> ..." clause. -- failed
> {noformat}
> 15/10/13 18:44:35 ERROR SparkExecuteStatementOperation: Error executing 
> query, currentState RUNNING, 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.spark.sql.hive.client.Shim_v1_2.loadDynamicPartitions(HiveShim.scala:520)
>   at 
> org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$loadDynamicPartitions$1.apply$mcV$sp(ClientWrapper.scala:506)
>   at 
> org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$loadDynamicPartitions$1.apply(ClientWrapper.scala:506)
>   at 
> org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$loadDynamicPartitions$1.apply(ClientWrapper.scala:506)
>   at 
> org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$withHiveState$1.apply(ClientWrapper.scala:256)
>   at 
> org.apache.spark.sql.hive.client.ClientWrapper.retryLocked(ClientWrapper.scala:211)
>   at 
> org.apache.spark.sql.hive.client.ClientWrapper.withHiveState(ClientWrapper.scala:248)
>   at 
> org.apache.spark.sql.hive.client.ClientWrapper.loadDynamicPartitions(ClientWrapper.scala:505)
>   at 
> org.apache.spark.sql.hive.execution.InsertIntoHiveTable.sideEffectResult$lzycompute(InsertIntoHiveTable.scala:225)
>   at 
> org.apache.spark.sql.hive.execution.InsertIntoHiveTable.sideEffectResult(InsertIntoHiveTable.scala:127)
>   at 
> org.apache.spark.sql.hive.execution.InsertIntoHiveTable.doExecute(InsertIntoHiveTable.scala:276)
>   at 
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:140)
>   at 
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:138)
>   at 
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
>   at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:138)
>   at 
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:58)
>   at 
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:58)
>   at org.apache.spark.sql.DataFrame.(DataFrame.scala:144)
>   at org.apache.spark.sql.DataFrame.(DataFrame.scala:129)
>   at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:51)
>   at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:739)
>   at 
> org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.runInternal(SparkExecuteStatementOperation.scala:224)
>   at 
> org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:171)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>   at 
> org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1.run(SparkExecuteStatementOperation.scala:182)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:744)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move 
> source 
> hdfs://9.91.8.214:9000/user/hive/warehouse/tpcds_bin_partitioned_orc_2.db/catalog_returns/.hive-staging_hive_2015-10-13_18-44-17_606_2400736035447406540-2/-ext-1/cr_returned_date=2003-08-27/part-00048
>  to destination 
> 

[jira] [Updated] (SPARK-21054) Reset Command support reset specific property which is compatible with Hive

[Sean Owen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Owen updated SPARK-21054:
--
Target Version/s:   (was: 2.1.1)

> Reset Command support reset specific property which is compatible with Hive
> ---
>
> Key: SPARK-21054
> URL: https://issues.apache.org/jira/browse/SPARK-21054
> Project: Spark
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 2.1.1
>Reporter: Wang Haihua
>Priority: Minor
>
> When we transfer our SQL in production from Hive to SparkSQL,  we found some 
> SQL like {{reset dfs.blocksize;}} will throw error in SparkSQL.
> We found cause is, 
> after SPARK-15330, Spark 2.0.0 support Reset command which will Reset all 
> properties and don't support {{reset key}}. 
> But after Hive 2.1.1 (HIVE-14418), Hive also support Reset Command for 
> specific property. 
> SQL compatibility with Hive will ease user more, and lower the cost that 
> transfer SQL from Hive to SparkSQL.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-21054) Reset Command support reset specific property which is compatible with Hive

[Sean Owen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Owen updated SPARK-21054:
--
Fix Version/s: (was: 2.1.1)

> Reset Command support reset specific property which is compatible with Hive
> ---
>
> Key: SPARK-21054
> URL: https://issues.apache.org/jira/browse/SPARK-21054
> Project: Spark
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 2.1.1
>Reporter: Wang Haihua
>Priority: Minor
>
> When we transfer our SQL in production from Hive to SparkSQL,  we found some 
> SQL like {{reset dfs.blocksize;}} will throw error in SparkSQL.
> We found cause is, 
> after SPARK-15330, Spark 2.0.0 support Reset command which will Reset all 
> properties and don't support {{reset key}}. 
> But after Hive 2.1.1 (HIVE-14418), Hive also support Reset Command for 
> specific property. 
> SQL compatibility with Hive will ease user more, and lower the cost that 
> transfer SQL from Hive to SparkSQL.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-21121) Set up StorageLevel for CACHE TABLE command

[Oleg Danilov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleg Danilov updated SPARK-21121:
-
Description: 
Currently, "CACHE TABLE" always uses the default MEMORY_AND_DISK storage level. 
We can add a possibility to specify it using variable, let say, 
spark.sql.inMemoryColumnarStorage.level. It will give user a chance to fit data 
into the memory with using MEMORY_AND_DISK_SER storage level.

Going to submit PR for this change.


  was:
Currently, "CACHE TABLE" always uses the default MEMORY_AND_DISK storage level. 
We can add a possibility to specify it using variable, let say, 
spark.sql.inMemoryColumnarStorage.compressed. It will give user a chance to fit 
data into the memory with using MEMORY_AND_DISK_SER storage level.

Going to submit PR for this change.



> Set up StorageLevel for CACHE TABLE command
> ---
>
> Key: SPARK-21121
> URL: https://issues.apache.org/jira/browse/SPARK-21121
> Project: Spark
>  Issue Type: New Feature
>  Components: SQL
>Affects Versions: 2.1.1
>Reporter: Oleg Danilov
>Priority: Minor
>
> Currently, "CACHE TABLE" always uses the default MEMORY_AND_DISK storage 
> level. We can add a possibility to specify it using variable, let say, 
> spark.sql.inMemoryColumnarStorage.level. It will give user a chance to fit 
> data into the memory with using MEMORY_AND_DISK_SER storage level.
> Going to submit PR for this change.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-21121) Set up StorageLevel for CACHE TABLE command

[Oleg Danilov (JIRA)]

Oleg Danilov created SPARK-21121:


 Summary: Set up StorageLevel for CACHE TABLE command
 Key: SPARK-21121
 URL: https://issues.apache.org/jira/browse/SPARK-21121
 Project: Spark
  Issue Type: New Feature
  Components: SQL
Affects Versions: 2.1.1
Reporter: Oleg Danilov
Priority: Minor


Currently, "CACHE TABLE" always uses the default MEMORY_AND_DISK storage level. 
We can add a possibility to specify it using variable, let say, 
spark.sql.inMemoryColumnarStorage.compressed. It will give user a chance to fit 
data into the memory with using MEMORY_AND_DISK_SER storage level.

Going to submit PR for this change.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-20760) Memory Leak of RDD blocks

[Ravi Teja (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-20760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051897#comment-16051897
 ] 

Ravi Teja commented on SPARK-20760:
---

We are also facing similar issue but in spark streaming. Please see this issue 
[SPARK-19644|https://issues.apache.org/jira/browse/SPARK-19644] thread.
Can you please tell me if you found the solution to this issue?

> Memory Leak of RDD blocks 
> --
>
> Key: SPARK-20760
> URL: https://issues.apache.org/jira/browse/SPARK-20760
> Project: Spark
>  Issue Type: Bug
>  Components: Block Manager
>Affects Versions: 2.1.0
> Environment: Spark 2.1.0
>Reporter: Binzi Cao
> Attachments: RDD blocks in spark 2.1.1.png, RDD Blocks .png, Storage 
> in spark 2.1.1.png
>
>
> Memory leak for RDD blocks for a long time running rdd process.
> We  have a long term running application, which is doing computations of 
> RDDs. and we found the RDD blocks are keep increasing in the spark ui page. 
> The rdd blocks and memory usage do not mach the cached rdds and memory. It 
> looks like spark keeps old rdd in memory and never released it or never got a 
> chance to release it. The job will eventually die of out of memory. 
> In addition, I'm not seeing this issue in spark 1.6. We are seeing the same 
> issue in Yarn Cluster mode both in kafka streaming and batch applications. 
> The issue in streaming is similar, however, it seems the rdd blocks grows a 
> bit slower than batch jobs. 
> The below is the sample code and it is reproducible by justing running it in 
> local mode. 
> Scala file:
> {code}
> import scala.concurrent.duration.Duration
> import scala.util.{Try, Failure, Success}
> import org.apache.spark.SparkConf
> import org.apache.spark.SparkContext
> import org.apache.spark.rdd.RDD
> import scala.concurrent._
> import ExecutionContext.Implicits.global
> case class Person(id: String, name: String)
> object RDDApp {
>   def run(sc: SparkContext) = {
> while (true) {
>   val r = scala.util.Random
>   val data = (1 to r.nextInt(100)).toList.map { a =>
> Person(a.toString, a.toString)
>   }
>   val rdd = sc.parallelize(data)
>   rdd.cache
>   println("running")
>   val a = (1 to 100).toList.map { x =>
> Future(rdd.filter(_.id == x.toString).collect)
>   }
>   a.foreach { f =>
> println(Await.ready(f, Duration.Inf).value.get)
>   }
>   rdd.unpersist()
> }
>   }
>   def main(args: Array[String]): Unit = {
>val conf = new SparkConf().setAppName("test")
> val sc   = new SparkContext(conf)
> run(sc)
>   }
> }
> {code}
> build sbt file:
> {code}
> name := "RDDTest"
> version := "0.1.1"
> scalaVersion := "2.11.5"
> libraryDependencies ++= Seq (
> "org.scalaz" %% "scalaz-core" % "7.2.0",
> "org.scalaz" %% "scalaz-concurrent" % "7.2.0",
> "org.apache.spark" % "spark-core_2.11" % "2.1.0" % "provided",
> "org.apache.spark" % "spark-hive_2.11" % "2.1.0" % "provided"
>   )
> addCompilerPlugin("org.spire-math" %% "kind-projector" % "0.7.1")
> mainClass in assembly := Some("RDDApp")
> test in assembly := {}
> {code}
> To reproduce it: 
> Just 
> {code}
> spark-2.1.0-bin-hadoop2.7/bin/spark-submit   --driver-memory 4G \
> --executor-memory 4G \
> --executor-cores 1 \
> --num-executors 1 \
> --class "RDDApp" --master local[4] RDDTest-assembly-0.1.1.jar
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-20984) Reading back from ORC format gives error on big endian systems.

[Adam Roberts (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-20984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051878#comment-16051878
 ] 

Adam Roberts commented on SPARK-20984:
--

For the Spark binary package we ship in IBM, we use snappy-0.4: 
https://mvnrepository.com/artifact/org.iq80.snappy/snappy/0.4 as a result of 
needing 
[this|https://github.com/dain/snappy/commit/a7c9c3cdbd6a0d997975ccbde5c946e07c1684e2]
 fix

As it's a package we don't directly reference in our pom, this is done after 
building (essentially the snappy used is "patched" to prevent the problem). If 
there's a better way to do this please let me know, IIRC this is downloaded and 
used as part of another package we do refer to in the main pom.

> Reading back from ORC format gives error on big endian systems.
> ---
>
> Key: SPARK-20984
> URL: https://issues.apache.org/jira/browse/SPARK-20984
> Project: Spark
>  Issue Type: Bug
>  Components: Input/Output
>Affects Versions: 2.0.0
> Environment: Redhat 7 on power 7 Big endian platform.
> [testuser@soe10-vm12 spark]$ cat /etc/redhat-
> redhat-access-insights/ redhat-release
> [testuser@soe10-vm12 spark]$ cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 7.2 (Maipo)
> [testuser@soe10-vm12 spark]$ lscpu
> Architecture:  ppc64
> CPU op-mode(s):32-bit, 64-bit
> Byte Order:Big Endian
> CPU(s):8
> On-line CPU(s) list:   0-7
> Thread(s) per core:1
> Core(s) per socket:1
> Socket(s): 8
> NUMA node(s):  1
> Model: IBM pSeries (emulated by qemu)
> L1d cache: 32K
> L1i cache: 32K
> NUMA node0 CPU(s): 0-7
> [testuser@soe10-vm12 spark]$
>Reporter: Mahesh
>  Labels: big-endian
>
> All orc test cases seem to be failing here. Looks like spark is not able to 
> read back what is written. Following is a way to check it on spark shell. I 
> am also pasting the test case which probably passes on x86. 
> All test cases in OrcHadoopFsRelationSuite.scala are failing.
>  test("SPARK-12218: 'Not' is included in ORC filter pushdown") {
> import testImplicits._
> withSQLConf(SQLConf.ORC_FILTER_PUSHDOWN_ENABLED.key -> "true") {
>   withTempPath { dir =>
> val path = s"${dir.getCanonicalPath}/table1"
> (1 to 5).map(i => (i, (i % 2).toString)).toDF("a", 
> "b").write.orc(path)
> checkAnswer(
>   spark.read.orc(path).where("not (a = 2) or not(b in ('1'))"),
>   (1 to 5).map(i => Row(i, (i % 2).toString)))
> checkAnswer(
>   spark.read.orc(path).where("not (a = 2 and b in ('1'))"),
>   (1 to 5).map(i => Row(i, (i % 2).toString)))
>   }
> }
>   }
> Same can be reproduced on spark shell
> **Create a DF and write it in orc
> scala> (1 to 5).map(i => (i, (i % 2).toString)).toDF("a", 
> "b").write.orc("test")
> **Now try to read it back
> scala> spark.read.orc("test").where("not (a = 2) or not(b in ('1'))").show
> 17/06/05 04:20:48 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
> org.iq80.snappy.CorruptionException: Invalid copy offset for opcode starting 
> at 13
> at 
> org.iq80.snappy.SnappyDecompressor.decompressAllTags(SnappyDecompressor.java:165)
> at 
> org.iq80.snappy.SnappyDecompressor.uncompress(SnappyDecompressor.java:76)
> at org.iq80.snappy.Snappy.uncompress(Snappy.java:43)
> at 
> org.apache.hadoop.hive.ql.io.orc.SnappyCodec.decompress(SnappyCodec.java:71)
> at 
> org.apache.hadoop.hive.ql.io.orc.InStream$CompressedStream.readHeader(InStream.java:214)
> at 
> org.apache.hadoop.hive.ql.io.orc.InStream$CompressedStream.read(InStream.java:238)
> at java.io.InputStream.read(InputStream.java:101)
> at 
> org.apache.hive.com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:737)
> at 
> org.apache.hive.com.google.protobuf.CodedInputStream.isAtEnd(CodedInputStream.java:701)
> at 
> org.apache.hive.com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:99)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcProto$StripeFooter.(OrcProto.java:10661)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcProto$StripeFooter.(OrcProto.java:10625)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcProto$StripeFooter$1.parsePartialFrom(OrcProto.java:10730)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcProto$StripeFooter$1.parsePartialFrom(OrcProto.java:10725)
> at 
> org.apache.hive.com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
> at 
> org.apache.hive.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
> at 
> org.apache.hive.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
>  

[jira] [Commented] (SPARK-20984) Reading back from ORC format gives error on big endian systems.

[Mahesh (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-20984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051861#comment-16051861
 ] 

Mahesh commented on SPARK-20984:


[~aroberts] Mentioned that snappy 0.2 version needs to be used.

> Reading back from ORC format gives error on big endian systems.
> ---
>
> Key: SPARK-20984
> URL: https://issues.apache.org/jira/browse/SPARK-20984
> Project: Spark
>  Issue Type: Bug
>  Components: Input/Output
>Affects Versions: 2.0.0
> Environment: Redhat 7 on power 7 Big endian platform.
> [testuser@soe10-vm12 spark]$ cat /etc/redhat-
> redhat-access-insights/ redhat-release
> [testuser@soe10-vm12 spark]$ cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 7.2 (Maipo)
> [testuser@soe10-vm12 spark]$ lscpu
> Architecture:  ppc64
> CPU op-mode(s):32-bit, 64-bit
> Byte Order:Big Endian
> CPU(s):8
> On-line CPU(s) list:   0-7
> Thread(s) per core:1
> Core(s) per socket:1
> Socket(s): 8
> NUMA node(s):  1
> Model: IBM pSeries (emulated by qemu)
> L1d cache: 32K
> L1i cache: 32K
> NUMA node0 CPU(s): 0-7
> [testuser@soe10-vm12 spark]$
>Reporter: Mahesh
>  Labels: big-endian
>
> All orc test cases seem to be failing here. Looks like spark is not able to 
> read back what is written. Following is a way to check it on spark shell. I 
> am also pasting the test case which probably passes on x86. 
> All test cases in OrcHadoopFsRelationSuite.scala are failing.
>  test("SPARK-12218: 'Not' is included in ORC filter pushdown") {
> import testImplicits._
> withSQLConf(SQLConf.ORC_FILTER_PUSHDOWN_ENABLED.key -> "true") {
>   withTempPath { dir =>
> val path = s"${dir.getCanonicalPath}/table1"
> (1 to 5).map(i => (i, (i % 2).toString)).toDF("a", 
> "b").write.orc(path)
> checkAnswer(
>   spark.read.orc(path).where("not (a = 2) or not(b in ('1'))"),
>   (1 to 5).map(i => Row(i, (i % 2).toString)))
> checkAnswer(
>   spark.read.orc(path).where("not (a = 2 and b in ('1'))"),
>   (1 to 5).map(i => Row(i, (i % 2).toString)))
>   }
> }
>   }
> Same can be reproduced on spark shell
> **Create a DF and write it in orc
> scala> (1 to 5).map(i => (i, (i % 2).toString)).toDF("a", 
> "b").write.orc("test")
> **Now try to read it back
> scala> spark.read.orc("test").where("not (a = 2) or not(b in ('1'))").show
> 17/06/05 04:20:48 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
> org.iq80.snappy.CorruptionException: Invalid copy offset for opcode starting 
> at 13
> at 
> org.iq80.snappy.SnappyDecompressor.decompressAllTags(SnappyDecompressor.java:165)
> at 
> org.iq80.snappy.SnappyDecompressor.uncompress(SnappyDecompressor.java:76)
> at org.iq80.snappy.Snappy.uncompress(Snappy.java:43)
> at 
> org.apache.hadoop.hive.ql.io.orc.SnappyCodec.decompress(SnappyCodec.java:71)
> at 
> org.apache.hadoop.hive.ql.io.orc.InStream$CompressedStream.readHeader(InStream.java:214)
> at 
> org.apache.hadoop.hive.ql.io.orc.InStream$CompressedStream.read(InStream.java:238)
> at java.io.InputStream.read(InputStream.java:101)
> at 
> org.apache.hive.com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:737)
> at 
> org.apache.hive.com.google.protobuf.CodedInputStream.isAtEnd(CodedInputStream.java:701)
> at 
> org.apache.hive.com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:99)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcProto$StripeFooter.(OrcProto.java:10661)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcProto$StripeFooter.(OrcProto.java:10625)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcProto$StripeFooter$1.parsePartialFrom(OrcProto.java:10730)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcProto$StripeFooter$1.parsePartialFrom(OrcProto.java:10725)
> at 
> org.apache.hive.com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
> at 
> org.apache.hive.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
> at 
> org.apache.hive.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
> at 
> org.apache.hive.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcProto$StripeFooter.parseFrom(OrcProto.java:10937)
> at 
> org.apache.hadoop.hive.ql.io.orc.MetadataReader.readStripeFooter(MetadataReader.java:113)
> at 
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readStripeFooter(RecordReaderImpl.java:228)
> at 
> 

[jira] [Comment Edited] (SPARK-18649) sc.textFile(my_file).collect() raises socket.timeout on large files

[Serge Vilvovsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-18649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050940#comment-16050940
 ] 

Serge Vilvovsky edited comment on SPARK-18649 at 6/16/17 12:13 PM:
---

Does anybody work on the issue? Same problem (PythonRDD: Error while sending 
iterator
java.net.SocketException: Connection reset) occurs for me on collect() of the 
large elasticsearch RDD using Spark 2.1.1


was (Author: sergevil):
Does anybody work on the issue? Same problem (PythonRDD: Error while sending 
iterator
java.net.SocketException: Connection reset) occurs for me on collect() of the 
large elasticsearch RDD. 

> sc.textFile(my_file).collect() raises socket.timeout on large files
> ---
>
> Key: SPARK-18649
> URL: https://issues.apache.org/jira/browse/SPARK-18649
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
> Environment: PySpark version 1.6.2
>Reporter: Erik Cederstrand
>
> I'm trying to load a file into the driver with this code:
> contents = sc.textFile('hdfs://path/to/big_file.csv').collect()
> Loading into the driver instead of creating a distributed RDD is intentional 
> in this case. The file is ca. 6GB, and I have adjusted driver memory 
> accordingly to fit the local data. After some time, my spark/submitted job 
> crashes with the stack trace below.
> I have traced this to pyspark/rdd.py where the _load_from_socket() method 
> creates a socket with a hard-coded timeout of 3 seconds (this code is also 
> present in HEAD although I'm on PySpark 1.6.2). Raising this hard-coded value 
> to e.g. 600 lets me read the entire file.
> Is there any reason that this value does not use e.g. the 
> 'spark.network.timeout' setting instead?
> Traceback (most recent call last):
>   File "my_textfile_test.py", line 119, in 
> contents = sc.textFile('hdfs://path/to/file.csv').collect()
>   File "/usr/hdp/2.5.0.0-1245/spark/python/lib/pyspark.zip/pyspark/rdd.py", 
> line 772, in collect
>   File "/usr/hdp/2.5.0.0-1245/spark/python/lib/pyspark.zip/pyspark/rdd.py", 
> line 142, in _load_from_socket
>   File 
> "/usr/hdp/2.5.0.0-1245/spark/python/lib/pyspark.zip/pyspark/serializers.py", 
> line 517, in load_stream
>   File 
> "/usr/hdp/2.5.0.0-1245/spark/python/lib/pyspark.zip/pyspark/serializers.py", 
> line 511, in loads
>   File "/usr/lib/python2.7/socket.py", line 380, in read
> data = self._sock.recv(left)
> socket.timeout: timed out
> 16/11/30 13:33:14 WARN Utils: Suppressing exception in finally: Broken pipe
> java.net.SocketException: Broken pipe
>   at java.net.SocketOutputStream.socketWrite0(Native Method)
>   at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
>   at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
>   at 
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>   at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
>   at java.io.DataOutputStream.flush(DataOutputStream.java:123)
>   at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
>   at 
> org.apache.spark.api.python.PythonRDD$$anon$2$$anonfun$run$2.apply$mcV$sp(PythonRDD.scala:650)
>   at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1248)
>   at 
> org.apache.spark.api.python.PythonRDD$$anon$2.run(PythonRDD.scala:649)
>   Suppressed: java.net.SocketException: Broken pipe
>   at java.net.SocketOutputStream.socketWrite0(Native Method)
>   at 
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
>   at 
> java.net.SocketOutputStream.write(SocketOutputStream.java:153)
>   at 
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>   at 
> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
>   at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
>   at java.io.FilterOutputStream.close(FilterOutputStream.java:159)
>   ... 3 more
> 16/11/30 13:33:14 ERROR PythonRDD: Error while sending iterator
> java.net.SocketException: Connection reset
>   at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>   at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
>   at 
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>   at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>   at java.io.DataOutputStream.write(DataOutputStream.java:107)
>   at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
>   at org.apache.spark.api.python.PythonRDD$.writeUTF(PythonRDD.scala:622)
>   at 
> 

[jira] [Assigned] (SPARK-20994) Alleviate memory pressure in StreamManager

[Wenchen Fan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-20994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wenchen Fan reassigned SPARK-20994:
---

Assignee: jin xing

> Alleviate memory pressure in StreamManager
> --
>
> Key: SPARK-20994
> URL: https://issues.apache.org/jira/browse/SPARK-20994
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core
>Affects Versions: 2.1.1
>Reporter: jin xing
>Assignee: jin xing
> Fix For: 2.3.0
>
>
> In my cluster, we are suffering from OOM of shuffle-service.
> We found that a lot of executors are fetching blocks from a single 
> shuffle-service. Analyzing the memory, we found that the 
> blockIds({{shuffle_shuffleId_mapId_reduceId}}) takes about 1.5GBytes.
> In current code, chunks are fetched from shuffle service in two steps:
> Step-1. Send {{OpenBlocks}}, which contains the blocks list to to fetch;
> Step-2. Fetch the consecutive chunks from shuffle-service by {{streamId}} and 
> {{chunkIndex}}
> Thus memory cost can be improved for step-1.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Resolved] (SPARK-20994) Alleviate memory pressure in StreamManager

[Wenchen Fan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-20994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wenchen Fan resolved SPARK-20994.
-
   Resolution: Fixed
Fix Version/s: 2.3.0

Issue resolved by pull request 18231
[https://github.com/apache/spark/pull/18231]

> Alleviate memory pressure in StreamManager
> --
>
> Key: SPARK-20994
> URL: https://issues.apache.org/jira/browse/SPARK-20994
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core
>Affects Versions: 2.1.1
>Reporter: jin xing
> Fix For: 2.3.0
>
>
> In my cluster, we are suffering from OOM of shuffle-service.
> We found that a lot of executors are fetching blocks from a single 
> shuffle-service. Analyzing the memory, we found that the 
> blockIds({{shuffle_shuffleId_mapId_reduceId}}) takes about 1.5GBytes.
> In current code, chunks are fetched from shuffle service in two steps:
> Step-1. Send {{OpenBlocks}}, which contains the blocks list to to fetch;
> Step-2. Fetch the consecutive chunks from shuffle-service by {{streamId}} and 
> {{chunkIndex}}
> Thus memory cost can be improved for step-1.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-21047) Add test suites for complicated cases in ColumnarBatchSuite

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-21047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051769#comment-16051769
 ] 

jin xing commented on SPARK-21047:
--

[~kiszk]
Would you mind if I make a try for this JIRA?

> Add test suites for complicated cases in ColumnarBatchSuite
> ---
>
> Key: SPARK-21047
> URL: https://issues.apache.org/jira/browse/SPARK-21047
> Project: Spark
>  Issue Type: Sub-task
>  Components: SQL
>Affects Versions: 2.3.0
>Reporter: Kazuaki Ishizaki
>
> Current {{ColumnarBatchSuite}} has very simple test cases for array. This 
> JIRA will add test suites for complicated cases such as nested array in 
> {{ColumnVector}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-21047) Add test suites for complicated cases in ColumnarBatchSuite

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-21047:


Assignee: (was: Apache Spark)

> Add test suites for complicated cases in ColumnarBatchSuite
> ---
>
> Key: SPARK-21047
> URL: https://issues.apache.org/jira/browse/SPARK-21047
> Project: Spark
>  Issue Type: Sub-task
>  Components: SQL
>Affects Versions: 2.3.0
>Reporter: Kazuaki Ishizaki
>
> Current {{ColumnarBatchSuite}} has very simple test cases for array. This 
> JIRA will add test suites for complicated cases such as nested array in 
> {{ColumnVector}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-21047) Add test suites for complicated cases in ColumnarBatchSuite

[Apache Spark (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-21047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051768#comment-16051768
 ] 

Apache Spark commented on SPARK-21047:
--

User 'jinxing64' has created a pull request for this issue:
https://github.com/apache/spark/pull/18327

> Add test suites for complicated cases in ColumnarBatchSuite
> ---
>
> Key: SPARK-21047
> URL: https://issues.apache.org/jira/browse/SPARK-21047
> Project: Spark
>  Issue Type: Sub-task
>  Components: SQL
>Affects Versions: 2.3.0
>Reporter: Kazuaki Ishizaki
>
> Current {{ColumnarBatchSuite}} has very simple test cases for array. This 
> JIRA will add test suites for complicated cases such as nested array in 
> {{ColumnVector}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-21047) Add test suites for complicated cases in ColumnarBatchSuite

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-21047:


Assignee: Apache Spark

> Add test suites for complicated cases in ColumnarBatchSuite
> ---
>
> Key: SPARK-21047
> URL: https://issues.apache.org/jira/browse/SPARK-21047
> Project: Spark
>  Issue Type: Sub-task
>  Components: SQL
>Affects Versions: 2.3.0
>Reporter: Kazuaki Ishizaki
>Assignee: Apache Spark
>
> Current {{ColumnarBatchSuite}} has very simple test cases for array. This 
> JIRA will add test suites for complicated cases such as nested array in 
> {{ColumnVector}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-21118) OOM with 2 handred million vertex when mitrx multply

[tao (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-21118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051766#comment-16051766
 ] 

tao commented on SPARK-21118:
-

ok,thanks


> OOM with 2 handred million vertex when mitrx multply
> 
>
> Key: SPARK-21118
> URL: https://issues.apache.org/jira/browse/SPARK-21118
> Project: Spark
>  Issue Type: Bug
>  Components: MLlib
>Affects Versions: 2.1.0
> Environment: on yarn cluster,19 node.30GB per node
>Reporter: tao
>
> i have 2 matrix each one is 200milions*200milions.
> i want to multiply them ,but run out with oom .
> finally i find the oom appear at blockmatrix.simulateMultiply . there is a 
> collect action at this method. 
>  the collect will return all dataset that is too large to driver so the 
> driver will go to oom.
> class BlockMatrix @Since("1.3.0") (
> private[distributed] def simulateMultiply(
>   other: BlockMatrix,
>   partitioner: GridPartitioner): (BlockDestinations, BlockDestinations) = 
> {
> val leftMatrix = {color:red}blockInfo.keys.collect() {color}// blockInfo 
> should already be cached
> val rightMatrix = other.blocks.keys.collect()
> ..



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-12606) Scala/Java compatibility issue Re: how to extend java transformer from Scala UnaryTransformer ?

[Ilyes Hachani (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-12606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051726#comment-16051726
 ] 

Ilyes Hachani commented on SPARK-12606:
---

[~srowen] Can I rewrite it ?
What does setting the UID in the constructor means?

> Scala/Java compatibility issue Re: how to extend java transformer from Scala 
> UnaryTransformer ?
> ---
>
> Key: SPARK-12606
> URL: https://issues.apache.org/jira/browse/SPARK-12606
> Project: Spark
>  Issue Type: Bug
>  Components: ML
>Affects Versions: 1.5.2
> Environment: Java 8, Mac OS, Spark-1.5.2
>Reporter: Andrew Davidson
>  Labels: transformers
>
> Hi Andy,
> I suspect that you hit the Scala/Java compatibility issue, I can also 
> reproduce this issue, so could you file a JIRA to track this issue?
> Yanbo
> 2016-01-02 3:38 GMT+08:00 Andy Davidson :
> I am trying to write a trivial transformer I use use in my pipeline. I am 
> using java and spark 1.5.2. It was suggested that I use the Tokenize.scala 
> class as an example. This should be very easy how ever I do not understand 
> Scala, I am having trouble debugging the following exception.
> Any help would be greatly appreciated.
> Happy New Year
> Andy
> java.lang.IllegalArgumentException: requirement failed: Param null__inputCol 
> does not belong to Stemmer_2f3aa96d-7919-4eaa-ad54-f7c620b92d1c.
>   at scala.Predef$.require(Predef.scala:233)
>   at org.apache.spark.ml.param.Params$class.shouldOwn(params.scala:557)
>   at org.apache.spark.ml.param.Params$class.set(params.scala:436)
>   at org.apache.spark.ml.PipelineStage.set(Pipeline.scala:37)
>   at org.apache.spark.ml.param.Params$class.set(params.scala:422)
>   at org.apache.spark.ml.PipelineStage.set(Pipeline.scala:37)
>   at 
> org.apache.spark.ml.UnaryTransformer.setInputCol(Transformer.scala:83)
>   at com.pws.xxx.ml.StemmerTest.test(StemmerTest.java:30)
> public class StemmerTest extends AbstractSparkTest {
> @Test
> public void test() {
> Stemmer stemmer = new Stemmer()
> .setInputCol("raw”) //line 30
> .setOutputCol("filtered");
> }
> }
> /**
>  * @ see 
> spark-1.5.1/mllib/src/main/scala/org/apache/spark/ml/feature/Tokenizer.scala
>  * @ see 
> https://chimpler.wordpress.com/2014/06/11/classifiying-documents-using-naive-bayes-on-apache-spark-mllib/
>  * @ see 
> http://www.tonytruong.net/movie-rating-prediction-with-apache-spark-and-hortonworks/
>  * 
>  * @author andrewdavidson
>  *
>  */
> public class Stemmer extends UnaryTransformer Stemmer> implements Serializable{
> static Logger logger = LoggerFactory.getLogger(Stemmer.class);
> private static final long serialVersionUID = 1L;
> private static final  ArrayType inputType = 
> DataTypes.createArrayType(DataTypes.StringType, true);
> private final String uid = Stemmer.class.getSimpleName() + "_" + 
> UUID.randomUUID().toString();
> @Override
> public String uid() {
> return uid;
> }
> /*
>override protected def validateInputType(inputType: DataType): Unit = {
> require(inputType == StringType, s"Input type must be string type but got 
> $inputType.")
>   }
>  */
> @Override
> public void validateInputType(DataType inputTypeArg) {
> String msg = "inputType must be " + inputType.simpleString() + " but 
> got " + inputTypeArg.simpleString();
> assert (inputType.equals(inputTypeArg)) : msg; 
> }
> 
> @Override
> public Function1 createTransformFunc() {
> // 
> http://stackoverflow.com/questions/6545066/using-scala-from-java-passing-functions-as-parameters
> Function1 f = new 
> AbstractFunction1() {
> public List apply(List words) {
> for(String word : words) {
> logger.error("AEDWIP input word: {}", word);
> }
> return words;
> }
> };
> 
> return f;
> }
> @Override
> public DataType outputDataType() {
> return DataTypes.createArrayType(DataTypes.StringType, true);
> }
> }



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Resolved] (SPARK-12606) Scala/Java compatibility issue Re: how to extend java transformer from Scala UnaryTransformer ?

[Sean Owen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-12606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Owen resolved SPARK-12606.
---
Resolution: Invalid

This wasn't really a well-formed JIRA to begin with.
[~ihachani] I don't think that's what setting UID in the constructor means here.
This should be on the mailing list.

> Scala/Java compatibility issue Re: how to extend java transformer from Scala 
> UnaryTransformer ?
> ---
>
> Key: SPARK-12606
> URL: https://issues.apache.org/jira/browse/SPARK-12606
> Project: Spark
>  Issue Type: Bug
>  Components: ML
>Affects Versions: 1.5.2
> Environment: Java 8, Mac OS, Spark-1.5.2
>Reporter: Andrew Davidson
>  Labels: transformers
>
> Hi Andy,
> I suspect that you hit the Scala/Java compatibility issue, I can also 
> reproduce this issue, so could you file a JIRA to track this issue?
> Yanbo
> 2016-01-02 3:38 GMT+08:00 Andy Davidson :
> I am trying to write a trivial transformer I use use in my pipeline. I am 
> using java and spark 1.5.2. It was suggested that I use the Tokenize.scala 
> class as an example. This should be very easy how ever I do not understand 
> Scala, I am having trouble debugging the following exception.
> Any help would be greatly appreciated.
> Happy New Year
> Andy
> java.lang.IllegalArgumentException: requirement failed: Param null__inputCol 
> does not belong to Stemmer_2f3aa96d-7919-4eaa-ad54-f7c620b92d1c.
>   at scala.Predef$.require(Predef.scala:233)
>   at org.apache.spark.ml.param.Params$class.shouldOwn(params.scala:557)
>   at org.apache.spark.ml.param.Params$class.set(params.scala:436)
>   at org.apache.spark.ml.PipelineStage.set(Pipeline.scala:37)
>   at org.apache.spark.ml.param.Params$class.set(params.scala:422)
>   at org.apache.spark.ml.PipelineStage.set(Pipeline.scala:37)
>   at 
> org.apache.spark.ml.UnaryTransformer.setInputCol(Transformer.scala:83)
>   at com.pws.xxx.ml.StemmerTest.test(StemmerTest.java:30)
> public class StemmerTest extends AbstractSparkTest {
> @Test
> public void test() {
> Stemmer stemmer = new Stemmer()
> .setInputCol("raw”) //line 30
> .setOutputCol("filtered");
> }
> }
> /**
>  * @ see 
> spark-1.5.1/mllib/src/main/scala/org/apache/spark/ml/feature/Tokenizer.scala
>  * @ see 
> https://chimpler.wordpress.com/2014/06/11/classifiying-documents-using-naive-bayes-on-apache-spark-mllib/
>  * @ see 
> http://www.tonytruong.net/movie-rating-prediction-with-apache-spark-and-hortonworks/
>  * 
>  * @author andrewdavidson
>  *
>  */
> public class Stemmer extends UnaryTransformer Stemmer> implements Serializable{
> static Logger logger = LoggerFactory.getLogger(Stemmer.class);
> private static final long serialVersionUID = 1L;
> private static final  ArrayType inputType = 
> DataTypes.createArrayType(DataTypes.StringType, true);
> private final String uid = Stemmer.class.getSimpleName() + "_" + 
> UUID.randomUUID().toString();
> @Override
> public String uid() {
> return uid;
> }
> /*
>override protected def validateInputType(inputType: DataType): Unit = {
> require(inputType == StringType, s"Input type must be string type but got 
> $inputType.")
>   }
>  */
> @Override
> public void validateInputType(DataType inputTypeArg) {
> String msg = "inputType must be " + inputType.simpleString() + " but 
> got " + inputTypeArg.simpleString();
> assert (inputType.equals(inputTypeArg)) : msg; 
> }
> 
> @Override
> public Function1 createTransformFunc() {
> // 
> http://stackoverflow.com/questions/6545066/using-scala-from-java-passing-functions-as-parameters
> Function1 f = new 
> AbstractFunction1() {
> public List apply(List words) {
> for(String word : words) {
> logger.error("AEDWIP input word: {}", word);
> }
> return words;
> }
> };
> 
> return f;
> }
> @Override
> public DataType outputDataType() {
> return DataTypes.createArrayType(DataTypes.StringType, true);
> }
> }



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-21120) Increasing the master's metric is conducive to the spark cluster management system monitoring.

[Sean Owen (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-21120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051699#comment-16051699
 ] 

Sean Owen commented on SPARK-21120:
---

There's no detail about the motivation here. Please don't open JIRAs with no 
context or detail.

> Increasing the master's metric is conducive to the spark cluster management 
> system monitoring.
> --
>
> Key: SPARK-21120
> URL: https://issues.apache.org/jira/browse/SPARK-21120
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core
>Affects Versions: 2.3.0
>Reporter: guoxiaolongzte
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-12606) Scala/Java compatibility issue Re: how to extend java transformer from Scala UnaryTransformer ?

[Ilyes Hachani (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-12606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044508#comment-16044508
 ] 

Ilyes Hachani edited comment on SPARK-12606 at 6/16/17 9:52 AM:


Any update on this?

I am using Java and extending the class but I get problem when I use 
"setInputCol"

{code}
"java.lang.IllegalArgumentException: requirement failed: Param null__inputCol 
does not belong to TextCleaner_40f8c0be7bc7."
{code}

I tried setting the uid inside the constructor to no result.
{code:title=TextCleaner.java|borderStyle=solid}
private final String uid ;
public TextCleaner(){
uid =  Identifiable$.MODULE$.randomUID("TextCleaner");
}
@Override
public String uid() {
return uid;
}
{code}
Spark version 2.1.1


was (Author: ihachani):
Any update on this?

I am using Java and extending the class but I get problem when I use 
"setInputCol"

{code}
"java.lang.IllegalArgumentException: requirement failed: Param null__inputCol 
does not belong to TextCleaner_40f8c0be7bc7."
{code}

I tried setting the uid inside the constructor to no result.
{code:title=TextCleaner.java|borderStyle=solid}
private final String uid ;
public TextCleaner(){
uid =  Identifiable$.MODULE$.randomUID("TextCleaner");
}
@Override
public String uid() {
return uid;
}
{code}

> Scala/Java compatibility issue Re: how to extend java transformer from Scala 
> UnaryTransformer ?
> ---
>
> Key: SPARK-12606
> URL: https://issues.apache.org/jira/browse/SPARK-12606
> Project: Spark
>  Issue Type: Bug
>  Components: ML
>Affects Versions: 1.5.2
> Environment: Java 8, Mac OS, Spark-1.5.2
>Reporter: Andrew Davidson
>  Labels: transformers
>
> Hi Andy,
> I suspect that you hit the Scala/Java compatibility issue, I can also 
> reproduce this issue, so could you file a JIRA to track this issue?
> Yanbo
> 2016-01-02 3:38 GMT+08:00 Andy Davidson :
> I am trying to write a trivial transformer I use use in my pipeline. I am 
> using java and spark 1.5.2. It was suggested that I use the Tokenize.scala 
> class as an example. This should be very easy how ever I do not understand 
> Scala, I am having trouble debugging the following exception.
> Any help would be greatly appreciated.
> Happy New Year
> Andy
> java.lang.IllegalArgumentException: requirement failed: Param null__inputCol 
> does not belong to Stemmer_2f3aa96d-7919-4eaa-ad54-f7c620b92d1c.
>   at scala.Predef$.require(Predef.scala:233)
>   at org.apache.spark.ml.param.Params$class.shouldOwn(params.scala:557)
>   at org.apache.spark.ml.param.Params$class.set(params.scala:436)
>   at org.apache.spark.ml.PipelineStage.set(Pipeline.scala:37)
>   at org.apache.spark.ml.param.Params$class.set(params.scala:422)
>   at org.apache.spark.ml.PipelineStage.set(Pipeline.scala:37)
>   at 
> org.apache.spark.ml.UnaryTransformer.setInputCol(Transformer.scala:83)
>   at com.pws.xxx.ml.StemmerTest.test(StemmerTest.java:30)
> public class StemmerTest extends AbstractSparkTest {
> @Test
> public void test() {
> Stemmer stemmer = new Stemmer()
> .setInputCol("raw”) //line 30
> .setOutputCol("filtered");
> }
> }
> /**
>  * @ see 
> spark-1.5.1/mllib/src/main/scala/org/apache/spark/ml/feature/Tokenizer.scala
>  * @ see 
> https://chimpler.wordpress.com/2014/06/11/classifiying-documents-using-naive-bayes-on-apache-spark-mllib/
>  * @ see 
> http://www.tonytruong.net/movie-rating-prediction-with-apache-spark-and-hortonworks/
>  * 
>  * @author andrewdavidson
>  *
>  */
> public class Stemmer extends UnaryTransformer Stemmer> implements Serializable{
> static Logger logger = LoggerFactory.getLogger(Stemmer.class);
> private static final long serialVersionUID = 1L;
> private static final  ArrayType inputType = 
> DataTypes.createArrayType(DataTypes.StringType, true);
> private final String uid = Stemmer.class.getSimpleName() + "_" + 
> UUID.randomUUID().toString();
> @Override
> public String uid() {
> return uid;
> }
> /*
>override protected def validateInputType(inputType: DataType): Unit = {
> require(inputType == StringType, s"Input type must be string type but got 
> $inputType.")
>   }
>  */
> @Override
> public void validateInputType(DataType inputTypeArg) {
> String msg = "inputType must be " + inputType.simpleString() + " but 
> got " + inputTypeArg.simpleString();
> assert (inputType.equals(inputTypeArg)) : msg; 
> }
> 
> @Override
> public Function1 createTransformFunc() {
> // 

[jira] [Resolved] (SPARK-21118) OOM with 2 handred million vertex when mitrx multply

[Sean Owen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Owen resolved SPARK-21118.
---
Resolution: Invalid

[~icesxrun] do not reopen this. This doesn't describe a problem with Spark, but 
a question about usage.

> OOM with 2 handred million vertex when mitrx multply
> 
>
> Key: SPARK-21118
> URL: https://issues.apache.org/jira/browse/SPARK-21118
> Project: Spark
>  Issue Type: Bug
>  Components: MLlib
>Affects Versions: 2.1.0
> Environment: on yarn cluster,19 node.30GB per node
>Reporter: tao
>
> i have 2 matrix each one is 200milions*200milions.
> i want to multiply them ,but run out with oom .
> finally i find the oom appear at blockmatrix.simulateMultiply . there is a 
> collect action at this method. 
>  the collect will return all dataset that is too large to driver so the 
> driver will go to oom.
> class BlockMatrix @Since("1.3.0") (
> private[distributed] def simulateMultiply(
>   other: BlockMatrix,
>   partitioner: GridPartitioner): (BlockDestinations, BlockDestinations) = 
> {
> val leftMatrix = {color:red}blockInfo.keys.collect() {color}// blockInfo 
> should already be cached
> val rightMatrix = other.blocks.keys.collect()
> ..



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Closed] (SPARK-21118) OOM with 2 handred million vertex when mitrx multply

[Sean Owen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Owen closed SPARK-21118.
-

> OOM with 2 handred million vertex when mitrx multply
> 
>
> Key: SPARK-21118
> URL: https://issues.apache.org/jira/browse/SPARK-21118
> Project: Spark
>  Issue Type: Bug
>  Components: MLlib
>Affects Versions: 2.1.0
> Environment: on yarn cluster,19 node.30GB per node
>Reporter: tao
>
> i have 2 matrix each one is 200milions*200milions.
> i want to multiply them ,but run out with oom .
> finally i find the oom appear at blockmatrix.simulateMultiply . there is a 
> collect action at this method. 
>  the collect will return all dataset that is too large to driver so the 
> driver will go to oom.
> class BlockMatrix @Since("1.3.0") (
> private[distributed] def simulateMultiply(
>   other: BlockMatrix,
>   partitioner: GridPartitioner): (BlockDestinations, BlockDestinations) = 
> {
> val leftMatrix = {color:red}blockInfo.keys.collect() {color}// blockInfo 
> should already be cached
> val rightMatrix = other.blocks.keys.collect()
> ..



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Resolved] (SPARK-21077) Cannot access public files over S3 protocol

[Sean Owen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Owen resolved SPARK-21077.
---
Resolution: Not A Problem

> Cannot access public files over S3 protocol
> ---
>
> Key: SPARK-21077
> URL: https://issues.apache.org/jira/browse/SPARK-21077
> Project: Spark
>  Issue Type: Bug
>  Components: EC2
>Affects Versions: 2.1.0
> Environment: Spark 2.1.0 default installation. No existing hadoop, 
> using the one distributed with Spark.
> Added in $SPARK_HOME/jars:  
> hadoop-aws-2.7.3.jar and aws-java-sdk-1.7.4.jar
> Added endpoint configuration in $SPARK_HOME/conf/core-site.xml (I want to 
> access datasets hosted by organisation with CEPH; follows S3 protocols).
> Ubuntu 14.04 x64.
>Reporter: Ciprian Tomoiaga
>
> I am trying to access a dataset with public (anonymous) credentials via the 
> S3 (or S3a, s3n) protocol. 
> It fails with the error that no provider in chain can supply the credentials.
> I asked our sysadmin to add some dummy credentials, and if I set them up (via 
> link or config) then I have access.
> I tried setting the config :
> {code:xml}
> 
>   fs.s3a.credentials.provider
>   org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider
> 
> {code}
> but it still doesn't work.
> I suggested that it is a java-aws issue 
> [here|https://github.com/aws/aws-sdk-java/issues/1122#issuecomment-307814540],
>  but they said it is not.
> Any hints on how to use public S3 files from Spark ?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-21116) Support map_contains function

[darion yaphet (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

darion yaphet updated SPARK-21116:
--
Summary: Support map_contains function  (was: Support MapKeyContains 
function)

> Support map_contains function
> -
>
> Key: SPARK-21116
> URL: https://issues.apache.org/jira/browse/SPARK-21116
> Project: Spark
>  Issue Type: New Feature
>  Components: SQL
>Affects Versions: 2.0.2, 2.1.1
>Reporter: darion yaphet
>
> map_contains(map , key)
> Check whether the map contains the key . Returns true if the map contains the 
> key. It's similar with *array_contains*
> for example :  map_contains(map(1, 'a', 2, 'b') , 1) will return true . 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-21116) Support MapKeyContains function

[darion yaphet (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

darion yaphet updated SPARK-21116:
--
Description: 
map_contains(map , key)

Check whether the map contains the key . Returns true if the map contains the 
key. It's similar with *array_contains*

for example :  map_contains(map(1, 'a', 2, 'b') , 1) will return true . 

  was:
map_contains(map , key)

Check whether the map contains the key . Returns true if the map contains the 
key. It's similar with *array_contains*


> Support MapKeyContains function
> ---
>
> Key: SPARK-21116
> URL: https://issues.apache.org/jira/browse/SPARK-21116
> Project: Spark
>  Issue Type: New Feature
>  Components: SQL
>Affects Versions: 2.0.2, 2.1.1
>Reporter: darion yaphet
>
> map_contains(map , key)
> Check whether the map contains the key . Returns true if the map contains the 
> key. It's similar with *array_contains*
> for example :  map_contains(map(1, 'a', 2, 'b') , 1) will return true . 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-21077) Cannot access public files over S3 protocol

[Steve Loughran (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-21077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051680#comment-16051680
 ] 

Steve Loughran commented on SPARK-21077:


like people say, this is inevitably a config problem. Hadoop 2.7.x has the 
credential provider you need


you should be able to read {{s3a://landsat-pds/scene_list.gz}} as a csv file 
with anon credentials. 

> Cannot access public files over S3 protocol
> ---
>
> Key: SPARK-21077
> URL: https://issues.apache.org/jira/browse/SPARK-21077
> Project: Spark
>  Issue Type: Bug
>  Components: EC2
>Affects Versions: 2.1.0
> Environment: Spark 2.1.0 default installation. No existing hadoop, 
> using the one distributed with Spark.
> Added in $SPARK_HOME/jars:  
> hadoop-aws-2.7.3.jar and aws-java-sdk-1.7.4.jar
> Added endpoint configuration in $SPARK_HOME/conf/core-site.xml (I want to 
> access datasets hosted by organisation with CEPH; follows S3 protocols).
> Ubuntu 14.04 x64.
>Reporter: Ciprian Tomoiaga
>
> I am trying to access a dataset with public (anonymous) credentials via the 
> S3 (or S3a, s3n) protocol. 
> It fails with the error that no provider in chain can supply the credentials.
> I asked our sysadmin to add some dummy credentials, and if I set them up (via 
> link or config) then I have access.
> I tried setting the config :
> {code:xml}
> 
>   fs.s3a.credentials.provider
>   org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider
> 
> {code}
> but it still doesn't work.
> I suggested that it is a java-aws issue 
> [here|https://github.com/aws/aws-sdk-java/issues/1122#issuecomment-307814540],
>  but they said it is not.
> Any hints on how to use public S3 files from Spark ?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-21120) Increasing the master's metric is conducive to the spark cluster management system monitoring.

[Apache Spark (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-21120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051660#comment-16051660
 ] 

Apache Spark commented on SPARK-21120:
--

User 'guoxiaolongzte' has created a pull request for this issue:
https://github.com/apache/spark/pull/18326

> Increasing the master's metric is conducive to the spark cluster management 
> system monitoring.
> --
>
> Key: SPARK-21120
> URL: https://issues.apache.org/jira/browse/SPARK-21120
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core
>Affects Versions: 2.3.0
>Reporter: guoxiaolongzte
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-21120) Increasing the master's metric is conducive to the spark cluster management system monitoring.

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-21120:


Assignee: Apache Spark

> Increasing the master's metric is conducive to the spark cluster management 
> system monitoring.
> --
>
> Key: SPARK-21120
> URL: https://issues.apache.org/jira/browse/SPARK-21120
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core
>Affects Versions: 2.3.0
>Reporter: guoxiaolongzte
>Assignee: Apache Spark
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (SPARK-21120) Increasing the master's metric is conducive to the spark cluster management system monitoring.

[Apache Spark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-21120:


Assignee: (was: Apache Spark)

> Increasing the master's metric is conducive to the spark cluster management 
> system monitoring.
> --
>
> Key: SPARK-21120
> URL: https://issues.apache.org/jira/browse/SPARK-21120
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core
>Affects Versions: 2.3.0
>Reporter: guoxiaolongzte
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-21116) Support MapKeyContains function

[darion yaphet (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

darion yaphet updated SPARK-21116:
--
Description: 
map_contains(map , key)

Check whether the map contains the key . Returns true if the map contains the 
key. It's similar with *array_contains*

  was:Check the map contains the key . Returns true if the map contains the key.


> Support MapKeyContains function
> ---
>
> Key: SPARK-21116
> URL: https://issues.apache.org/jira/browse/SPARK-21116
> Project: Spark
>  Issue Type: New Feature
>  Components: SQL
>Affects Versions: 2.0.2, 2.1.1
>Reporter: darion yaphet
>
> map_contains(map , key)
> Check whether the map contains the key . Returns true if the map contains the 
> key. It's similar with *array_contains*



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-21118) OOM with 2 handred million vertex when mitrx multply

[JIRA]

[ 
https://issues.apache.org/jira/browse/SPARK-21118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051641#comment-16051641
 ] 

Lorenz Bühmann commented on SPARK-21118:


The first point would be to use a subject title without typos. I mean "handred" 
and "mitrx multply"? Come on - how can others search for similar problems?!

Secondly, you're using `collect()` for both matrices. That's more or less 
breaking the idea of Spark, since you're collecting everything to the driver in 
memory. Of course, this will mean for large data to an OOM. You should read 
more about the principles of Spark I guess.

> OOM with 2 handred million vertex when mitrx multply
> 
>
> Key: SPARK-21118
> URL: https://issues.apache.org/jira/browse/SPARK-21118
> Project: Spark
>  Issue Type: Bug
>  Components: MLlib
>Affects Versions: 2.1.0
> Environment: on yarn cluster,19 node.30GB per node
>Reporter: tao
>
> i have 2 matrix each one is 200milions*200milions.
> i want to multiply them ,but run out with oom .
> finally i find the oom appear at blockmatrix.simulateMultiply . there is a 
> collect action at this method. 
>  the collect will return all dataset that is too large to driver so the 
> driver will go to oom.
> class BlockMatrix @Since("1.3.0") (
> private[distributed] def simulateMultiply(
>   other: BlockMatrix,
>   partitioner: GridPartitioner): (BlockDestinations, BlockDestinations) = 
> {
> val leftMatrix = {color:red}blockInfo.keys.collect() {color}// blockInfo 
> should already be cached
> val rightMatrix = other.blocks.keys.collect()
> ..



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-21120) Increasing the master's metric is conducive to the spark cluster management system monitoring.

[guoxiaolongzte (JIRA)]

guoxiaolongzte created SPARK-21120:
--

 Summary: Increasing the master's metric is conducive to the spark 
cluster management system monitoring.
 Key: SPARK-21120
 URL: https://issues.apache.org/jira/browse/SPARK-21120
 Project: Spark
  Issue Type: Improvement
  Components: Spark Core
Affects Versions: 2.3.0
Reporter: guoxiaolongzte
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[JiYeon OH (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051607#comment-16051607
 ] 

JiYeon OH edited comment on SPARK-12009 at 6/16/17 8:50 AM:


I'm having the same problem with Spark 2.1.0
I have some jobs with exact same code and had a few jobs failed.
In the jobs that finished successfully, there was this message after the job 
finished:

17/06/15 00:26:02 INFO YarnAllocator: Driver requested a total number of 0 
executor(s).

But in the jobs that failed, there was this message instead:

17/06/16 14:31:14 ERROR LiveListenerBus: SparkListenerBus has already stopped! 
Dropping event SparkListenerExecutorMetricsUpdate(10,WrappedArray())

I'm guessing the YarnAllocator must have requested some executors after spark 
job was finished, but can't' find out why.
and why is YarnAllocator requesting executors after job finished Does 
anyone know why??


was (Author: ogcheeze):
I'm having the same problem with Spark 2.1.0
I have some jobs with exact same code and had a few jobs failed.
In the jobs that finished successfully, there was this message after the job 
finished

17/06/15 00:26:02 INFO YarnAllocator: Driver requested a total number of 0 
executor(s).

But in the jobs that filaed, there was this message instead

17/06/16 14:31:14 ERROR LiveListenerBus: SparkListenerBus has already stopped! 
Dropping event SparkListenerExecutorMetricsUpdate(10,WrappedArray())

I'm guessing the YarnAllocator must have requested some executors after spark 
job was finished, but can't' find out why.
and why is YarnAllocator requesting executors after job finished Does 
anyone know why??

> Avoid re-allocate yarn container while driver want to stop all Executors
> 
>
> Key: SPARK-12009
> URL: https://issues.apache.org/jira/browse/SPARK-12009
> Project: Spark
>  Issue Type: Bug
>  Components: YARN
>Affects Versions: 1.5.2
>Reporter: SuYan
>Assignee: SuYan
>Priority: Minor
> Fix For: 2.0.0
>
>
> Log based 1.4.0
> 2015-11-26,03:05:16,176 WARN 
> org.spark-project.jetty.util.thread.QueuedThreadPool: 8 threads could not be 
> stopped
> 2015-11-26,03:05:16,177 INFO org.apache.spark.ui.SparkUI: Stopped Spark web 
> UI at http://
> 2015-11-26,03:05:16,401 INFO org.apache.spark.scheduler.DAGScheduler: 
> Stopping DAGScheduler
> 2015-11-26,03:05:16,450 INFO 
> org.apache.spark.scheduler.cluster.YarnClusterSchedulerBackend: Shutting down 
> all executors
> 2015-11-26,03:05:16,525 INFO 
> org.apache.spark.scheduler.cluster.YarnClusterSchedulerBackend: Asking each 
> executor to shut down
> 2015-11-26,03:05:16,791 INFO 
> org.apache.spark.deploy.yarn.ApplicationMaster$AMEndpoint: Driver terminated 
> or disconnected! Shutting down. XX.XX.XX.XX:38734
> 2015-11-26,03:05:16,847 ERROR org.apache.spark.scheduler.LiveListenerBus: 
> SparkListenerBus has already stopped! Dropping event 
> SparkListenerExecutorMetricsUpdate(164,WrappedArray())
> 2015-11-26,03:05:27,242 INFO org.apache.spark.deploy.yarn.YarnAllocator: Will 
> request 13 executor containers, each with 1 cores and 4608 MB memory 
> including 1024 MB overhead



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-21118) OOM with 2 handred million vertex when mitrx multply

[tao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

tao updated SPARK-21118:

Description: 
i have 2 matrix each one is 200milions*200milions.
i want to multiply them ,but run out with oom .
finally i find the oom appear at blockmatrix.simulateMultiply . there is a 
collect action at this method. 
 the collect will return all dataset that is too large to driver so the driver 
will go to oom.

class BlockMatrix @Since("1.3.0") (
private[distributed] def simulateMultiply(
  other: BlockMatrix,
  partitioner: GridPartitioner): (BlockDestinations, BlockDestinations) = {
val leftMatrix = {color:red}blockInfo.keys.collect() {color}// blockInfo 
should already be cached
val rightMatrix = other.blocks.keys.collect()
..



  was:
i have 2 matrix each one is 200milions*200milions.
i want to multiply them ,but run out with oom .
finally i find the oom appear at blockmatrix.simulateMultiply . there is a 
collect action at this method. 
 the collect will return all dataset to driver so the driver will go to oom.

class BlockMatrix @Since("1.3.0") (
private[distributed] def simulateMultiply(
  other: BlockMatrix,
  partitioner: GridPartitioner): (BlockDestinations, BlockDestinations) = {
val leftMatrix = {color:red}blockInfo.keys.collect() {color}// blockInfo 
should already be cached
val rightMatrix = other.blocks.keys.collect()
..




> OOM with 2 handred million vertex when mitrx multply
> 
>
> Key: SPARK-21118
> URL: https://issues.apache.org/jira/browse/SPARK-21118
> Project: Spark
>  Issue Type: Bug
>  Components: MLlib
>Affects Versions: 2.1.0
> Environment: on yarn cluster,19 node.30GB per node
>Reporter: tao
>
> i have 2 matrix each one is 200milions*200milions.
> i want to multiply them ,but run out with oom .
> finally i find the oom appear at blockmatrix.simulateMultiply . there is a 
> collect action at this method. 
>  the collect will return all dataset that is too large to driver so the 
> driver will go to oom.
> class BlockMatrix @Since("1.3.0") (
> private[distributed] def simulateMultiply(
>   other: BlockMatrix,
>   partitioner: GridPartitioner): (BlockDestinations, BlockDestinations) = 
> {
> val leftMatrix = {color:red}blockInfo.keys.collect() {color}// blockInfo 
> should already be cached
> val rightMatrix = other.blocks.keys.collect()
> ..



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[JiYeon OH (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051607#comment-16051607
 ] 

JiYeon OH commented on SPARK-12009:
---

I'm having the same problem with Spark 2.1.0
I have some jobs with exact same code and had a few jobs failed.
In the jobs that finished successfully, there was this message after the job 
finished

17/06/15 00:26:02 INFO YarnAllocator: Driver requested a total number of 0 
executor(s).

But in the jobs that filaed, there was this message instead

17/06/16 14:31:14 ERROR LiveListenerBus: SparkListenerBus has already stopped! 
Dropping event SparkListenerExecutorMetricsUpdate(10,WrappedArray())

I'm guessing the YarnAllocator must have requested some executors after spark 
job was finished, but can't' find out why.
and why is YarnAllocator requesting executors after job finished Does 
anyone know why??

> Avoid re-allocate yarn container while driver want to stop all Executors
> 
>
> Key: SPARK-12009
> URL: https://issues.apache.org/jira/browse/SPARK-12009
> Project: Spark
>  Issue Type: Bug
>  Components: YARN
>Affects Versions: 1.5.2
>Reporter: SuYan
>Assignee: SuYan
>Priority: Minor
> Fix For: 2.0.0
>
>
> Log based 1.4.0
> 2015-11-26,03:05:16,176 WARN 
> org.spark-project.jetty.util.thread.QueuedThreadPool: 8 threads could not be 
> stopped
> 2015-11-26,03:05:16,177 INFO org.apache.spark.ui.SparkUI: Stopped Spark web 
> UI at http://
> 2015-11-26,03:05:16,401 INFO org.apache.spark.scheduler.DAGScheduler: 
> Stopping DAGScheduler
> 2015-11-26,03:05:16,450 INFO 
> org.apache.spark.scheduler.cluster.YarnClusterSchedulerBackend: Shutting down 
> all executors
> 2015-11-26,03:05:16,525 INFO 
> org.apache.spark.scheduler.cluster.YarnClusterSchedulerBackend: Asking each 
> executor to shut down
> 2015-11-26,03:05:16,791 INFO 
> org.apache.spark.deploy.yarn.ApplicationMaster$AMEndpoint: Driver terminated 
> or disconnected! Shutting down. XX.XX.XX.XX:38734
> 2015-11-26,03:05:16,847 ERROR org.apache.spark.scheduler.LiveListenerBus: 
> SparkListenerBus has already stopped! Dropping event 
> SparkListenerExecutorMetricsUpdate(164,WrappedArray())
> 2015-11-26,03:05:27,242 INFO org.apache.spark.deploy.yarn.YarnAllocator: Will 
> request 13 executor containers, each with 1 cores and 4608 MB memory 
> including 1024 MB overhead



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-21118) OOM with 2 handred million vertex when mitrx multply

[tao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

tao updated SPARK-21118:

Description: 
i have 2 matrix each one is 200milions*200milions.
i want to multiply them ,but run out with oom .
finally i find the oom appear at blockmatrix.simulateMultiply . there is a 
collect action at this method. 
 the collect will return all dataset to driver so the driver will go to oom.

class BlockMatrix @Since("1.3.0") (
private[distributed] def simulateMultiply(
  other: BlockMatrix,
  partitioner: GridPartitioner): (BlockDestinations, BlockDestinations) = {
val leftMatrix = {color:red}blockInfo.keys.collect() {color}// blockInfo 
should already be cached
val rightMatrix = other.blocks.keys.collect()
..



  was:
i have 2 matrix each one is 200milions*200milions.
i want to multiply them ,but run out with oom .
finally i find the oom appear at blockmatrix.simulateMultiply . there is a 
collect action at this method. when application run there the driver will go to 
oom.

class BlockMatrix @Since("1.3.0") (
private[distributed] def simulateMultiply(
  other: BlockMatrix,
  partitioner: GridPartitioner): (BlockDestinations, BlockDestinations) = {
val leftMatrix = {color:red}blockInfo.keys.collect() {color}// blockInfo 
should already be cached
val rightMatrix = other.blocks.keys.collect()
..




> OOM with 2 handred million vertex when mitrx multply
> 
>
> Key: SPARK-21118
> URL: https://issues.apache.org/jira/browse/SPARK-21118
> Project: Spark
>  Issue Type: Bug
>  Components: MLlib
>Affects Versions: 2.1.0
> Environment: on yarn cluster,19 node.30GB per node
>Reporter: tao
>
> i have 2 matrix each one is 200milions*200milions.
> i want to multiply them ,but run out with oom .
> finally i find the oom appear at blockmatrix.simulateMultiply . there is a 
> collect action at this method. 
>  the collect will return all dataset to driver so the driver will go to oom.
> class BlockMatrix @Since("1.3.0") (
> private[distributed] def simulateMultiply(
>   other: BlockMatrix,
>   partitioner: GridPartitioner): (BlockDestinations, BlockDestinations) = 
> {
> val leftMatrix = {color:red}blockInfo.keys.collect() {color}// blockInfo 
> should already be cached
> val rightMatrix = other.blocks.keys.collect()
> ..



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-21118) OOM with 2 handred million vertex when mitrx multply

[tao (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-21118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051604#comment-16051604
 ] 

tao edited comment on SPARK-21118 at 6/16/17 8:47 AM:
--

i add some  information to desciption,please have a look 


was (Author: icesxrun):
i add some mot information to desciption,please have a look 

> OOM with 2 handred million vertex when mitrx multply
> 
>
> Key: SPARK-21118
> URL: https://issues.apache.org/jira/browse/SPARK-21118
> Project: Spark
>  Issue Type: Bug
>  Components: MLlib
>Affects Versions: 2.1.0
> Environment: on yarn cluster,19 node.30GB per node
>Reporter: tao
>
> i have 2 matrix each one is 200milions*200milions.
> i want to multiply them ,but run out with oom .
> finally i find the oom appear at blockmatrix.simulateMultiply . there is a 
> collect action at this method. when application run there the driver will go 
> to oom.
> class BlockMatrix @Since("1.3.0") (
> private[distributed] def simulateMultiply(
>   other: BlockMatrix,
>   partitioner: GridPartitioner): (BlockDestinations, BlockDestinations) = 
> {
> val leftMatrix = {color:red}blockInfo.keys.collect() {color}// blockInfo 
> should already be cached
> val rightMatrix = other.blocks.keys.collect()
> ..



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Reopened] (SPARK-21118) OOM with 2 handred million vertex when mitrx multply

[tao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

tao reopened SPARK-21118:
-

i add some mot information to desciption,please have a look 

> OOM with 2 handred million vertex when mitrx multply
> 
>
> Key: SPARK-21118
> URL: https://issues.apache.org/jira/browse/SPARK-21118
> Project: Spark
>  Issue Type: Bug
>  Components: MLlib
>Affects Versions: 2.1.0
> Environment: on yarn cluster,19 node.30GB per node
>Reporter: tao
>
> i have 2 matrix each one is 200milions*200milions.
> i want to multiply them ,but run out with oom .
> finally i find the oom appear at blockmatrix.simulateMultiply . there is a 
> collect action at this method. when application run there the driver will go 
> to oom.
> class BlockMatrix @Since("1.3.0") (
> private[distributed] def simulateMultiply(
>   other: BlockMatrix,
>   partitioner: GridPartitioner): (BlockDestinations, BlockDestinations) = 
> {
> val leftMatrix = {color:red}blockInfo.keys.collect() {color}// blockInfo 
> should already be cached
> val rightMatrix = other.blocks.keys.collect()
> ..



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-21118) OOM with 2 handred million vertex when mitrx multply

[tao (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-21118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051602#comment-16051602
 ] 

tao edited comment on SPARK-21118 at 6/16/17 8:43 AM:
--

is there other way to do big matirx multiply


was (Author: icesxrun):
is there other way to do matirx multiply

> OOM with 2 handred million vertex when mitrx multply
> 
>
> Key: SPARK-21118
> URL: https://issues.apache.org/jira/browse/SPARK-21118
> Project: Spark
>  Issue Type: Bug
>  Components: MLlib
>Affects Versions: 2.1.0
> Environment: on yarn cluster,19 node.30GB per node
>Reporter: tao
>
> i have 2 matrix each one is 200milions*200milions.
> i want to multiply them ,but run out with oom .
> finally i find the oom appear at blockmatrix.simulateMultiply . there is a 
> collect action at this method. when application run there the driver will go 
> to oom.
> class BlockMatrix @Since("1.3.0") (
> private[distributed] def simulateMultiply(
>   other: BlockMatrix,
>   partitioner: GridPartitioner): (BlockDestinations, BlockDestinations) = 
> {
> val leftMatrix = {color:red}blockInfo.keys.collect() {color}// blockInfo 
> should already be cached
> val rightMatrix = other.blocks.keys.collect()
> ..



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-21118) OOM with 2 handred million vertex when mitrx multply

[tao (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-21118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051602#comment-16051602
 ] 

tao commented on SPARK-21118:
-

is there other way to do matirx multiply

> OOM with 2 handred million vertex when mitrx multply
> 
>
> Key: SPARK-21118
> URL: https://issues.apache.org/jira/browse/SPARK-21118
> Project: Spark
>  Issue Type: Bug
>  Components: MLlib
>Affects Versions: 2.1.0
> Environment: on yarn cluster,19 node.30GB per node
>Reporter: tao
>
> i have 2 matrix each one is 200milions*200milions.
> i want to multiply them ,but run out with oom .
> finally i find the oom appear at blockmatrix.simulateMultiply . there is a 
> collect action at this method. when application run there the driver will go 
> to oom.
> class BlockMatrix @Since("1.3.0") (
> private[distributed] def simulateMultiply(
>   other: BlockMatrix,
>   partitioner: GridPartitioner): (BlockDestinations, BlockDestinations) = 
> {
> val leftMatrix = {color:red}blockInfo.keys.collect() {color}// blockInfo 
> should already be cached
> val rightMatrix = other.blocks.keys.collect()
> ..



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-21118) OOM with 2 handred million vertex when mitrx multply

[tao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

tao updated SPARK-21118:

Description: 
i have 2 matrix each one is 200milions*200milions.
i want to multiply them ,but run out with oom .
finally i find the oom appear at blockmatrix.simulateMultiply . there is a 
collect action at this method. when application run there the driver will be 
oom.

class BlockMatrix @Since("1.3.0") (
private[distributed] def simulateMultiply(
  other: BlockMatrix,
  partitioner: GridPartitioner): (BlockDestinations, BlockDestinations) = {
val leftMatrix = {color:red}blockInfo.keys.collect() {color}// blockInfo 
should already be cached
val rightMatrix = other.blocks.keys.collect()
..



  was:
i have 2 matrix each one is 200milions*200milions.
i want to multiply them ,but run out with oom .
finally i find the oom appear at blockmatrix.simulateMultiply . there is a 
collect action at this method. when application run there the driver will oom.

class BlockMatrix @Since("1.3.0") (
private[distributed] def simulateMultiply(
  other: BlockMatrix,
  partitioner: GridPartitioner): (BlockDestinations, BlockDestinations) = {
val leftMatrix = {color:red}blockInfo.keys.collect() {color}// blockInfo 
should already be cached
val rightMatrix = other.blocks.keys.collect()
..




> OOM with 2 handred million vertex when mitrx multply
> 
>
> Key: SPARK-21118
> URL: https://issues.apache.org/jira/browse/SPARK-21118
> Project: Spark
>  Issue Type: Bug
>  Components: MLlib
>Affects Versions: 2.1.0
> Environment: on yarn cluster,19 node.30GB per node
>Reporter: tao
>
> i have 2 matrix each one is 200milions*200milions.
> i want to multiply them ,but run out with oom .
> finally i find the oom appear at blockmatrix.simulateMultiply . there is a 
> collect action at this method. when application run there the driver will be 
> oom.
> class BlockMatrix @Since("1.3.0") (
> private[distributed] def simulateMultiply(
>   other: BlockMatrix,
>   partitioner: GridPartitioner): (BlockDestinations, BlockDestinations) = 
> {
> val leftMatrix = {color:red}blockInfo.keys.collect() {color}// blockInfo 
> should already be cached
> val rightMatrix = other.blocks.keys.collect()
> ..



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org