[jira] [Commented] (SPARK-18879) Spark SQL support for Hive hooks regressed
[ https://issues.apache.org/jira/browse/SPARK-18879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15756708#comment-15756708 ] Sean Owen commented on SPARK-18879: --- I don't know that this function of Hive itself is something that Spark promises to support Spark has its own API and that is what is generally supported and guaranteed. > Spark SQL support for Hive hooks regressed > -- > > Key: SPARK-18879 > URL: https://issues.apache.org/jira/browse/SPARK-18879 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.0, 2.0.2 >Reporter: Atul Payapilly > > As per the stack trace from this post: > http://ihorbobak.com/index.php/2015/05/08/113/ > run on Spark 1.3.1 > hive.exec.pre.hooks Class not found:org.apache.hadoop.hive.ql.hooks.ATSHook > FAILED: Hive Internal Error: > java.lang.ClassNotFoundException(org.apache.hadoop.hive.ql.hooks.ATSHook) > java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.hooks.ATSHook > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:270) > at org.apache.hadoop.hive.ql.hooks.HookUtils.getHooks(HookUtils.java:59) > at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1172) > at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1156) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1206) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1088) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901) > at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:318) > at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:290) > at > org.apache.spark.sql.hive.execution.HiveNativeCommand.run(HiveNativeCommand.scala:33) > at > org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:54) > at > org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:54) > at > org.apache.spark.sql.execution.ExecutedCommand.execute(commands.scala:64) > at > org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:1099) > at > org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:1099) > at org.apache.spark.sql.DataFrame.(DataFrame.scala:147) > at org.apache.spark.sql.DataFrame.(DataFrame.scala:130) > at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:51) > at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:101) > at > org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.run(Shim13.scala:164) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:231) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:218) > at > org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:233) > at > org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:344) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:55) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > It looks like Spark used to rely on the Hive Driver for execution and > supported hive hooks. The current code path does not rely on the Hive Driver > and support for Hive hooks regressed. This is problematic, for example, there > is no way to tell which partitions were updated as part of a query. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e
[jira] [Commented] (SPARK-18879) Spark SQL support for Hive hooks regressed
[ https://issues.apache.org/jira/browse/SPARK-18879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15756759#comment-15756759 ] Atul Payapilly commented on SPARK-18879: That's a fair point, the feature was supported incidentally as a side effect of using the HiveDriver initially, so its technically not a regression of a supported feature. To add some context, the reason I'm looking into this is that some of the functionality of the Hive hooks are quite useful. Maybe I should close this and open a feature request? The functionality I think would be most important is being able to tell at the end of a query which partitions were updated, specifically in a dynamic partition insert, since this can't be derived by analyzing the query alone. What are your thoughts on this? > Spark SQL support for Hive hooks regressed > -- > > Key: SPARK-18879 > URL: https://issues.apache.org/jira/browse/SPARK-18879 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.0, 2.0.2 >Reporter: Atul Payapilly > > As per the stack trace from this post: > http://ihorbobak.com/index.php/2015/05/08/113/ > run on Spark 1.3.1 > hive.exec.pre.hooks Class not found:org.apache.hadoop.hive.ql.hooks.ATSHook > FAILED: Hive Internal Error: > java.lang.ClassNotFoundException(org.apache.hadoop.hive.ql.hooks.ATSHook) > java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.hooks.ATSHook > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:270) > at org.apache.hadoop.hive.ql.hooks.HookUtils.getHooks(HookUtils.java:59) > at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1172) > at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1156) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1206) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1088) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901) > at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:318) > at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:290) > at > org.apache.spark.sql.hive.execution.HiveNativeCommand.run(HiveNativeCommand.scala:33) > at > org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:54) > at > org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:54) > at > org.apache.spark.sql.execution.ExecutedCommand.execute(commands.scala:64) > at > org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:1099) > at > org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:1099) > at org.apache.spark.sql.DataFrame.(DataFrame.scala:147) > at org.apache.spark.sql.DataFrame.(DataFrame.scala:130) > at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:51) > at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:101) > at > org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.run(Shim13.scala:164) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:231) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:218) > at > org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:233) > at > org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:344) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:55) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > It looks like Spark used to rely on the Hive Driver for execut
[jira] [Commented] (SPARK-18891) Support for specific collection types
[
https://issues.apache.org/jira/browse/SPARK-18891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15756842#comment-15756842
]
Michal Šenkýř commented on SPARK-18891:
---
[PR #16240|https://github.com/apache/spark/pull/16240] should be relevant. It
adds deserialization support for arbitrary sequence types (serialization was
already supported). The only thing not working yet is the {{Seq.toDS}} method
due to implicits resolution ({{SparkSession.createDataset}} works fine).
> Support for specific collection types
> -
>
> Key: SPARK-18891
> URL: https://issues.apache.org/jira/browse/SPARK-18891
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 1.6.3, 2.1.0
>Reporter: Michael Armbrust
>Priority: Critical
>
> Encoders treat all collections the same (i.e. {{Seq}} vs {{List}}) which
> force users to only define classes with the most generic type.
> An [example
> error|https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/1023043053387187/2398463439880241/2840265927289860/latest.html]:
> {code}
> case class SpecificCollection(aList: List[Int])
> Seq(SpecificCollection(1 :: Nil)).toDS().collect()
> {code}
> {code}
> java.lang.RuntimeException: Error while decoding:
> java.util.concurrent.ExecutionException: java.lang.Exception: failed to
> compile: org.codehaus.commons.compiler.CompileException: File
> 'generated.java', Line 98, Column 120: No applicable constructor/method found
> for actual parameters "scala.collection.Seq"; candidates are:
> "line29e7e4b1e36445baa3505b2e102aa86b29.$read$$iw$$iw$$iw$$iw$SpecificCollection(scala.collection.immutable.List)"
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-18891) Support for specific collection types
[
https://issues.apache.org/jira/browse/SPARK-18891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15756842#comment-15756842
]
Michal Šenkýř edited comment on SPARK-18891 at 12/17/16 11:18 AM:
--
[PR #16240|https://github.com/apache/spark/pull/16240] should be relevant. It
adds encoders and deserialization support for arbitrary sequence types
(serialization was already supported). The only thing not working yet is the
{{Seq.toDS}} method due to implicits resolution ({{SparkSession.createDataset}}
works fine).
was (Author: michalsenkyr):
[PR #16240|https://github.com/apache/spark/pull/16240] should be relevant. It
adds deserialization support for arbitrary sequence types (serialization was
already supported). The only thing not working yet is the {{Seq.toDS}} method
due to implicits resolution ({{SparkSession.createDataset}} works fine).
> Support for specific collection types
> -
>
> Key: SPARK-18891
> URL: https://issues.apache.org/jira/browse/SPARK-18891
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 1.6.3, 2.1.0
>Reporter: Michael Armbrust
>Priority: Critical
>
> Encoders treat all collections the same (i.e. {{Seq}} vs {{List}}) which
> force users to only define classes with the most generic type.
> An [example
> error|https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/1023043053387187/2398463439880241/2840265927289860/latest.html]:
> {code}
> case class SpecificCollection(aList: List[Int])
> Seq(SpecificCollection(1 :: Nil)).toDS().collect()
> {code}
> {code}
> java.lang.RuntimeException: Error while decoding:
> java.util.concurrent.ExecutionException: java.lang.Exception: failed to
> compile: org.codehaus.commons.compiler.CompileException: File
> 'generated.java', Line 98, Column 120: No applicable constructor/method found
> for actual parameters "scala.collection.Seq"; candidates are:
> "line29e7e4b1e36445baa3505b2e102aa86b29.$read$$iw$$iw$$iw$$iw$SpecificCollection(scala.collection.immutable.List)"
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-18891) Support for specific collection types
[
https://issues.apache.org/jira/browse/SPARK-18891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15756842#comment-15756842
]
Michal Šenkýř edited comment on SPARK-18891 at 12/17/16 11:18 AM:
--
[PR #16240|https://github.com/apache/spark/pull/16240] should be relevant. It
adds implicit encoders and deserialization support for arbitrary sequence types
(serialization was already supported). The only thing not working yet is the
{{Seq.toDS}} method due to implicits resolution ({{SparkSession.createDataset}}
works fine).
was (Author: michalsenkyr):
[PR #16240|https://github.com/apache/spark/pull/16240] should be relevant. It
adds encoders and deserialization support for arbitrary sequence types
(serialization was already supported). The only thing not working yet is the
{{Seq.toDS}} method due to implicits resolution ({{SparkSession.createDataset}}
works fine).
> Support for specific collection types
> -
>
> Key: SPARK-18891
> URL: https://issues.apache.org/jira/browse/SPARK-18891
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 1.6.3, 2.1.0
>Reporter: Michael Armbrust
>Priority: Critical
>
> Encoders treat all collections the same (i.e. {{Seq}} vs {{List}}) which
> force users to only define classes with the most generic type.
> An [example
> error|https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/1023043053387187/2398463439880241/2840265927289860/latest.html]:
> {code}
> case class SpecificCollection(aList: List[Int])
> Seq(SpecificCollection(1 :: Nil)).toDS().collect()
> {code}
> {code}
> java.lang.RuntimeException: Error while decoding:
> java.util.concurrent.ExecutionException: java.lang.Exception: failed to
> compile: org.codehaus.commons.compiler.CompileException: File
> 'generated.java', Line 98, Column 120: No applicable constructor/method found
> for actual parameters "scala.collection.Seq"; candidates are:
> "line29e7e4b1e36445baa3505b2e102aa86b29.$read$$iw$$iw$$iw$$iw$SpecificCollection(scala.collection.immutable.List)"
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-18912) append to a non-file-based data source table should detect columns number mismatch
Wenchen Fan created SPARK-18912: --- Summary: append to a non-file-based data source table should detect columns number mismatch Key: SPARK-18912 URL: https://issues.apache.org/jira/browse/SPARK-18912 Project: Spark Issue Type: Bug Components: SQL Reporter: Wenchen Fan Assignee: Wenchen Fan -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-18913) append to a table with special column names should work
Wenchen Fan created SPARK-18913: --- Summary: append to a table with special column names should work Key: SPARK-18913 URL: https://issues.apache.org/jira/browse/SPARK-18913 Project: Spark Issue Type: Bug Components: SQL Reporter: Wenchen Fan Assignee: Wenchen Fan -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-18913) append to a table with special column names should work
[ https://issues.apache.org/jira/browse/SPARK-18913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18913: Assignee: Apache Spark (was: Wenchen Fan) > append to a table with special column names should work > --- > > Key: SPARK-18913 > URL: https://issues.apache.org/jira/browse/SPARK-18913 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: Wenchen Fan >Assignee: Apache Spark > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-18912) append to a non-file-based data source table should detect columns number mismatch
[ https://issues.apache.org/jira/browse/SPARK-18912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18912: Assignee: Wenchen Fan (was: Apache Spark) > append to a non-file-based data source table should detect columns number > mismatch > -- > > Key: SPARK-18912 > URL: https://issues.apache.org/jira/browse/SPARK-18912 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: Wenchen Fan >Assignee: Wenchen Fan > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18913) append to a table with special column names should work
[ https://issues.apache.org/jira/browse/SPARK-18913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15756891#comment-15756891 ] Apache Spark commented on SPARK-18913: -- User 'cloud-fan' has created a pull request for this issue: https://github.com/apache/spark/pull/16313 > append to a table with special column names should work > --- > > Key: SPARK-18913 > URL: https://issues.apache.org/jira/browse/SPARK-18913 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: Wenchen Fan >Assignee: Wenchen Fan > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-18912) append to a non-file-based data source table should detect columns number mismatch
[ https://issues.apache.org/jira/browse/SPARK-18912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18912: Assignee: Apache Spark (was: Wenchen Fan) > append to a non-file-based data source table should detect columns number > mismatch > -- > > Key: SPARK-18912 > URL: https://issues.apache.org/jira/browse/SPARK-18912 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: Wenchen Fan >Assignee: Apache Spark > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18912) append to a non-file-based data source table should detect columns number mismatch
[ https://issues.apache.org/jira/browse/SPARK-18912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15756890#comment-15756890 ] Apache Spark commented on SPARK-18912: -- User 'cloud-fan' has created a pull request for this issue: https://github.com/apache/spark/pull/16313 > append to a non-file-based data source table should detect columns number > mismatch > -- > > Key: SPARK-18912 > URL: https://issues.apache.org/jira/browse/SPARK-18912 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: Wenchen Fan >Assignee: Wenchen Fan > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-18913) append to a table with special column names should work
[ https://issues.apache.org/jira/browse/SPARK-18913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18913: Assignee: Wenchen Fan (was: Apache Spark) > append to a table with special column names should work > --- > > Key: SPARK-18913 > URL: https://issues.apache.org/jira/browse/SPARK-18913 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: Wenchen Fan >Assignee: Wenchen Fan > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18879) Spark SQL support for Hive hooks regressed
[ https://issues.apache.org/jira/browse/SPARK-18879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15756917#comment-15756917 ] Sean Owen commented on SPARK-18879: --- I don't think it would be a goal to support Hive functionality. I don't know enough to say what if anything the equivalent is in Spark or whether there would be one. > Spark SQL support for Hive hooks regressed > -- > > Key: SPARK-18879 > URL: https://issues.apache.org/jira/browse/SPARK-18879 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.0, 2.0.2 >Reporter: Atul Payapilly > > As per the stack trace from this post: > http://ihorbobak.com/index.php/2015/05/08/113/ > run on Spark 1.3.1 > hive.exec.pre.hooks Class not found:org.apache.hadoop.hive.ql.hooks.ATSHook > FAILED: Hive Internal Error: > java.lang.ClassNotFoundException(org.apache.hadoop.hive.ql.hooks.ATSHook) > java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.hooks.ATSHook > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:270) > at org.apache.hadoop.hive.ql.hooks.HookUtils.getHooks(HookUtils.java:59) > at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1172) > at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1156) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1206) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1088) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901) > at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:318) > at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:290) > at > org.apache.spark.sql.hive.execution.HiveNativeCommand.run(HiveNativeCommand.scala:33) > at > org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:54) > at > org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:54) > at > org.apache.spark.sql.execution.ExecutedCommand.execute(commands.scala:64) > at > org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:1099) > at > org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:1099) > at org.apache.spark.sql.DataFrame.(DataFrame.scala:147) > at org.apache.spark.sql.DataFrame.(DataFrame.scala:130) > at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:51) > at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:101) > at > org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.run(Shim13.scala:164) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:231) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:218) > at > org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:233) > at > org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:344) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:55) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > It looks like Spark used to rely on the Hive Driver for execution and > supported hive hooks. The current code path does not rely on the Hive Driver > and support for Hive hooks regressed. This is problematic, for example, there > is no way to tell which partitions were updated as part of a query. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-ma
[jira] [Commented] (SPARK-18891) Support for specific collection types
[
https://issues.apache.org/jira/browse/SPARK-18891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15756964#comment-15756964
]
Michal Šenkýř commented on SPARK-18891:
---
I should also add, that {{Seq.toDS}} doesn't work only for non-Seq collections
containing case classes, so the example in the description actually works with
the proposed changes.
> Support for specific collection types
> -
>
> Key: SPARK-18891
> URL: https://issues.apache.org/jira/browse/SPARK-18891
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 1.6.3, 2.1.0
>Reporter: Michael Armbrust
>Priority: Critical
>
> Encoders treat all collections the same (i.e. {{Seq}} vs {{List}}) which
> force users to only define classes with the most generic type.
> An [example
> error|https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/1023043053387187/2398463439880241/2840265927289860/latest.html]:
> {code}
> case class SpecificCollection(aList: List[Int])
> Seq(SpecificCollection(1 :: Nil)).toDS().collect()
> {code}
> {code}
> java.lang.RuntimeException: Error while decoding:
> java.util.concurrent.ExecutionException: java.lang.Exception: failed to
> compile: org.codehaus.commons.compiler.CompileException: File
> 'generated.java', Line 98, Column 120: No applicable constructor/method found
> for actual parameters "scala.collection.Seq"; candidates are:
> "line29e7e4b1e36445baa3505b2e102aa86b29.$read$$iw$$iw$$iw$$iw$SpecificCollection(scala.collection.immutable.List)"
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18648) spark-shell --jars option does not add jars to classpath on windows
[
https://issues.apache.org/jira/browse/SPARK-18648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15757006#comment-15757006
]
Sean Owen commented on SPARK-18648:
---
I see, I get it now. It is complaining about c, not e. The call to textFile is
actually not relevant.
I don't know the answer, but would have thought that {{file:///c:/test/my.jar}}
would work, or {{/c:/test/my.jar}}
[~hyukjin.kwon] given your familiarity with Windows, do you know the right way
to pass a path?
> spark-shell --jars option does not add jars to classpath on windows
> ---
>
> Key: SPARK-18648
> URL: https://issues.apache.org/jira/browse/SPARK-18648
> Project: Spark
> Issue Type: Bug
> Components: Spark Shell, Windows
>Affects Versions: 2.0.2
> Environment: Windows 7 x64
>Reporter: Michel Lemay
> Labels: windows
>
> I can't import symbols from command line jars when in the shell:
> Adding jars via --jars:
> {code}
> spark-shell --master local[*] --jars path\to\deeplearning4j-core-0.7.0.jar
> {code}
> Same result if I add it through maven coordinates:
> {code}spark-shell --master local[*] --packages
> org.deeplearning4j:deeplearning4j-core:0.7.0
> {code}
> I end up with:
> {code}
> scala> import org.deeplearning4j
> :23: error: object deeplearning4j is not a member of package org
>import org.deeplearning4j
> {code}
> NOTE: It is working as expected when running on linux.
> Sample output with --verbose:
> {code}
> Using properties file: null
> Parsed arguments:
> master local[*]
> deployMode null
> executorMemory null
> executorCores null
> totalExecutorCores null
> propertiesFile null
> driverMemorynull
> driverCores null
> driverExtraClassPathnull
> driverExtraLibraryPath null
> driverExtraJavaOptions null
> supervise false
> queue null
> numExecutorsnull
> files null
> pyFiles null
> archivesnull
> mainClass org.apache.spark.repl.Main
> primaryResource spark-shell
> nameSpark shell
> childArgs []
> jars
> file:/C:/Apps/Spark/spark-2.0.2-bin-hadoop2.4/bin/../deeplearning4j-core-0.7.0.jar
> packagesnull
> packagesExclusions null
> repositoriesnull
> verbose true
> Spark properties used, including those specified through
> --conf and those from the properties file null:
> Main class:
> org.apache.spark.repl.Main
> Arguments:
> System properties:
> SPARK_SUBMIT -> true
> spark.app.name -> Spark shell
> spark.jars ->
> file:/C:/Apps/Spark/spark-2.0.2-bin-hadoop2.4/bin/../deeplearning4j-core-0.7.0.jar
> spark.submit.deployMode -> client
> spark.master -> local[*]
> Classpath elements:
> file:/C:/Apps/Spark/spark-2.0.2-bin-hadoop2.4/bin/../deeplearning4j-core-0.7.0.jar
> 16/11/30 08:30:49 WARN NativeCodeLoader: Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 16/11/30 08:30:51 WARN SparkContext: Use an existing SparkContext, some
> configuration may not take effect.
> Spark context Web UI available at http://192.168.70.164:4040
> Spark context available as 'sc' (master = local[*], app id =
> local-1480512651325).
> Spark session available as 'spark'.
> Welcome to
> __
> / __/__ ___ _/ /__
> _\ \/ _ \/ _ `/ __/ '_/
>/___/ .__/\_,_/_/ /_/\_\ version 2.0.2
> /_/
> Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_101)
> Type in expressions to have them evaluated.
> Type :help for more information.
> scala> import org.deeplearning4j
> :23: error: object deeplearning4j is not a member of package org
>import org.deeplearning4j
> ^
> scala>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-18485) Underlying integer overflow when create ChunkedByteBufferOutputStream in MemoryStore
[ https://issues.apache.org/jira/browse/SPARK-18485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-18485. --- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 15915 [https://github.com/apache/spark/pull/15915] > Underlying integer overflow when create ChunkedByteBufferOutputStream in > MemoryStore > > > Key: SPARK-18485 > URL: https://issues.apache.org/jira/browse/SPARK-18485 > Project: Spark > Issue Type: Bug >Affects Versions: 1.6.3, 2.0.2 >Reporter: Genmao Yu > Fix For: 2.2.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-18485) Underlying integer overflow when create ChunkedByteBufferOutputStream in MemoryStore
[ https://issues.apache.org/jira/browse/SPARK-18485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-18485: -- Assignee: Genmao Yu Priority: Minor (was: Major) > Underlying integer overflow when create ChunkedByteBufferOutputStream in > MemoryStore > > > Key: SPARK-18485 > URL: https://issues.apache.org/jira/browse/SPARK-18485 > Project: Spark > Issue Type: Bug >Affects Versions: 1.6.3, 2.0.2 >Reporter: Genmao Yu >Assignee: Genmao Yu >Priority: Minor > Fix For: 2.2.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-18914) Local UDTs test fails due to "ClassCastException: java.lang.Integer cannot be cast to org.apache.spark.sql.UDT$MyDenseVector"
Jacek Laskowski created SPARK-18914:
---
Summary: Local UDTs test fails due to "ClassCastException:
java.lang.Integer cannot be cast to org.apache.spark.sql.UDT$MyDenseVector"
Key: SPARK-18914
URL: https://issues.apache.org/jira/browse/SPARK-18914
Project: Spark
Issue Type: Test
Components: SQL, Tests
Affects Versions: 2.2.0
Reporter: Jacek Laskowski
Priority: Minor
{code}
[info] - Local UDTs *** FAILED *** (110 milliseconds)
[info] java.lang.ClassCastException: java.lang.Integer cannot be cast to
org.apache.spark.sql.UDT$MyDenseVector
[info] at
org.apache.spark.sql.UserDefinedTypeSuite$$anonfun$9.apply(UserDefinedTypeSuite.scala:135)
[info] at
org.apache.spark.sql.UserDefinedTypeSuite$$anonfun$9.apply(UserDefinedTypeSuite.scala:129)
[info] at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
[info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info] at org.scalatest.Transformer.apply(Transformer.scala:22)
[info] at org.scalatest.Transformer.apply(Transformer.scala:20)
[info] at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
[info] at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:68)
[info] at
org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:183)
[info] at
org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:196)
[info] at
org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:196)
[info] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
[info] at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:196)
[info] at
org.apache.spark.sql.UserDefinedTypeSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(UserDefinedTypeSuite.scala:74)
[info] at
org.scalatest.BeforeAndAfterEach$class.runTest(BeforeAndAfterEach.scala:221)
[info] at
org.apache.spark.sql.UserDefinedTypeSuite.runTest(UserDefinedTypeSuite.scala:74)
[info] at
org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:229)
[info] at
org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:229)
[info] at
org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:396)
[info] at
org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384)
[info] at scala.collection.immutable.List.foreach(List.scala:381)
[info] at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
[info] at
org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:379)
[info] at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
[info] at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:229)
[info] at org.scalatest.FunSuite.runTests(FunSuite.scala:1560)
[info] at org.scalatest.Suite$class.run(Suite.scala:1147)
[info] at
org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560)
[info] at
org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:233)
[info] at
org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:233)
[info] at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
[info] at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:233)
[info] at
org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:31)
[info] at
org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:213)
[info] at
org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:210)
[info] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:31)
[info] at
org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
[info] at
org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:480)
[info] at sbt.ForkMain$Run$2.call(ForkMain.java:296)
[info] at sbt.ForkMain$Run$2.call(ForkMain.java:286)
[info] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[info] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[info] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[info] at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-18914) Local UDTs test (org.apache.spark.sql.UserDefinedTypeSuite) fails due to "ClassCastException: java.lang.Integer cannot be cast to org.apache.spark.sql.UDT$MyDenseVector"
[
https://issues.apache.org/jira/browse/SPARK-18914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jacek Laskowski updated SPARK-18914:
Summary: Local UDTs test (org.apache.spark.sql.UserDefinedTypeSuite) fails
due to "ClassCastException: java.lang.Integer cannot be cast to
org.apache.spark.sql.UDT$MyDenseVector" (was: Local UDTs test fails due to
"ClassCastException: java.lang.Integer cannot be cast to
org.apache.spark.sql.UDT$MyDenseVector")
> Local UDTs test (org.apache.spark.sql.UserDefinedTypeSuite) fails due to
> "ClassCastException: java.lang.Integer cannot be cast to
> org.apache.spark.sql.UDT$MyDenseVector"
> -
>
> Key: SPARK-18914
> URL: https://issues.apache.org/jira/browse/SPARK-18914
> Project: Spark
> Issue Type: Test
> Components: SQL, Tests
>Affects Versions: 2.2.0
>Reporter: Jacek Laskowski
>Priority: Minor
>
> {code}
> [info] ScalaTest
> [info] Run completed in 1 hour, 44 minutes, 27 seconds.
> [info] Total number of tests run: 2710
> [info] Suites: completed 173, aborted 0
> [info] Tests: succeeded 2709, failed 1, canceled 0, ignored 53, pending 0
> [info] *** 1 TEST FAILED ***
> [error] Failed: Total 2710, Failed 1, Errors 0, Passed 2709, Ignored 53
> [error] Failed tests:
> [error] org.apache.spark.sql.UserDefinedTypeSuite
> {code}
> {code}
> [info] - Local UDTs *** FAILED *** (110 milliseconds)
> [info] java.lang.ClassCastException: java.lang.Integer cannot be cast to
> org.apache.spark.sql.UDT$MyDenseVector
> [info] at
> org.apache.spark.sql.UserDefinedTypeSuite$$anonfun$9.apply(UserDefinedTypeSuite.scala:135)
> [info] at
> org.apache.spark.sql.UserDefinedTypeSuite$$anonfun$9.apply(UserDefinedTypeSuite.scala:129)
> [info] at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
> [info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> [info] at org.scalatest.Transformer.apply(Transformer.scala:22)
> [info] at org.scalatest.Transformer.apply(Transformer.scala:20)
> [info] at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
> [info] at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:68)
> [info] at
> org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:183)
> [info] at
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:196)
> [info] at
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:196)
> [info] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
> [info] at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:196)
> [info] at
> org.apache.spark.sql.UserDefinedTypeSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(UserDefinedTypeSuite.scala:74)
> [info] at
> org.scalatest.BeforeAndAfterEach$class.runTest(BeforeAndAfterEach.scala:221)
> [info] at
> org.apache.spark.sql.UserDefinedTypeSuite.runTest(UserDefinedTypeSuite.scala:74)
> [info] at
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:229)
> [info] at
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:229)
> [info] at
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:396)
> [info] at
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384)
> [info] at scala.collection.immutable.List.foreach(List.scala:381)
> [info] at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
> [info] at
> org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:379)
> [info] at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
> [info] at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:229)
> [info] at org.scalatest.FunSuite.runTests(FunSuite.scala:1560)
> [info] at org.scalatest.Suite$class.run(Suite.scala:1147)
> [info] at
> org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560)
> [info] at
> org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:233)
> [info] at
> org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:233)
> [info] at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
> [info] at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:233)
> [info] at
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:31)
> [info] at
> org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:213)
> [info] at
> org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:210)
> [info] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:31)
> [info] at
> org.scalatest.tools.Framework.org$scalatest$tools$Framework$$
[jira] [Updated] (SPARK-18914) Local UDTs test fails due to "ClassCastException: java.lang.Integer cannot be cast to org.apache.spark.sql.UDT$MyDenseVector"
[
https://issues.apache.org/jira/browse/SPARK-18914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jacek Laskowski updated SPARK-18914:
Description:
{code}
[info] ScalaTest
[info] Run completed in 1 hour, 44 minutes, 27 seconds.
[info] Total number of tests run: 2710
[info] Suites: completed 173, aborted 0
[info] Tests: succeeded 2709, failed 1, canceled 0, ignored 53, pending 0
[info] *** 1 TEST FAILED ***
[error] Failed: Total 2710, Failed 1, Errors 0, Passed 2709, Ignored 53
[error] Failed tests:
[error] org.apache.spark.sql.UserDefinedTypeSuite
{code}
{code}
[info] - Local UDTs *** FAILED *** (110 milliseconds)
[info] java.lang.ClassCastException: java.lang.Integer cannot be cast to
org.apache.spark.sql.UDT$MyDenseVector
[info] at
org.apache.spark.sql.UserDefinedTypeSuite$$anonfun$9.apply(UserDefinedTypeSuite.scala:135)
[info] at
org.apache.spark.sql.UserDefinedTypeSuite$$anonfun$9.apply(UserDefinedTypeSuite.scala:129)
[info] at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
[info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info] at org.scalatest.Transformer.apply(Transformer.scala:22)
[info] at org.scalatest.Transformer.apply(Transformer.scala:20)
[info] at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
[info] at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:68)
[info] at
org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:183)
[info] at
org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:196)
[info] at
org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:196)
[info] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
[info] at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:196)
[info] at
org.apache.spark.sql.UserDefinedTypeSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(UserDefinedTypeSuite.scala:74)
[info] at
org.scalatest.BeforeAndAfterEach$class.runTest(BeforeAndAfterEach.scala:221)
[info] at
org.apache.spark.sql.UserDefinedTypeSuite.runTest(UserDefinedTypeSuite.scala:74)
[info] at
org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:229)
[info] at
org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:229)
[info] at
org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:396)
[info] at
org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384)
[info] at scala.collection.immutable.List.foreach(List.scala:381)
[info] at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
[info] at
org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:379)
[info] at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
[info] at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:229)
[info] at org.scalatest.FunSuite.runTests(FunSuite.scala:1560)
[info] at org.scalatest.Suite$class.run(Suite.scala:1147)
[info] at
org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560)
[info] at
org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:233)
[info] at
org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:233)
[info] at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
[info] at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:233)
[info] at
org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:31)
[info] at
org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:213)
[info] at
org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:210)
[info] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:31)
[info] at
org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
[info] at
org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:480)
[info] at sbt.ForkMain$Run$2.call(ForkMain.java:296)
[info] at sbt.ForkMain$Run$2.call(ForkMain.java:286)
[info] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[info] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[info] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[info] at java.lang.Thread.run(Thread.java:745)
{code}
was:
{code}
[info] - Local UDTs *** FAILED *** (110 milliseconds)
[info] java.lang.ClassCastException: java.lang.Integer cannot be cast to
org.apache.spark.sql.UDT$MyDenseVector
[info] at
org.apache.spark.sql.UserDefinedTypeSuite$$anonfun$9.apply(UserDefinedTypeSuite.scala:135)
[info] at
org.apache.spark.sql.UserDefinedTypeSuite$$anonfun$9.apply(UserDefinedTypeSuite.scala:129)
[info] at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
[info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info] at org.scalatest
[jira] [Commented] (SPARK-18914) Local UDTs test (org.apache.spark.sql.UserDefinedTypeSuite) fails due to "ClassCastException: java.lang.Integer cannot be cast to org.apache.spark.sql.UDT$MyDenseVecto
[
https://issues.apache.org/jira/browse/SPARK-18914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15757061#comment-15757061
]
Sean Owen commented on SPARK-18914:
---
Where does this fail? it doesn't fail on Jenkins builds, evidently.
> Local UDTs test (org.apache.spark.sql.UserDefinedTypeSuite) fails due to
> "ClassCastException: java.lang.Integer cannot be cast to
> org.apache.spark.sql.UDT$MyDenseVector"
> -
>
> Key: SPARK-18914
> URL: https://issues.apache.org/jira/browse/SPARK-18914
> Project: Spark
> Issue Type: Test
> Components: SQL, Tests
>Affects Versions: 2.2.0
>Reporter: Jacek Laskowski
>Priority: Minor
>
> {code}
> [info] ScalaTest
> [info] Run completed in 1 hour, 44 minutes, 27 seconds.
> [info] Total number of tests run: 2710
> [info] Suites: completed 173, aborted 0
> [info] Tests: succeeded 2709, failed 1, canceled 0, ignored 53, pending 0
> [info] *** 1 TEST FAILED ***
> [error] Failed: Total 2710, Failed 1, Errors 0, Passed 2709, Ignored 53
> [error] Failed tests:
> [error] org.apache.spark.sql.UserDefinedTypeSuite
> {code}
> {code}
> [info] - Local UDTs *** FAILED *** (110 milliseconds)
> [info] java.lang.ClassCastException: java.lang.Integer cannot be cast to
> org.apache.spark.sql.UDT$MyDenseVector
> [info] at
> org.apache.spark.sql.UserDefinedTypeSuite$$anonfun$9.apply(UserDefinedTypeSuite.scala:135)
> [info] at
> org.apache.spark.sql.UserDefinedTypeSuite$$anonfun$9.apply(UserDefinedTypeSuite.scala:129)
> [info] at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
> [info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> [info] at org.scalatest.Transformer.apply(Transformer.scala:22)
> [info] at org.scalatest.Transformer.apply(Transformer.scala:20)
> [info] at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
> [info] at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:68)
> [info] at
> org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:183)
> [info] at
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:196)
> [info] at
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:196)
> [info] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
> [info] at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:196)
> [info] at
> org.apache.spark.sql.UserDefinedTypeSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(UserDefinedTypeSuite.scala:74)
> [info] at
> org.scalatest.BeforeAndAfterEach$class.runTest(BeforeAndAfterEach.scala:221)
> [info] at
> org.apache.spark.sql.UserDefinedTypeSuite.runTest(UserDefinedTypeSuite.scala:74)
> [info] at
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:229)
> [info] at
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:229)
> [info] at
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:396)
> [info] at
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384)
> [info] at scala.collection.immutable.List.foreach(List.scala:381)
> [info] at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
> [info] at
> org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:379)
> [info] at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
> [info] at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:229)
> [info] at org.scalatest.FunSuite.runTests(FunSuite.scala:1560)
> [info] at org.scalatest.Suite$class.run(Suite.scala:1147)
> [info] at
> org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560)
> [info] at
> org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:233)
> [info] at
> org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:233)
> [info] at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
> [info] at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:233)
> [info] at
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:31)
> [info] at
> org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:213)
> [info] at
> org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:210)
> [info] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:31)
> [info] at
> org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
> [info] at
> org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:480)
> [info] at sbt.ForkMain$Run$2.call(ForkMain.java:296)
> [info] at sbt.ForkMain$Run
[jira] [Resolved] (SPARK-18829) Printing to logger
[
https://issues.apache.org/jira/browse/SPARK-18829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-18829.
---
Resolution: Not A Problem
I think the existing API provides all this info and the caller can log it as
desired.
> Printing to logger
> --
>
> Key: SPARK-18829
> URL: https://issues.apache.org/jira/browse/SPARK-18829
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 1.6.2
> Environment: ALL
>Reporter: David Hodeffi
>Priority: Trivial
> Labels: easyfix, patch
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> I would like to print dataframe.show or df.explain(true) into log file.
> right now the code print to standard output without a way to redirect it.
> It also cannot be configured on log4j.properties.
> My suggestion is to write to the logger and standard output.
> i.e
> class DataFrame {..
> override def explain(extended: Boolean): Unit = {
> val explain = ExplainCommand(queryExecution.logical, extended = extended)
> sqlContext.executePlan(explain).executedPlan.executeCollect().foreach {
> // scalastyle:off println
> r => {
> println(r.getString(0))
> logger.debug(r.getString(0))
> }
> }
> // scalastyle:on println
> }
> }
> def show(numRows: Int, truncate: Boolean): Unit = {
> val str =showString(numRows, truncate)
> println(str)
> logger.debug(str)
> }
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18914) Local UDTs test (org.apache.spark.sql.UserDefinedTypeSuite) fails due to "ClassCastException: java.lang.Integer cannot be cast to org.apache.spark.sql.UDT$MyDenseVecto
[
https://issues.apache.org/jira/browse/SPARK-18914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15757105#comment-15757105
]
Jacek Laskowski commented on SPARK-18914:
-
That's mysterious indeed as it fails consistently on my laptop with no local
changes but for ScalaTest 3.1.0 (in https://github.com/apache/spark/pull/16309).
> Local UDTs test (org.apache.spark.sql.UserDefinedTypeSuite) fails due to
> "ClassCastException: java.lang.Integer cannot be cast to
> org.apache.spark.sql.UDT$MyDenseVector"
> -
>
> Key: SPARK-18914
> URL: https://issues.apache.org/jira/browse/SPARK-18914
> Project: Spark
> Issue Type: Test
> Components: SQL, Tests
>Affects Versions: 2.2.0
>Reporter: Jacek Laskowski
>Priority: Minor
>
> {code}
> [info] ScalaTest
> [info] Run completed in 1 hour, 44 minutes, 27 seconds.
> [info] Total number of tests run: 2710
> [info] Suites: completed 173, aborted 0
> [info] Tests: succeeded 2709, failed 1, canceled 0, ignored 53, pending 0
> [info] *** 1 TEST FAILED ***
> [error] Failed: Total 2710, Failed 1, Errors 0, Passed 2709, Ignored 53
> [error] Failed tests:
> [error] org.apache.spark.sql.UserDefinedTypeSuite
> {code}
> {code}
> [info] - Local UDTs *** FAILED *** (110 milliseconds)
> [info] java.lang.ClassCastException: java.lang.Integer cannot be cast to
> org.apache.spark.sql.UDT$MyDenseVector
> [info] at
> org.apache.spark.sql.UserDefinedTypeSuite$$anonfun$9.apply(UserDefinedTypeSuite.scala:135)
> [info] at
> org.apache.spark.sql.UserDefinedTypeSuite$$anonfun$9.apply(UserDefinedTypeSuite.scala:129)
> [info] at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
> [info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> [info] at org.scalatest.Transformer.apply(Transformer.scala:22)
> [info] at org.scalatest.Transformer.apply(Transformer.scala:20)
> [info] at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
> [info] at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:68)
> [info] at
> org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:183)
> [info] at
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:196)
> [info] at
> org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:196)
> [info] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
> [info] at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:196)
> [info] at
> org.apache.spark.sql.UserDefinedTypeSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(UserDefinedTypeSuite.scala:74)
> [info] at
> org.scalatest.BeforeAndAfterEach$class.runTest(BeforeAndAfterEach.scala:221)
> [info] at
> org.apache.spark.sql.UserDefinedTypeSuite.runTest(UserDefinedTypeSuite.scala:74)
> [info] at
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:229)
> [info] at
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:229)
> [info] at
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:396)
> [info] at
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384)
> [info] at scala.collection.immutable.List.foreach(List.scala:381)
> [info] at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
> [info] at
> org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:379)
> [info] at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
> [info] at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:229)
> [info] at org.scalatest.FunSuite.runTests(FunSuite.scala:1560)
> [info] at org.scalatest.Suite$class.run(Suite.scala:1147)
> [info] at
> org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560)
> [info] at
> org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:233)
> [info] at
> org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:233)
> [info] at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
> [info] at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:233)
> [info] at
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:31)
> [info] at
> org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:213)
> [info] at
> org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:210)
> [info] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:31)
> [info] at
> org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
> [info] at
> org.scalatest.tools.Framework$ScalaTestTask.execute(Frame
[jira] [Commented] (SPARK-18648) spark-shell --jars option does not add jars to classpath on windows
[
https://issues.apache.org/jira/browse/SPARK-18648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15757178#comment-15757178
]
Hyukjin Kwon commented on SPARK-18648:
--
Thank you for cc'ing me.
Yes, I think {{textFile}} seems irrelevant. Also, yes, I think the safe choice
is always to use a URI format on Windows where applicable.
I just manually tested and I can reproduce this on Windows although I see these
seem parsed fine as below with {{--jars}}.
{code}
C:\spark\bin>spark-shell.cmd --jars
C:\Users\IEUser\Downloads\spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
{code}
C:\spark\bin>spark-shell.cmd --jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
{code}
C:\spark\bin>spark-shell.cmd --jars
/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
In those cases, It seems I can't import the contents in the jar on Windows
whereas I can on my Mac if I haven't done something wrong.
Personally, in my case, I always use
{{--driver-library-path XXX.jar}} and {{--conf
spark.executor.extraLibraryPath=XXX.jar}} in `spark-shell` or `spark-shell.cmd`
for sure.
or
{code}
spark.driver.extraClassPath
spark.executor.extraClassPath
{code}
in the configuration.
Anyway, I believe the case below {{C:/a/b/c}} is discouraged because I have
seen this is failed to be parsed properly as a URI ending up with recognising
C: as the scheme.
{code}
C:\spark\bin>spark-shell.cmd --jars
C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
Actually, I noticed such suspicious cases across Spark so I wanted to fix and
and fix them after making the tests passed on Windows all.
So, to cut this short,
- I can reproduce this on Windows
- Workaround seems setting {{spark.driver.extraClassPath}} and
{{spark.executor.extraClassPath}} too (or equivalent options).
- The right paths should be URI forms as suggested above.
- I think we should investigate this.
> spark-shell --jars option does not add jars to classpath on windows
> ---
>
> Key: SPARK-18648
> URL: https://issues.apache.org/jira/browse/SPARK-18648
> Project: Spark
> Issue Type: Bug
> Components: Spark Shell, Windows
>Affects Versions: 2.0.2
> Environment: Windows 7 x64
>Reporter: Michel Lemay
> Labels: windows
>
> I can't import symbols from command line jars when in the shell:
> Adding jars via --jars:
> {code}
> spark-shell --master local[*] --jars path\to\deeplearning4j-core-0.7.0.jar
> {code}
> Same result if I add it through maven coordinates:
> {code}spark-shell --master local[*] --packages
> org.deeplearning4j:deeplearning4j-core:0.7.0
> {code}
> I end up with:
> {code}
> scala> import org.deeplearning4j
> :23: error: object deeplearning4j is not a member of package org
>import org.deeplearning4j
> {code}
> NOTE: It is working as expected when running on linux.
> Sample output with --verbose:
> {code}
> Using properties file: null
> Parsed arguments:
> master local[*]
> deployMode null
> executorMemory null
> executorCores null
> totalExecutorCores null
> propertiesFile null
> driverMemorynull
> driverCores null
> driverExtraClassPathnull
> driverExtraLibraryPath null
> driverExtraJavaOptions null
> supervise false
> queue null
> numExecutorsnull
> files null
> pyFiles null
> archivesnull
> mainClass org.apache.spark.repl.Main
> primaryResource spark-shell
> nameSpark shell
> childArgs []
> jars
> file:/C:/Apps/Spark/spark-2.0.2-bin-hadoop2.4/bin/../deeplearning4j-core-0.7.0.jar
> packagesnull
> packagesExclusions null
> repositoriesnull
> verbose true
> Spark properties used, including those specified through
> --conf and those from the properties file null:
> Main class:
> org.apache.spark.repl.Main
> Arguments:
> System properties:
> SPARK_SUBMIT -> true
> spark.app.name -> Spark shell
> spark.jars ->
> file:/C:/Apps/Spark/spark-2.0.2-bin-hadoop2.4/bin/../deeplearning4j-core-0.7.0.jar
> spark.submit.de
[jira] [Comment Edited] (SPARK-18648) spark-shell --jars option does not add jars to classpath on windows
[
https://issues.apache.org/jira/browse/SPARK-18648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15757178#comment-15757178
]
Hyukjin Kwon edited comment on SPARK-18648 at 12/17/16 3:21 PM:
Thank you for cc'ing me.
Yes, I think {{textFile}} seems irrelevant. Also, yes, I think the safe choice
is always to use a URI format on Windows where applicable.
I just manually tested and I can reproduce this on Windows although I see these
seem parsed fine as below with {{--jars}}.
{code}
C:\spark\bin>spark-shell.cmd --jars
C:\Users\IEUser\Downloads\spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
{code}
C:\spark\bin>spark-shell.cmd --jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
{code}
C:\spark\bin>spark-shell.cmd --jars
/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
In those cases, It seems I can't import the contents in the jar on Windows
whereas I can on my Mac if I haven't done something wrong.
Personally, in my case, I always use {{--driver-library-path XXX.jar}} and
{{--conf spark.executor.extraLibraryPath=XXX.jar}} in {{spark-shell}} or
{{spark-shell.cmd}} for sure.
or
{code}
spark.driver.extraClassPath
spark.executor.extraClassPath
{code}
in the configuration.
Anyway, I believe the case below {{C:/a/b/c}} is discouraged because I have
seen this is failed to be parsed properly as a URI ending up with recognising
C: as the scheme.
{code}
C:\spark\bin>spark-shell.cmd --jars
C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
Actually, I noticed such suspicious cases across Spark so I wanted to fix and
and fix them after making the tests passed on Windows all.
So, to cut this short,
- I can reproduce this on Windows
- Workaround seems setting {{spark.driver.extraClassPath}} and
{{spark.executor.extraClassPath}} too (or equivalent options).
- The right paths should be URI forms as suggested above.
- I think we should investigate this.
was (Author: hyukjin.kwon):
Thank you for cc'ing me.
Yes, I think {{textFile}} seems irrelevant. Also, yes, I think the safe choice
is always to use a URI format on Windows where applicable.
I just manually tested and I can reproduce this on Windows although I see these
seem parsed fine as below with {{--jars}}.
{code}
C:\spark\bin>spark-shell.cmd --jars
C:\Users\IEUser\Downloads\spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
{code}
C:\spark\bin>spark-shell.cmd --jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
{code}
C:\spark\bin>spark-shell.cmd --jars
/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
In those cases, It seems I can't import the contents in the jar on Windows
whereas I can on my Mac if I haven't done something wrong.
Personally, in my case, I always use
"--driver-library-path XXX.jar" and "--conf
spark.executor.extraLibraryPath=XXX.jar" in {{spark-shell}} or
{{spark-shell.cmd}} for sure.
or
{code}
spark.driver.extraClassPath
spark.executor.extraClassPath
{code}
in the configuration.
Anyway, I believe the case below {{C:/a/b/c}} is discouraged because I have
seen this is failed to be parsed properly as a URI ending up with recognising
C: as the scheme.
{code}
C:\spark\bin>spark-shell.cmd --jars
C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
Actually, I noticed such suspicious cases across Spark so I wanted to fix and
and fix them after making the tests passed on Windows all.
So, to cut this short,
- I can reproduce this on Windows
- Workaround seems setting {{spark.driver.extraClassPath}} and
{{spark.executor.extraClassPath}} too (or equivalent options).
- The right paths should be URI forms as suggested above.
- I think we should investigate this.
> spark-shell --jars option does not add jars to classpath on windows
> ---
>
>
[jira] [Comment Edited] (SPARK-18648) spark-shell --jars option does not add jars to classpath on windows
[
https://issues.apache.org/jira/browse/SPARK-18648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15757178#comment-15757178
]
Hyukjin Kwon edited comment on SPARK-18648 at 12/17/16 3:20 PM:
Thank you for cc'ing me.
Yes, I think {{textFile}} seems irrelevant. Also, yes, I think the safe choice
is always to use a URI format on Windows where applicable.
I just manually tested and I can reproduce this on Windows although I see these
seem parsed fine as below with {{--jars}}.
{code}
C:\spark\bin>spark-shell.cmd --jars
C:\Users\IEUser\Downloads\spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
{code}
C:\spark\bin>spark-shell.cmd --jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
{code}
C:\spark\bin>spark-shell.cmd --jars
/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
In those cases, It seems I can't import the contents in the jar on Windows
whereas I can on my Mac if I haven't done something wrong.
Personally, in my case, I always use
"--driver-library-path XXX.jar" and "--conf
spark.executor.extraLibraryPath=XXX.jar" in {{spark-shell}} or
{{spark-shell.cmd}} for sure.
or
{code}
spark.driver.extraClassPath
spark.executor.extraClassPath
{code}
in the configuration.
Anyway, I believe the case below {{C:/a/b/c}} is discouraged because I have
seen this is failed to be parsed properly as a URI ending up with recognising
C: as the scheme.
{code}
C:\spark\bin>spark-shell.cmd --jars
C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
Actually, I noticed such suspicious cases across Spark so I wanted to fix and
and fix them after making the tests passed on Windows all.
So, to cut this short,
- I can reproduce this on Windows
- Workaround seems setting {{spark.driver.extraClassPath}} and
{{spark.executor.extraClassPath}} too (or equivalent options).
- The right paths should be URI forms as suggested above.
- I think we should investigate this.
was (Author: hyukjin.kwon):
Thank you for cc'ing me.
Yes, I think {{textFile}} seems irrelevant. Also, yes, I think the safe choice
is always to use a URI format on Windows where applicable.
I just manually tested and I can reproduce this on Windows although I see these
seem parsed fine as below with {{--jars}}.
{code}
C:\spark\bin>spark-shell.cmd --jars
C:\Users\IEUser\Downloads\spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
{code}
C:\spark\bin>spark-shell.cmd --jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
{code}
C:\spark\bin>spark-shell.cmd --jars
/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
In those cases, It seems I can't import the contents in the jar on Windows
whereas I can on my Mac if I haven't done something wrong.
Personally, in my case, I always use
{{--driver-library-path XXX.jar}} and {{--conf
spark.executor.extraLibraryPath=XXX.jar}} in `spark-shell` or `spark-shell.cmd`
for sure.
or
{code}
spark.driver.extraClassPath
spark.executor.extraClassPath
{code}
in the configuration.
Anyway, I believe the case below {{C:/a/b/c}} is discouraged because I have
seen this is failed to be parsed properly as a URI ending up with recognising
C: as the scheme.
{code}
C:\spark\bin>spark-shell.cmd --jars
C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
Actually, I noticed such suspicious cases across Spark so I wanted to fix and
and fix them after making the tests passed on Windows all.
So, to cut this short,
- I can reproduce this on Windows
- Workaround seems setting {{spark.driver.extraClassPath}} and
{{spark.executor.extraClassPath}} too (or equivalent options).
- The right paths should be URI forms as suggested above.
- I think we should investigate this.
> spark-shell --jars option does not add jars to classpath on windows
> ---
>
>
[jira] [Comment Edited] (SPARK-18648) spark-shell --jars option does not add jars to classpath on windows
[
https://issues.apache.org/jira/browse/SPARK-18648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15757178#comment-15757178
]
Hyukjin Kwon edited comment on SPARK-18648 at 12/17/16 3:24 PM:
Thank you for cc'ing me.
Yes, I think {{textFile}} seems irrelevant. Also, yes, I think the safe choice
is always to use a URI format on Windows where applicable.
I just manually tested and I can reproduce this on Windows although I see these
seem parsed fine as below with {{--jars}}.
{code}
C:\spark\bin>spark-shell.cmd --jars
C:\Users\IEUser\Downloads\spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
{code}
C:\spark\bin>spark-shell.cmd --jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
{code}
C:\spark\bin>spark-shell.cmd --jars
/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
In those cases, It seems I can't import the contents in the jar on Windows
whereas I can on my Mac if I haven't done something wrong.
Personally, in my case, I always use {{driver-library-path}} and
{{spark.executor.extraLibraryPath}} in {{spark-shell}} or {{spark-shell.cmd}}
for sure.
or
{code}
spark.driver.extraLibraryPath
spark.executor.extraLibraryPath
{code}
in the configuration.
Anyway, I believe the case below {{C:/a/b/c}} is discouraged because I have
seen this is failed to be parsed properly as a URI ending up with recognising
C: as the scheme.
{code}
C:\spark\bin>spark-shell.cmd --jars
C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
Actually, I noticed such suspicious cases across Spark so I wanted to fix and
and fix them after making the tests passed on Windows all.
So, to cut this short,
- I can reproduce this on Windows
- Workaround seems setting {{spark.driver.extraClassPath}} and
{{spark.executor.extraClassPath}} too (or equivalent options).
- The right paths should be URI forms as suggested above.
- I think we should investigate this.
was (Author: hyukjin.kwon):
Thank you for cc'ing me.
Yes, I think {{textFile}} seems irrelevant. Also, yes, I think the safe choice
is always to use a URI format on Windows where applicable.
I just manually tested and I can reproduce this on Windows although I see these
seem parsed fine as below with {{--jars}}.
{code}
C:\spark\bin>spark-shell.cmd --jars
C:\Users\IEUser\Downloads\spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
{code}
C:\spark\bin>spark-shell.cmd --jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
{code}
C:\spark\bin>spark-shell.cmd --jars
/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
In those cases, It seems I can't import the contents in the jar on Windows
whereas I can on my Mac if I haven't done something wrong.
Personally, in my case, I always use {{--driver-library-path XXX.jar}} and
{{--conf spark.executor.extraLibraryPath=XXX.jar}} in {{spark-shell}} or
{{spark-shell.cmd}} for sure.
or
{code}
spark.driver.extraClassPath
spark.executor.extraClassPath
{code}
in the configuration.
Anyway, I believe the case below {{C:/a/b/c}} is discouraged because I have
seen this is failed to be parsed properly as a URI ending up with recognising
C: as the scheme.
{code}
C:\spark\bin>spark-shell.cmd --jars
C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
Actually, I noticed such suspicious cases across Spark so I wanted to fix and
and fix them after making the tests passed on Windows all.
So, to cut this short,
- I can reproduce this on Windows
- Workaround seems setting {{spark.driver.extraClassPath}} and
{{spark.executor.extraClassPath}} too (or equivalent options).
- The right paths should be URI forms as suggested above.
- I think we should investigate this.
> spark-shell --jars option does not add jars to classpath on windows
> ---
>
> Key: SPARK-18
[jira] [Comment Edited] (SPARK-18648) spark-shell --jars option does not add jars to classpath on windows
[
https://issues.apache.org/jira/browse/SPARK-18648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15757178#comment-15757178
]
Hyukjin Kwon edited comment on SPARK-18648 at 12/17/16 3:29 PM:
Thank you for cc'ing me.
Yes, I think {{textFile}} seems irrelevant. Also, yes, I think the safe choice
is always to use a URI format on Windows where applicable.
I just manually tested and I can reproduce this on Windows although I see these
seem parsed fine as below with {{--jars}}.
{code}
C:\spark\bin>spark-shell.cmd --jars
C:\Users\IEUser\Downloads\spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
{code}
C:\spark\bin>spark-shell.cmd --jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
{code}
C:\spark\bin>spark-shell.cmd --jars
/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
In those cases, It seems I can't import the contents in the jar on Windows
whereas I can on my Mac if I haven't done something wrong.
Personally, in my case, I always use options related with path and classpaths
such as {{driver-library-path}} and {{spark.executor.extraLibraryPath}} in
{{spark-shell}} or {{spark-shell.cmd}} for sure.
or
{code}
spark.driver.extraClassPath
spark.driver.extraLibraryPath
spark.executor.extraClassPath
spark.executor.extraLibraryPath
{code}
in the configuration.
Anyway, I believe the case below {{C:/a/b/c}} is discouraged because I have
seen this is failed to be parsed properly as a URI ending up with recognising
C: as the scheme.
{code}
C:\spark\bin>spark-shell.cmd --jars
C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
Actually, I noticed such suspicious cases across Spark so I wanted to fix and
and fix them after making the tests passed on Windows all.
So, to cut this short,
- I can reproduce this on Windows
- Workaround seems setting options such as {{spark.driver.extraClassPath}} and
{{spark.executor.extraClassPath}} (or equivalent options).
- The right paths should be URI forms as suggested above.
- I think we should investigate this.
was (Author: hyukjin.kwon):
Thank you for cc'ing me.
Yes, I think {{textFile}} seems irrelevant. Also, yes, I think the safe choice
is always to use a URI format on Windows where applicable.
I just manually tested and I can reproduce this on Windows although I see these
seem parsed fine as below with {{--jars}}.
{code}
C:\spark\bin>spark-shell.cmd --jars
C:\Users\IEUser\Downloads\spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
{code}
C:\spark\bin>spark-shell.cmd --jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
{code}
C:\spark\bin>spark-shell.cmd --jars
/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
In those cases, It seems I can't import the contents in the jar on Windows
whereas I can on my Mac if I haven't done something wrong.
Personally, in my case, I always use {{driver-library-path}} and
{{spark.executor.extraLibraryPath}} in {{spark-shell}} or {{spark-shell.cmd}}
for sure.
or
{code}
spark.driver.extraLibraryPath
spark.executor.extraLibraryPath
{code}
in the configuration.
Anyway, I believe the case below {{C:/a/b/c}} is discouraged because I have
seen this is failed to be parsed properly as a URI ending up with recognising
C: as the scheme.
{code}
C:\spark\bin>spark-shell.cmd --jars
C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
Actually, I noticed such suspicious cases across Spark so I wanted to fix and
and fix them after making the tests passed on Windows all.
So, to cut this short,
- I can reproduce this on Windows
- Workaround seems setting {{spark.driver.extraClassPath}} and
{{spark.executor.extraClassPath}} too (or equivalent options).
- The right paths should be URI forms as suggested above.
- I think we should investigate this.
> spark-shell --jars option does not add jars to classpath on windows
> --
[jira] [Comment Edited] (SPARK-18648) spark-shell --jars option does not add jars to classpath on windows
[
https://issues.apache.org/jira/browse/SPARK-18648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15757178#comment-15757178
]
Hyukjin Kwon edited comment on SPARK-18648 at 12/17/16 3:45 PM:
Thank you for cc'ing me.
Yes, I think {{textFile}} seems irrelevant. Also, yes, I think the safe choice
is always to use a URI format on Windows where applicable.
I just manually tested and I can reproduce this on Windows although I see these
seem parsed fine as below with {{--jars}}.
{code}
C:\spark\bin>spark-shell.cmd --jars
C:\Users\IEUser\Downloads\spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
{code}
C:\spark\bin>spark-shell.cmd --jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
{code}
C:\spark\bin>spark-shell.cmd --jars
/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
In those cases, It seems I can't import the contents in the jar on Windows
whereas I can on my Mac if I haven't done something wrong.
Personally, in my case, I always use options related with path and classpaths
such as {{driver-library-path}} and {{spark.executor.extraLibraryPath}} in
{{spark-shell}} or {{spark-shell.cmd}} for sure (I use JNI).
or
{code}
spark.driver.extraClassPath
spark.driver.extraLibraryPath
spark.executor.extraClassPath
spark.executor.extraLibraryPath
{code}
in the configuration.
Anyway, I believe the case below {{C:/a/b/c}} is discouraged because I have
seen this is failed to be parsed properly as a URI ending up with recognising
C: as the scheme.
{code}
C:\spark\bin>spark-shell.cmd --jars
C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
Actually, I noticed such suspicious cases across Spark so I wanted to fix and
and fix them after making the tests passed on Windows all.
So, to cut this short,
- I can reproduce this on Windows
- Workaround seems setting options such as {{spark.driver.extraClassPath}} and
{{spark.executor.extraClassPath}} (or equivalent options).
- The right paths should be URI forms as suggested above.
- I think we should investigate this.
was (Author: hyukjin.kwon):
Thank you for cc'ing me.
Yes, I think {{textFile}} seems irrelevant. Also, yes, I think the safe choice
is always to use a URI format on Windows where applicable.
I just manually tested and I can reproduce this on Windows although I see these
seem parsed fine as below with {{--jars}}.
{code}
C:\spark\bin>spark-shell.cmd --jars
C:\Users\IEUser\Downloads\spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
{code}
C:\spark\bin>spark-shell.cmd --jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
{code}
C:\spark\bin>spark-shell.cmd --jars
/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
file:/C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
In those cases, It seems I can't import the contents in the jar on Windows
whereas I can on my Mac if I haven't done something wrong.
Personally, in my case, I always use options related with path and classpaths
such as {{driver-library-path}} and {{spark.executor.extraLibraryPath}} in
{{spark-shell}} or {{spark-shell.cmd}} for sure.
or
{code}
spark.driver.extraClassPath
spark.driver.extraLibraryPath
spark.executor.extraClassPath
spark.executor.extraLibraryPath
{code}
in the configuration.
Anyway, I believe the case below {{C:/a/b/c}} is discouraged because I have
seen this is failed to be parsed properly as a URI ending up with recognising
C: as the scheme.
{code}
C:\spark\bin>spark-shell.cmd --jars
C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar --verbose
...
jars
C:/Users/IEUser/Downloads/spark-redshift_2.11-3.0.0-preview1.jar
...
{code}
Actually, I noticed such suspicious cases across Spark so I wanted to fix and
and fix them after making the tests passed on Windows all.
So, to cut this short,
- I can reproduce this on Windows
- Workaround seems setting options such as {{spark.driver.extraClassPath}} and
{{spark.executor.extraClassPath}} (or equivalent options).
- The right paths should be URI forms as sug
[jira] [Commented] (SPARK-18703) Insertion/CTAS against Hive Tables: Staging Directories and Data Files Not Dropped Until Normal Termination of JVM
[
https://issues.apache.org/jira/browse/SPARK-18703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15757476#comment-15757476
]
Apache Spark commented on SPARK-18703:
--
User 'gatorsmile' has created a pull request for this issue:
https://github.com/apache/spark/pull/16325
> Insertion/CTAS against Hive Tables: Staging Directories and Data Files Not
> Dropped Until Normal Termination of JVM
> --
>
> Key: SPARK-18703
> URL: https://issues.apache.org/jira/browse/SPARK-18703
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.0.2
>Reporter: Xiao Li
>Assignee: Xiao Li
>Priority: Critical
> Fix For: 2.2.0
>
>
> Below are the files/directories generated for three inserts againsts a Hive
> table:
> {noformat}
> /private/var/folders/4b/sgmfldk15js406vk7lw5llzwgn/T/spark-41eaa5ce-0288-471e-bba1-09cc482813ff/.hive-staging_hive_2016-12-03_20-56-29_149_4298858301766472202-1
> /private/var/folders/4b/sgmfldk15js406vk7lw5llzwgn/T/spark-41eaa5ce-0288-471e-bba1-09cc482813ff/.hive-staging_hive_2016-12-03_20-56-29_149_4298858301766472202-1/-ext-1
> /private/var/folders/4b/sgmfldk15js406vk7lw5llzwgn/T/spark-41eaa5ce-0288-471e-bba1-09cc482813ff/.hive-staging_hive_2016-12-03_20-56-29_149_4298858301766472202-1/-ext-1/._SUCCESS.crc
> /private/var/folders/4b/sgmfldk15js406vk7lw5llzwgn/T/spark-41eaa5ce-0288-471e-bba1-09cc482813ff/.hive-staging_hive_2016-12-03_20-56-29_149_4298858301766472202-1/-ext-1/.part-0.crc
> /private/var/folders/4b/sgmfldk15js406vk7lw5llzwgn/T/spark-41eaa5ce-0288-471e-bba1-09cc482813ff/.hive-staging_hive_2016-12-03_20-56-29_149_4298858301766472202-1/-ext-1/_SUCCESS
> /private/var/folders/4b/sgmfldk15js406vk7lw5llzwgn/T/spark-41eaa5ce-0288-471e-bba1-09cc482813ff/.hive-staging_hive_2016-12-03_20-56-29_149_4298858301766472202-1/-ext-1/part-0
> /private/var/folders/4b/sgmfldk15js406vk7lw5llzwgn/T/spark-41eaa5ce-0288-471e-bba1-09cc482813ff/.hive-staging_hive_2016-12-03_20-56-30_454_6445008511655931341-1
> /private/var/folders/4b/sgmfldk15js406vk7lw5llzwgn/T/spark-41eaa5ce-0288-471e-bba1-09cc482813ff/.hive-staging_hive_2016-12-03_20-56-30_454_6445008511655931341-1/-ext-1
> /private/var/folders/4b/sgmfldk15js406vk7lw5llzwgn/T/spark-41eaa5ce-0288-471e-bba1-09cc482813ff/.hive-staging_hive_2016-12-03_20-56-30_454_6445008511655931341-1/-ext-1/._SUCCESS.crc
> /private/var/folders/4b/sgmfldk15js406vk7lw5llzwgn/T/spark-41eaa5ce-0288-471e-bba1-09cc482813ff/.hive-staging_hive_2016-12-03_20-56-30_454_6445008511655931341-1/-ext-1/.part-0.crc
> /private/var/folders/4b/sgmfldk15js406vk7lw5llzwgn/T/spark-41eaa5ce-0288-471e-bba1-09cc482813ff/.hive-staging_hive_2016-12-03_20-56-30_454_6445008511655931341-1/-ext-1/_SUCCESS
> /private/var/folders/4b/sgmfldk15js406vk7lw5llzwgn/T/spark-41eaa5ce-0288-471e-bba1-09cc482813ff/.hive-staging_hive_2016-12-03_20-56-30_454_6445008511655931341-1/-ext-1/part-0
> /private/var/folders/4b/sgmfldk15js406vk7lw5llzwgn/T/spark-41eaa5ce-0288-471e-bba1-09cc482813ff/.hive-staging_hive_2016-12-03_20-56-30_722_3388423608658711001-1
> /private/var/folders/4b/sgmfldk15js406vk7lw5llzwgn/T/spark-41eaa5ce-0288-471e-bba1-09cc482813ff/.hive-staging_hive_2016-12-03_20-56-30_722_3388423608658711001-1/-ext-1
> /private/var/folders/4b/sgmfldk15js406vk7lw5llzwgn/T/spark-41eaa5ce-0288-471e-bba1-09cc482813ff/.hive-staging_hive_2016-12-03_20-56-30_722_3388423608658711001-1/-ext-1/._SUCCESS.crc
> /private/var/folders/4b/sgmfldk15js406vk7lw5llzwgn/T/spark-41eaa5ce-0288-471e-bba1-09cc482813ff/.hive-staging_hive_2016-12-03_20-56-30_722_3388423608658711001-1/-ext-1/.part-0.crc
> /private/var/folders/4b/sgmfldk15js406vk7lw5llzwgn/T/spark-41eaa5ce-0288-471e-bba1-09cc482813ff/.hive-staging_hive_2016-12-03_20-56-30_722_3388423608658711001-1/-ext-1/_SUCCESS
> /private/var/folders/4b/sgmfldk15js406vk7lw5llzwgn/T/spark-41eaa5ce-0288-471e-bba1-09cc482813ff/.hive-staging_hive_2016-12-03_20-56-30_722_3388423608658711001-1/-ext-1/part-0
> /private/var/folders/4b/sgmfldk15js406vk7lw5llzwgn/T/spark-41eaa5ce-0288-471e-bba1-09cc482813ff/.part-0.crc
> /private/var/folders/4b/sgmfldk15js406vk7lw5llzwgn/T/spark-41eaa5ce-0288-471e-bba1-09cc482813ff/part-0
> {noformat}
> The first 18 files are temporary. We do not drop it until the end of JVM
> termination. If JVM does not appropriately terminate, these temporary
> files/directories will not be dropped.
> Only the last two files are needed, as shown below.
> {noformat}
> /private/var/folders/4b/sgmfldk15js406vk7lw5llzwgn/T/spark-41eaa5ce-0288-471e-bba1-09cc482813ff/.part-0.crc
> /private/v
[jira] [Commented] (SPARK-18675) CTAS for hive serde table should work for all hive versions
[ https://issues.apache.org/jira/browse/SPARK-18675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15757477#comment-15757477 ] Apache Spark commented on SPARK-18675: -- User 'gatorsmile' has created a pull request for this issue: https://github.com/apache/spark/pull/16325 > CTAS for hive serde table should work for all hive versions > --- > > Key: SPARK-18675 > URL: https://issues.apache.org/jira/browse/SPARK-18675 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: Wenchen Fan >Assignee: Wenchen Fan > Fix For: 2.2.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18808) ml.KMeansModel.transform is very inefficient
[ https://issues.apache.org/jira/browse/SPARK-18808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15757839#comment-15757839 ] yuhao yang commented on SPARK-18808: [~FlamingMike] Are you interested in sending a fix? > ml.KMeansModel.transform is very inefficient > > > Key: SPARK-18808 > URL: https://issues.apache.org/jira/browse/SPARK-18808 > Project: Spark > Issue Type: Improvement > Components: ML >Affects Versions: 2.0.2 >Reporter: Michel Lemay > > The function ml.KMeansModel.transform will call the > parentModel.predict(features) method on each row which in turns will > normalize all clusterCenters from mllib.KMeansModel.clusterCentersWithNorm > every time! > This is a serious waste of resources! In my profiling, > clusterCentersWithNorm represent 99% of the sampling! > This should have been implemented with a broadcast variable as it is done in > other functions like computeCost. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18877) Unable to read given csv data. Excepion: java.lang.IllegalArgumentException: requirement failed: Decimal precision 28 exceeds max precision 20
[
https://issues.apache.org/jira/browse/SPARK-18877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15757924#comment-15757924
]
Dongjoon Hyun commented on SPARK-18877:
---
Hi, [~Navya Krishnappa].
As you see in the stack trace, that is a different exception from Apache
Parquet code.
{code}
Caused by: java.lang.IllegalArgumentException: Invalid DECIMAL scale: -9
at org.apache.parquet.Preconditions.checkArgument(Preconditions.java:55)
{code}
You can register another issue for that, but maybe in Apache Parquet JIRA.
> Unable to read given csv data. Excepion: java.lang.IllegalArgumentException:
> requirement failed: Decimal precision 28 exceeds max precision 20
> --
>
> Key: SPARK-18877
> URL: https://issues.apache.org/jira/browse/SPARK-18877
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.0.2
>Reporter: Navya Krishnappa
>
> When reading below mentioned csv data, even though the maximum decimal
> precision is 38, following exception is thrown
> java.lang.IllegalArgumentException: requirement failed: Decimal precision 28
> exceeds max precision 20
> Decimal
> 2323366225312000
> 2433573971400
> 23233662253000
> 23233662253
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18877) Unable to read given csv data. Excepion: java.lang.IllegalArgumentException: requirement failed: Decimal precision 28 exceeds max precision 20
[ https://issues.apache.org/jira/browse/SPARK-18877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15757940#comment-15757940 ] Dongjoon Hyun commented on SPARK-18877: --- For a Spark usage workaround, I think you can change the schema into a Parquet-acceptable one manually. > Unable to read given csv data. Excepion: java.lang.IllegalArgumentException: > requirement failed: Decimal precision 28 exceeds max precision 20 > -- > > Key: SPARK-18877 > URL: https://issues.apache.org/jira/browse/SPARK-18877 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.2 >Reporter: Navya Krishnappa > > When reading below mentioned csv data, even though the maximum decimal > precision is 38, following exception is thrown > java.lang.IllegalArgumentException: requirement failed: Decimal precision 28 > exceeds max precision 20 > Decimal > 2323366225312000 > 2433573971400 > 23233662253000 > 23233662253 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-18877) Unable to read given csv data. Excepion: java.lang.IllegalArgumentException: requirement failed: Decimal precision 28 exceeds max precision 20
[
https://issues.apache.org/jira/browse/SPARK-18877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15757924#comment-15757924
]
Dongjoon Hyun edited comment on SPARK-18877 at 12/18/16 1:11 AM:
-
Hi, [~Navya Krishnappa].
As you see in the stack trace, that is a different exception from Apache
Parquet code.
{code}
Caused by: java.lang.IllegalArgumentException: Invalid DECIMAL scale: -9
at org.apache.parquet.Preconditions.checkArgument(Preconditions.java:55)
{code}
And, this is the Parquet code
[here|https://github.com/Parquet/parquet-mr/blob/master/parquet-column/src/main/java/parquet/schema/Types.java#L405-L417].
It apparently does not fully support BigDecimal.
{code}
protected DecimalMetadata decimalMetadata() {
DecimalMetadata meta = null;
if (OriginalType.DECIMAL == originalType) {
Preconditions.checkArgument(precision > 0,
"Invalid DECIMAL precision: " + precision);
Preconditions.checkArgument(scale >= 0,
"Invalid DECIMAL scale: " + scale);
Preconditions.checkArgument(scale <= precision,
"Invalid DECIMAL scale: cannot be greater than precision");
meta = new DecimalMetadata(precision, scale);
}
return meta;
}
}
{code}
I cannot make a PR for Parquet.
I hope you register another issue for that in Apache Parquet JIRA.
was (Author: dongjoon):
Hi, [~Navya Krishnappa].
As you see in the stack trace, that is a different exception from Apache
Parquet code.
{code}
Caused by: java.lang.IllegalArgumentException: Invalid DECIMAL scale: -9
at org.apache.parquet.Preconditions.checkArgument(Preconditions.java:55)
{code}
You can register another issue for that, but maybe in Apache Parquet JIRA.
> Unable to read given csv data. Excepion: java.lang.IllegalArgumentException:
> requirement failed: Decimal precision 28 exceeds max precision 20
> --
>
> Key: SPARK-18877
> URL: https://issues.apache.org/jira/browse/SPARK-18877
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.0.2
>Reporter: Navya Krishnappa
>
> When reading below mentioned csv data, even though the maximum decimal
> precision is 38, following exception is thrown
> java.lang.IllegalArgumentException: requirement failed: Decimal precision 28
> exceeds max precision 20
> Decimal
> 2323366225312000
> 2433573971400
> 23233662253000
> 23233662253
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15757998#comment-15757998
]
Felix Cheung commented on SPARK-18817:
--
Aside from changing the existing shipped behavior, there are a few mentions of
this behavior in various documentation that would become wrong and would need
to be updated.
IMO more importantly we still have a feature that can be turned on (as
documented or suggested in documentations) that would cause files to be written
without the user explicitly agreeing to it (or understanding it). This to me
doesn't seem like we would be addressing the root of the issue fully, merely
side-stepping it?
I've managed to track down the fix to move metastore_db and derby.log though.
There are two separate switches to set that it doable from pure R; but I'd
recommend doing in
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala#L116
in order to respect any existing value from hive-site.xml if given one.
How about we introduce something like spark.sql.default.derby.dir and fix this
that way?
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15757998#comment-15757998
]
Felix Cheung edited comment on SPARK-18817 at 12/18/16 2:04 AM:
Aside from changing the existing shipped behavior, there are a few mentions of
this behavior in various documentation that would become wrong and would need
to be updated.
IMO more importantly we still have a feature that can be turned on (as
documented or suggested in documentations) that would cause files to be written
without the user explicitly agreeing to it (or understanding it). This to me
doesn't seem like we would be addressing the root of the issue fully, merely
side-stepping it?
I've managed to track down the fix to move metastore_db and derby.log though.
There are two separate switches to set and it is doable from pure R (have
tested that); but I'd recommend doing in
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala#L116
in order to respect any existing value from hive-site.xml if given one.
How about we introduce something like spark.sql.default.derby.dir and fix this
that way?
was (Author: felixcheung):
Aside from changing the existing shipped behavior, there are a few mentions of
this behavior in various documentation that would become wrong and would need
to be updated.
IMO more importantly we still have a feature that can be turned on (as
documented or suggested in documentations) that would cause files to be written
without the user explicitly agreeing to it (or understanding it). This to me
doesn't seem like we would be addressing the root of the issue fully, merely
side-stepping it?
I've managed to track down the fix to move metastore_db and derby.log though.
There are two separate switches to set that it doable from pure R; but I'd
recommend doing in
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala#L116
in order to respect any existing value from hive-site.xml if given one.
How about we introduce something like spark.sql.default.derby.dir and fix this
that way?
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15758005#comment-15758005
]
Felix Cheung commented on SPARK-18817:
--
And as a side note, I feel like spark-warehouse, metastore_db, and derby.log
should all be in temporary directory that is cleaned out when the app is done,
much like how Spark Thrift Server is doing currently (at least to metastore_db,
derby.log). Perhaps that is the more correct fix longer term.
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-18915) Return Nothing when Querying a Partitioned Data Source Table without Repairing it
Xiao Li created SPARK-18915:
---
Summary: Return Nothing when Querying a Partitioned Data Source
Table without Repairing it
Key: SPARK-18915
URL: https://issues.apache.org/jira/browse/SPARK-18915
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.1.0
Reporter: Xiao Li
Priority: Critical
In Spark 2.1, if we create a parititoned data source table given a specified
path, it returns nothing when we try to query it. To get the data, we have to
manually issue a DDL to repair the table.
In Spark 2.0, it can return the data stored in the specified path, without
repairing the table.
Below is the output of Spark 2.1.
{noformat}
scala> spark.range(5).selectExpr("id as fieldOne", "id as
partCol").write.partitionBy("partCol").mode("overwrite").saveAsTable("test")
[Stage 0:==>(3 + 5) /
8]SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
scala> spark.sql("select * from test").show()
++---+
|fieldOne|partCol|
++---+
| 0| 0|
| 1| 1|
| 2| 2|
| 3| 3|
| 4| 4|
++---+
scala> spark.sql("desc formatted test").show(50, false)
++--+---+
|col_name|data_type
|comment|
++--+---+
|fieldOne|bigint
|null |
|partCol |bigint
|null |
|# Partition Information |
| |
|# col_name |data_type
|comment|
|partCol |bigint
|null |
||
| |
|# Detailed Table Information|
| |
|Database: |default
| |
|Owner: |xiaoli
| |
|Create Time:|Sat Dec 17 17:46:24 PST 2016
| |
|Last Access Time: |Wed Dec 31 16:00:00 PST 1969
| |
|Location:
|file:/Users/xiaoli/IdeaProjects/sparkDelivery/bin/spark-warehouse/test| |
|Table Type: |MANAGED
| |
|Table Parameters: |
| |
| transient_lastDdlTime |1482025584
| |
||
| |
|# Storage Information |
| |
|SerDe Library:
|org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe | |
|InputFormat:
|org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat | |
|OutputFormat:
|org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat| |
|Compressed: |No
| |
|Storage Desc Parameters:|
| |
| serialization.format |1
| |
|Partition Provider: |Catalog
| |
++--+---+
scala> spark.sql(s"create table newTab (fieldOne long, partCol int) using
parquet options (path
'file:/Users/xiaoli/IdeaProjects/sparkDelivery/bin/spark-warehouse/test')
partitioned by (partCol)")
res3: org.apache.spark.sql.DataFrame = []
scala> spark.table("newTab").show()
++---+
|fieldOne|partCol|
+
[jira] [Created] (SPARK-18916) Bug in Pregel / mergeMsg with hashmaps
Seth Bromberger created SPARK-18916:
---
Summary: Bug in Pregel / mergeMsg with hashmaps
Key: SPARK-18916
URL: https://issues.apache.org/jira/browse/SPARK-18916
Project: Spark
Issue Type: Bug
Components: GraphX
Affects Versions: 2.0.2
Environment: OSX / IntelliJ IDEA 2016.3 CE EAP, Scala 2.11.8, Spark
2.0.2
Reporter: Seth Bromberger
Consider the following (rough) code that attempts to calculate all-pairs
shortest paths via pregel:
{code:scala}
def allPairsShortestPaths: RDD[(VertexId, HashMap[VertexId, ParentDist])] =
{
val initialMsg = HashMap(-1L -> ParentDist(-1L, -1L))
val pregelg = g.mapVertices((vid, vd) => (vd, HashMap[VertexId,
ParentDist](vid -> ParentDist(vid, 0L.reverse
def vprog(v: VertexId, value: (VD, HashMap[VertexId, ParentDist]),
message: HashMap[VertexId, ParentDist]): (VD, HashMap[VertexId, ParentDist]) = {
val updatedValues = mm2(value._2, message).filter(v => v._2.dist >= 0)
(value._1, updatedValues)
}
def sendMsg(triplet: EdgeTriplet[(VD, HashMap[VertexId, ParentDist]),
ED]): Iterator[(VertexId, HashMap[VertexId, ParentDist])] = {
val dstVertexId = triplet.dstId
val srcMap = triplet.srcAttr._2
val dstMap = triplet.dstAttr._2 // guaranteed to have dstVertexId as a
key
val updatesToSend : HashMap[VertexId, ParentDist] = srcMap.filter {
case (vid, srcPD) => dstMap.get(vid) match {
case Some(dstPD) => dstPD.dist > srcPD.dist + 1 // if it exists,
is it cheaper?
case _ => true // not found - new update
}
}.map(u => u._1 -> ParentDist(triplet.srcId, u._2.dist +1))
if (updatesToSend.nonEmpty)
Iterator[(VertexId, HashMap[VertexId, ParentDist])]((dstVertexId,
updatesToSend))
else
Iterator.empty
}
def mergeMsg(m1: HashMap[VertexId, ParentDist], m2: HashMap[VertexId,
ParentDist]): HashMap[VertexId, ParentDist] = {
// when the following two lines are commented out, the program fails
with
// 16/12/17 19:53:50 INFO DAGScheduler: Job 24 failed: reduce at
VertexRDDImpl.scala:88, took 0.244042 s
// Exception in thread "main" org.apache.spark.SparkException: Job
aborted due to stage failure: Task 0 in stage 1099.0 failed 1 times, most
recent failure: Lost task 0.0 in stage 1099.0 (TID 129, localhost):
scala.MatchError: (null,null) (of class scala.Tuple2)
m1.foreach(_ => ())
m2.foreach(_ => ())
m1.merged(m2) {
case ((k1, v1), (_, v2)) => (k1, v1.min(v2))
}
}
// mm2 is here just to provide a separate function for vprog. Ideally
we'd just re-use mergeMsg.
def mm2(m1: HashMap[VertexId, ParentDist], m2: HashMap[VertexId,
ParentDist]): HashMap[VertexId, ParentDist] = {
m1.merged(m2) {
case ((k1, v1), (_, v2)) => (k1, v1.min(v2))
case n => throw new Exception("we've got a problem: " + n)
}
}
val pregelRun = pregelg.pregel(initialMsg)(vprog, sendMsg, mergeMsg)
val sps = pregelRun.vertices.map(v => v._1 -> v._2._2)
sps
}
}
{code}
Note the comment in the mergeMsg function: when the messages are explicitly
accessed prior to the .merged statement, the code works. If these side-effect
statements are removed / commented out, the error message in the comments is
generated.
This fails consistently on a 50-node undirected cyclegraph.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-18916) Bug in Pregel / mergeMsg with hashmaps
[
https://issues.apache.org/jira/browse/SPARK-18916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Seth Bromberger updated SPARK-18916:
Description:
Consider the following (rough) code that attempts to calculate all-pairs
shortest paths via pregel:
{code:java}
def allPairsShortestPaths: RDD[(VertexId, HashMap[VertexId, ParentDist])] =
{
val initialMsg = HashMap(-1L -> ParentDist(-1L, -1L))
val pregelg = g.mapVertices((vid, vd) => (vd, HashMap[VertexId,
ParentDist](vid -> ParentDist(vid, 0L.reverse
def vprog(v: VertexId, value: (VD, HashMap[VertexId, ParentDist]),
message: HashMap[VertexId, ParentDist]): (VD, HashMap[VertexId, ParentDist]) = {
val updatedValues = mm2(value._2, message).filter(v => v._2.dist >= 0)
(value._1, updatedValues)
}
def sendMsg(triplet: EdgeTriplet[(VD, HashMap[VertexId, ParentDist]),
ED]): Iterator[(VertexId, HashMap[VertexId, ParentDist])] = {
val dstVertexId = triplet.dstId
val srcMap = triplet.srcAttr._2
val dstMap = triplet.dstAttr._2 // guaranteed to have dstVertexId as a
key
val updatesToSend : HashMap[VertexId, ParentDist] = srcMap.filter {
case (vid, srcPD) => dstMap.get(vid) match {
case Some(dstPD) => dstPD.dist > srcPD.dist + 1 // if it exists,
is it cheaper?
case _ => true // not found - new update
}
}.map(u => u._1 -> ParentDist(triplet.srcId, u._2.dist +1))
if (updatesToSend.nonEmpty)
Iterator[(VertexId, HashMap[VertexId, ParentDist])]((dstVertexId,
updatesToSend))
else
Iterator.empty
}
def mergeMsg(m1: HashMap[VertexId, ParentDist], m2: HashMap[VertexId,
ParentDist]): HashMap[VertexId, ParentDist] = {
// when the following two lines are commented out, the program fails
with
// 16/12/17 19:53:50 INFO DAGScheduler: Job 24 failed: reduce at
VertexRDDImpl.scala:88, took 0.244042 s
// Exception in thread "main" org.apache.spark.SparkException: Job
aborted due to stage failure: Task 0 in stage 1099.0 failed 1 times, most
recent failure: Lost task 0.0 in stage 1099.0 (TID 129, localhost):
scala.MatchError: (null,null) (of class scala.Tuple2)
m1.foreach(_ => ())
m2.foreach(_ => ())
m1.merged(m2) {
case ((k1, v1), (_, v2)) => (k1, v1.min(v2))
}
}
// mm2 is here just to provide a separate function for vprog. Ideally
we'd just re-use mergeMsg.
def mm2(m1: HashMap[VertexId, ParentDist], m2: HashMap[VertexId,
ParentDist]): HashMap[VertexId, ParentDist] = {
m1.merged(m2) {
case ((k1, v1), (_, v2)) => (k1, v1.min(v2))
case n => throw new Exception("we've got a problem: " + n)
}
}
val pregelRun = pregelg.pregel(initialMsg)(vprog, sendMsg, mergeMsg)
val sps = pregelRun.vertices.map(v => v._1 -> v._2._2)
sps
}
}
{code}
Note the comment in the mergeMsg function: when the messages are explicitly
accessed prior to the .merged statement, the code works. If these side-effect
statements are removed / commented out, the error message in the comments is
generated.
This fails consistently on a 50-node undirected cyclegraph.
was:
Consider the following (rough) code that attempts to calculate all-pairs
shortest paths via pregel:
{code:scala}
def allPairsShortestPaths: RDD[(VertexId, HashMap[VertexId, ParentDist])] =
{
val initialMsg = HashMap(-1L -> ParentDist(-1L, -1L))
val pregelg = g.mapVertices((vid, vd) => (vd, HashMap[VertexId,
ParentDist](vid -> ParentDist(vid, 0L.reverse
def vprog(v: VertexId, value: (VD, HashMap[VertexId, ParentDist]),
message: HashMap[VertexId, ParentDist]): (VD, HashMap[VertexId, ParentDist]) = {
val updatedValues = mm2(value._2, message).filter(v => v._2.dist >= 0)
(value._1, updatedValues)
}
def sendMsg(triplet: EdgeTriplet[(VD, HashMap[VertexId, ParentDist]),
ED]): Iterator[(VertexId, HashMap[VertexId, ParentDist])] = {
val dstVertexId = triplet.dstId
val srcMap = triplet.srcAttr._2
val dstMap = triplet.dstAttr._2 // guaranteed to have dstVertexId as a
key
val updatesToSend : HashMap[VertexId, ParentDist] = srcMap.filter {
case (vid, srcPD) => dstMap.get(vid) match {
case Some(dstPD) => dstPD.dist > srcPD.dist + 1 // if it exists,
is it cheaper?
case _ => true // not found - new update
}
}.map(u => u._1 -> ParentDist(triplet.srcId, u._2.dist +1))
if (updatesToSend.nonEmpty)
Iterator[(VertexId, HashMap[VertexId, ParentDist])]((dstVertexId,
updatesToSend))
else
Iterator.empty
}
def mergeMsg(m1: HashMap[VertexId, ParentDist], m2: HashMap[VertexId,
ParentDist]): HashMap[VertexId, ParentDist] = {
// when the following two
[jira] [Updated] (SPARK-18916) Possible bug in Pregel / mergeMsg with hashmaps
[
https://issues.apache.org/jira/browse/SPARK-18916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Seth Bromberger updated SPARK-18916:
Summary: Possible bug in Pregel / mergeMsg with hashmaps (was: Bug in
Pregel / mergeMsg with hashmaps)
> Possible bug in Pregel / mergeMsg with hashmaps
> ---
>
> Key: SPARK-18916
> URL: https://issues.apache.org/jira/browse/SPARK-18916
> Project: Spark
> Issue Type: Bug
> Components: GraphX
>Affects Versions: 2.0.2
> Environment: OSX / IntelliJ IDEA 2016.3 CE EAP, Scala 2.11.8, Spark
> 2.0.2
>Reporter: Seth Bromberger
> Labels: error, graphx, pregel
>
> Consider the following (rough) code that attempts to calculate all-pairs
> shortest paths via pregel:
> {code:java}
> def allPairsShortestPaths: RDD[(VertexId, HashMap[VertexId, ParentDist])]
> = {
> val initialMsg = HashMap(-1L -> ParentDist(-1L, -1L))
> val pregelg = g.mapVertices((vid, vd) => (vd, HashMap[VertexId,
> ParentDist](vid -> ParentDist(vid, 0L.reverse
> def vprog(v: VertexId, value: (VD, HashMap[VertexId, ParentDist]),
> message: HashMap[VertexId, ParentDist]): (VD, HashMap[VertexId, ParentDist])
> = {
> val updatedValues = mm2(value._2, message).filter(v => v._2.dist >= 0)
> (value._1, updatedValues)
> }
> def sendMsg(triplet: EdgeTriplet[(VD, HashMap[VertexId, ParentDist]),
> ED]): Iterator[(VertexId, HashMap[VertexId, ParentDist])] = {
> val dstVertexId = triplet.dstId
> val srcMap = triplet.srcAttr._2
> val dstMap = triplet.dstAttr._2 // guaranteed to have dstVertexId as
> a key
> val updatesToSend : HashMap[VertexId, ParentDist] = srcMap.filter {
> case (vid, srcPD) => dstMap.get(vid) match {
> case Some(dstPD) => dstPD.dist > srcPD.dist + 1 // if it
> exists, is it cheaper?
> case _ => true // not found - new update
> }
> }.map(u => u._1 -> ParentDist(triplet.srcId, u._2.dist +1))
> if (updatesToSend.nonEmpty)
> Iterator[(VertexId, HashMap[VertexId, ParentDist])]((dstVertexId,
> updatesToSend))
> else
> Iterator.empty
> }
> def mergeMsg(m1: HashMap[VertexId, ParentDist], m2: HashMap[VertexId,
> ParentDist]): HashMap[VertexId, ParentDist] = {
> // when the following two lines are commented out, the program fails
> with
> // 16/12/17 19:53:50 INFO DAGScheduler: Job 24 failed: reduce at
> VertexRDDImpl.scala:88, took 0.244042 s
> // Exception in thread "main" org.apache.spark.SparkException: Job
> aborted due to stage failure: Task 0 in stage 1099.0 failed 1 times, most
> recent failure: Lost task 0.0 in stage 1099.0 (TID 129, localhost):
> scala.MatchError: (null,null) (of class scala.Tuple2)
> m1.foreach(_ => ())
> m2.foreach(_ => ())
> m1.merged(m2) {
> case ((k1, v1), (_, v2)) => (k1, v1.min(v2))
> }
> }
> // mm2 is here just to provide a separate function for vprog. Ideally
> we'd just re-use mergeMsg.
> def mm2(m1: HashMap[VertexId, ParentDist], m2: HashMap[VertexId,
> ParentDist]): HashMap[VertexId, ParentDist] = {
> m1.merged(m2) {
> case ((k1, v1), (_, v2)) => (k1, v1.min(v2))
> case n => throw new Exception("we've got a problem: " + n)
> }
> }
> val pregelRun = pregelg.pregel(initialMsg)(vprog, sendMsg, mergeMsg)
> val sps = pregelRun.vertices.map(v => v._1 -> v._2._2)
> sps
> }
> }
> {code}
> Note the comment in the mergeMsg function: when the messages are explicitly
> accessed prior to the .merged statement, the code works. If these side-effect
> statements are removed / commented out, the error message in the comments is
> generated.
> This fails consistently on a 50-node undirected cyclegraph.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-18915) Return Nothing when Querying a Partitioned Data Source Table without Repairing it
[
https://issues.apache.org/jira/browse/SPARK-18915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-18915:
Assignee: Apache Spark
> Return Nothing when Querying a Partitioned Data Source Table without
> Repairing it
> -
>
> Key: SPARK-18915
> URL: https://issues.apache.org/jira/browse/SPARK-18915
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.1.0
>Reporter: Xiao Li
>Assignee: Apache Spark
>Priority: Critical
>
> In Spark 2.1, if we create a parititoned data source table given a specified
> path, it returns nothing when we try to query it. To get the data, we have to
> manually issue a DDL to repair the table.
> In Spark 2.0, it can return the data stored in the specified path, without
> repairing the table.
> Below is the output of Spark 2.1.
> {noformat}
> scala> spark.range(5).selectExpr("id as fieldOne", "id as
> partCol").write.partitionBy("partCol").mode("overwrite").saveAsTable("test")
> [Stage 0:==>(3 + 5) /
> 8]SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
>
>
> scala> spark.sql("select * from test").show()
> ++---+
> |fieldOne|partCol|
> ++---+
> | 0| 0|
> | 1| 1|
> | 2| 2|
> | 3| 3|
> | 4| 4|
> ++---+
> scala> spark.sql("desc formatted test").show(50, false)
> ++--+---+
> |col_name|data_type
> |comment|
> ++--+---+
> |fieldOne|bigint
> |null |
> |partCol |bigint
> |null |
> |# Partition Information |
> | |
> |# col_name |data_type
> |comment|
> |partCol |bigint
> |null |
> ||
> | |
> |# Detailed Table Information|
> | |
> |Database: |default
> | |
> |Owner: |xiaoli
> | |
> |Create Time:|Sat Dec 17 17:46:24 PST 2016
> | |
> |Last Access Time: |Wed Dec 31 16:00:00 PST 1969
> | |
> |Location:
> |file:/Users/xiaoli/IdeaProjects/sparkDelivery/bin/spark-warehouse/test|
> |
> |Table Type: |MANAGED
> | |
> |Table Parameters: |
> | |
> | transient_lastDdlTime |1482025584
> | |
> ||
> | |
> |# Storage Information |
> | |
> |SerDe Library:
> |org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe |
> |
> |InputFormat:
> |org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat |
> |
> |OutputFormat:
> |org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat|
> |
> |Compressed: |No
> | |
> |Storage Desc Parameters:|
> | |
> | serialization.format |1
> | |
> |Partition Provider: |Catalog
>
[jira] [Commented] (SPARK-18915) Return Nothing when Querying a Partitioned Data Source Table without Repairing it
[
https://issues.apache.org/jira/browse/SPARK-18915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15758144#comment-15758144
]
Apache Spark commented on SPARK-18915:
--
User 'gatorsmile' has created a pull request for this issue:
https://github.com/apache/spark/pull/16326
> Return Nothing when Querying a Partitioned Data Source Table without
> Repairing it
> -
>
> Key: SPARK-18915
> URL: https://issues.apache.org/jira/browse/SPARK-18915
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.1.0
>Reporter: Xiao Li
>Priority: Critical
>
> In Spark 2.1, if we create a parititoned data source table given a specified
> path, it returns nothing when we try to query it. To get the data, we have to
> manually issue a DDL to repair the table.
> In Spark 2.0, it can return the data stored in the specified path, without
> repairing the table.
> Below is the output of Spark 2.1.
> {noformat}
> scala> spark.range(5).selectExpr("id as fieldOne", "id as
> partCol").write.partitionBy("partCol").mode("overwrite").saveAsTable("test")
> [Stage 0:==>(3 + 5) /
> 8]SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
>
>
> scala> spark.sql("select * from test").show()
> ++---+
> |fieldOne|partCol|
> ++---+
> | 0| 0|
> | 1| 1|
> | 2| 2|
> | 3| 3|
> | 4| 4|
> ++---+
> scala> spark.sql("desc formatted test").show(50, false)
> ++--+---+
> |col_name|data_type
> |comment|
> ++--+---+
> |fieldOne|bigint
> |null |
> |partCol |bigint
> |null |
> |# Partition Information |
> | |
> |# col_name |data_type
> |comment|
> |partCol |bigint
> |null |
> ||
> | |
> |# Detailed Table Information|
> | |
> |Database: |default
> | |
> |Owner: |xiaoli
> | |
> |Create Time:|Sat Dec 17 17:46:24 PST 2016
> | |
> |Last Access Time: |Wed Dec 31 16:00:00 PST 1969
> | |
> |Location:
> |file:/Users/xiaoli/IdeaProjects/sparkDelivery/bin/spark-warehouse/test|
> |
> |Table Type: |MANAGED
> | |
> |Table Parameters: |
> | |
> | transient_lastDdlTime |1482025584
> | |
> ||
> | |
> |# Storage Information |
> | |
> |SerDe Library:
> |org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe |
> |
> |InputFormat:
> |org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat |
> |
> |OutputFormat:
> |org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat|
> |
> |Compressed: |No
> | |
> |Storage Desc Parameters:|
> | |
> | serialization.format |1
> | |
> |Pa
[jira] [Assigned] (SPARK-18915) Return Nothing when Querying a Partitioned Data Source Table without Repairing it
[
https://issues.apache.org/jira/browse/SPARK-18915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-18915:
Assignee: (was: Apache Spark)
> Return Nothing when Querying a Partitioned Data Source Table without
> Repairing it
> -
>
> Key: SPARK-18915
> URL: https://issues.apache.org/jira/browse/SPARK-18915
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.1.0
>Reporter: Xiao Li
>Priority: Critical
>
> In Spark 2.1, if we create a parititoned data source table given a specified
> path, it returns nothing when we try to query it. To get the data, we have to
> manually issue a DDL to repair the table.
> In Spark 2.0, it can return the data stored in the specified path, without
> repairing the table.
> Below is the output of Spark 2.1.
> {noformat}
> scala> spark.range(5).selectExpr("id as fieldOne", "id as
> partCol").write.partitionBy("partCol").mode("overwrite").saveAsTable("test")
> [Stage 0:==>(3 + 5) /
> 8]SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
>
>
> scala> spark.sql("select * from test").show()
> ++---+
> |fieldOne|partCol|
> ++---+
> | 0| 0|
> | 1| 1|
> | 2| 2|
> | 3| 3|
> | 4| 4|
> ++---+
> scala> spark.sql("desc formatted test").show(50, false)
> ++--+---+
> |col_name|data_type
> |comment|
> ++--+---+
> |fieldOne|bigint
> |null |
> |partCol |bigint
> |null |
> |# Partition Information |
> | |
> |# col_name |data_type
> |comment|
> |partCol |bigint
> |null |
> ||
> | |
> |# Detailed Table Information|
> | |
> |Database: |default
> | |
> |Owner: |xiaoli
> | |
> |Create Time:|Sat Dec 17 17:46:24 PST 2016
> | |
> |Last Access Time: |Wed Dec 31 16:00:00 PST 1969
> | |
> |Location:
> |file:/Users/xiaoli/IdeaProjects/sparkDelivery/bin/spark-warehouse/test|
> |
> |Table Type: |MANAGED
> | |
> |Table Parameters: |
> | |
> | transient_lastDdlTime |1482025584
> | |
> ||
> | |
> |# Storage Information |
> | |
> |SerDe Library:
> |org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe |
> |
> |InputFormat:
> |org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat |
> |
> |OutputFormat:
> |org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat|
> |
> |Compressed: |No
> | |
> |Storage Desc Parameters:|
> | |
> | serialization.format |1
> | |
> |Partition Provider: |Catalog
> | |
> +
[jira] [Created] (SPARK-18917) Dataframe - Time Out Issues / Taking long time in append mode on object stores
Anbu Cheeralan created SPARK-18917:
--
Summary: Dataframe - Time Out Issues / Taking long time in append
mode on object stores
Key: SPARK-18917
URL: https://issues.apache.org/jira/browse/SPARK-18917
Project: Spark
Issue Type: Improvement
Components: EC2, SQL, YARN
Affects Versions: 2.0.2
Reporter: Anbu Cheeralan
Priority: Minor
Fix For: 2.1.0, 2.1.1
When using Dataframe write in append mode on object stores (S3 / Google
Storage), the writes are taking long time to write/ getting read time out. This
is because dataframe.write lists all leaf folders in the target directory. If
there are lot of subfolders due to partitions, this is taking for ever.
The code is In org.apache.spark.sql.execution.datasources.DataSource.write()
following code causes huge number of RPC calls when the file system is an
Object Store (S3, GS).
if (mode == SaveMode.Append) {
val existingPartitionColumns = Try {
resolveRelation()
.asInstanceOf[HadoopFsRelation]
.location
.partitionSpec()
.partitionColumns
.fieldNames
.toSeq
}.getOrElse(Seq.empty[String])
There should be a flag to skip Partition Match Check in append mode. I can work
on the patch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
