[jira] [Created] (SPARK-21223) Thread-safety issue in FsHistoryProvider
zenglinxi created SPARK-21223: - Summary: Thread-safety issue in FsHistoryProvider Key: SPARK-21223 URL: https://issues.apache.org/jira/browse/SPARK-21223 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 2.1.1 Reporter: zenglinxi Currently, Spark HistoryServer use a HashMap named fileToAppInfo in class FsHistoryProvider to store the map of eventlog path and attemptInfo. When use ThreadPool to Replay the log files in the list and merge the list of old applications with new ones, multi thread may update fileToAppInfo at the same time, which may cause Thread-safety issues. {code:java} for (file <- logInfos) { tasks += replayExecutor.submit(new Runnable { override def run(): Unit = mergeApplicationListing(file) }) } {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-21223) Thread-safety issue in FsHistoryProvider
[ https://issues.apache.org/jira/browse/SPARK-21223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21223: Assignee: Apache Spark > Thread-safety issue in FsHistoryProvider > - > > Key: SPARK-21223 > URL: https://issues.apache.org/jira/browse/SPARK-21223 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.1.1 >Reporter: zenglinxi >Assignee: Apache Spark > > Currently, Spark HistoryServer use a HashMap named fileToAppInfo in class > FsHistoryProvider to store the map of eventlog path and attemptInfo. > When use ThreadPool to Replay the log files in the list and merge the list of > old applications with new ones, multi thread may update fileToAppInfo at the > same time, which may cause Thread-safety issues. > {code:java} > for (file <- logInfos) { >tasks += replayExecutor.submit(new Runnable { > override def run(): Unit = mergeApplicationListing(file) > }) > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-21223) Thread-safety issue in FsHistoryProvider
[ https://issues.apache.org/jira/browse/SPARK-21223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21223: Assignee: (was: Apache Spark) > Thread-safety issue in FsHistoryProvider > - > > Key: SPARK-21223 > URL: https://issues.apache.org/jira/browse/SPARK-21223 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.1.1 >Reporter: zenglinxi > > Currently, Spark HistoryServer use a HashMap named fileToAppInfo in class > FsHistoryProvider to store the map of eventlog path and attemptInfo. > When use ThreadPool to Replay the log files in the list and merge the list of > old applications with new ones, multi thread may update fileToAppInfo at the > same time, which may cause Thread-safety issues. > {code:java} > for (file <- logInfos) { >tasks += replayExecutor.submit(new Runnable { > override def run(): Unit = mergeApplicationListing(file) > }) > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21223) Thread-safety issue in FsHistoryProvider
[ https://issues.apache.org/jira/browse/SPARK-21223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064406#comment-16064406 ] Apache Spark commented on SPARK-21223: -- User 'zenglinxi0615' has created a pull request for this issue: https://github.com/apache/spark/pull/18430 > Thread-safety issue in FsHistoryProvider > - > > Key: SPARK-21223 > URL: https://issues.apache.org/jira/browse/SPARK-21223 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.1.1 >Reporter: zenglinxi > > Currently, Spark HistoryServer use a HashMap named fileToAppInfo in class > FsHistoryProvider to store the map of eventlog path and attemptInfo. > When use ThreadPool to Replay the log files in the list and merge the list of > old applications with new ones, multi thread may update fileToAppInfo at the > same time, which may cause Thread-safety issues. > {code:java} > for (file <- logInfos) { >tasks += replayExecutor.submit(new Runnable { > override def run(): Unit = mergeApplicationListing(file) > }) > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-21224) Support a DDL-formatted string in DataFrameReader.schema in R
Hyukjin Kwon created SPARK-21224: Summary: Support a DDL-formatted string in DataFrameReader.schema in R Key: SPARK-21224 URL: https://issues.apache.org/jira/browse/SPARK-21224 Project: Spark Issue Type: Improvement Components: SparkR Affects Versions: 2.2.0 Reporter: Hyukjin Kwon Priority: Minor This might have to be a followup for SPARK-20431 but I just decided to make this separate for R specifically as it might be confused. Please refer the discussion in the PR and SPARK-20431. In a simple view, this JIRA describes the support for a DDL-formetted string as schema as below: {code} mockLines <- c("{\"name\":\"Michael\"}", "{\"name\":\"Andy\", \"age\":30}", "{\"name\":\"Justin\", \"age\":19}") jsonPath <- tempfile(pattern = "sparkr-test", fileext = ".tmp") writeLines(mockLines, jsonPath) df <- read.df(jsonPath, "json", "name STRING, age DOUBLE") collect(df) {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-21224) Support a DDL-formatted string as schema in reading for R
[ https://issues.apache.org/jira/browse/SPARK-21224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-21224: - Summary: Support a DDL-formatted string as schema in reading for R (was: Support a DDL-formatted string in DataFrameReader.schema in R) > Support a DDL-formatted string as schema in reading for R > - > > Key: SPARK-21224 > URL: https://issues.apache.org/jira/browse/SPARK-21224 > Project: Spark > Issue Type: Improvement > Components: SparkR >Affects Versions: 2.2.0 >Reporter: Hyukjin Kwon >Priority: Minor > > This might have to be a followup for SPARK-20431 but I just decided to make > this separate for R specifically as it might be confused. > Please refer the discussion in the PR and SPARK-20431. > In a simple view, this JIRA describes the support for a DDL-formetted string > as schema as below: > {code} > mockLines <- c("{\"name\":\"Michael\"}", >"{\"name\":\"Andy\", \"age\":30}", >"{\"name\":\"Justin\", \"age\":19}") > jsonPath <- tempfile(pattern = "sparkr-test", fileext = ".tmp") > writeLines(mockLines, jsonPath) > df <- read.df(jsonPath, "json", "name STRING, age DOUBLE") > collect(df) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-21208) Ability to "setLocalProperty" from sc, in sparkR
[ https://issues.apache.org/jira/browse/SPARK-21208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21208: Assignee: (was: Apache Spark) > Ability to "setLocalProperty" from sc, in sparkR > > > Key: SPARK-21208 > URL: https://issues.apache.org/jira/browse/SPARK-21208 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 2.1.1 >Reporter: Karuppayya > > Checked the API > [documentation|https://spark.apache.org/docs/latest/api/R/index.html] for > sparkR. > Was not able to find a way to *setLocalProperty* on sc. > Need ability to *setLocalProperty* on sparkContext(similar to available for > pyspark, scala) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-21224) Support a DDL-formatted string as schema in reading for R
[ https://issues.apache.org/jira/browse/SPARK-21224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-21224: - Description: This might have to be a followup for SPARK-20431 but I just decided to make this separate for R specifically as it might be confusing. Please refer the discussion in the PR and SPARK-20431. In a simple view, this JIRA describes the support for a DDL-formetted string as schema as below: {code} mockLines <- c("{\"name\":\"Michael\"}", "{\"name\":\"Andy\", \"age\":30}", "{\"name\":\"Justin\", \"age\":19}") jsonPath <- tempfile(pattern = "sparkr-test", fileext = ".tmp") writeLines(mockLines, jsonPath) df <- read.df(jsonPath, "json", "name STRING, age DOUBLE") collect(df) {code} was: This might have to be a followup for SPARK-20431 but I just decided to make this separate for R specifically as it might be confused. Please refer the discussion in the PR and SPARK-20431. In a simple view, this JIRA describes the support for a DDL-formetted string as schema as below: {code} mockLines <- c("{\"name\":\"Michael\"}", "{\"name\":\"Andy\", \"age\":30}", "{\"name\":\"Justin\", \"age\":19}") jsonPath <- tempfile(pattern = "sparkr-test", fileext = ".tmp") writeLines(mockLines, jsonPath) df <- read.df(jsonPath, "json", "name STRING, age DOUBLE") collect(df) {code} > Support a DDL-formatted string as schema in reading for R > - > > Key: SPARK-21224 > URL: https://issues.apache.org/jira/browse/SPARK-21224 > Project: Spark > Issue Type: Improvement > Components: SparkR >Affects Versions: 2.2.0 >Reporter: Hyukjin Kwon >Priority: Minor > > This might have to be a followup for SPARK-20431 but I just decided to make > this separate for R specifically as it might be confusing. > Please refer the discussion in the PR and SPARK-20431. > In a simple view, this JIRA describes the support for a DDL-formetted string > as schema as below: > {code} > mockLines <- c("{\"name\":\"Michael\"}", >"{\"name\":\"Andy\", \"age\":30}", >"{\"name\":\"Justin\", \"age\":19}") > jsonPath <- tempfile(pattern = "sparkr-test", fileext = ".tmp") > writeLines(mockLines, jsonPath) > df <- read.df(jsonPath, "json", "name STRING, age DOUBLE") > collect(df) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21222) Move elimination of Distinct clause from analyzer to optimizer
[ https://issues.apache.org/jira/browse/SPARK-21222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064441#comment-16064441 ] Sean Owen commented on SPARK-21222: --- Do you mean this is the same as "SELECT MAX(a) FROM ..." ? Your two examples are identical. > Move elimination of Distinct clause from analyzer to optimizer > -- > > Key: SPARK-21222 > URL: https://issues.apache.org/jira/browse/SPARK-21222 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.2.0 >Reporter: Gengliang Wang >Priority: Minor > > Distinct clause is after MAX/MIN clause > "Select MAX(distinct a) FROM src from" > is equivalent of > "Select MAX(distinct a) FROM src from" > However, this optimization is implemented in analyzer. It should be in > optimizer. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-21208) Ability to "setLocalProperty" from sc, in sparkR
[ https://issues.apache.org/jira/browse/SPARK-21208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21208: Assignee: Apache Spark > Ability to "setLocalProperty" from sc, in sparkR > > > Key: SPARK-21208 > URL: https://issues.apache.org/jira/browse/SPARK-21208 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 2.1.1 >Reporter: Karuppayya >Assignee: Apache Spark > > Checked the API > [documentation|https://spark.apache.org/docs/latest/api/R/index.html] for > sparkR. > Was not able to find a way to *setLocalProperty* on sc. > Need ability to *setLocalProperty* on sparkContext(similar to available for > pyspark, scala) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21208) Ability to "setLocalProperty" from sc, in sparkR
[ https://issues.apache.org/jira/browse/SPARK-21208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064438#comment-16064438 ] Apache Spark commented on SPARK-21208: -- User 'HyukjinKwon' has created a pull request for this issue: https://github.com/apache/spark/pull/18431 > Ability to "setLocalProperty" from sc, in sparkR > > > Key: SPARK-21208 > URL: https://issues.apache.org/jira/browse/SPARK-21208 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 2.1.1 >Reporter: Karuppayya > > Checked the API > [documentation|https://spark.apache.org/docs/latest/api/R/index.html] for > sparkR. > Was not able to find a way to *setLocalProperty* on sc. > Need ability to *setLocalProperty* on sparkContext(similar to available for > pyspark, scala) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21208) Ability to "setLocalProperty" from sc, in sparkR
[ https://issues.apache.org/jira/browse/SPARK-21208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064448#comment-16064448 ] Hyukjin Kwon commented on SPARK-21208: -- Ooops, I linked wrong JIRA. I renamed it back and removed the link. > Ability to "setLocalProperty" from sc, in sparkR > > > Key: SPARK-21208 > URL: https://issues.apache.org/jira/browse/SPARK-21208 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 2.1.1 >Reporter: Karuppayya > > Checked the API > [documentation|https://spark.apache.org/docs/latest/api/R/index.html] for > sparkR. > Was not able to find a way to *setLocalProperty* on sc. > Need ability to *setLocalProperty* on sparkContext(similar to available for > pyspark, scala) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21224) Support a DDL-formatted string as schema in reading for R
[ https://issues.apache.org/jira/browse/SPARK-21224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064446#comment-16064446 ] Apache Spark commented on SPARK-21224: -- User 'HyukjinKwon' has created a pull request for this issue: https://github.com/apache/spark/pull/18431 > Support a DDL-formatted string as schema in reading for R > - > > Key: SPARK-21224 > URL: https://issues.apache.org/jira/browse/SPARK-21224 > Project: Spark > Issue Type: Improvement > Components: SparkR >Affects Versions: 2.2.0 >Reporter: Hyukjin Kwon >Priority: Minor > > This might have to be a followup for SPARK-20431 but I just decided to make > this separate for R specifically as many PRs might be confusing. > Please refer the discussion in the PR and SPARK-20431. > In a simple view, this JIRA describes the support for a DDL-formetted string > as schema as below: > {code} > mockLines <- c("{\"name\":\"Michael\"}", >"{\"name\":\"Andy\", \"age\":30}", >"{\"name\":\"Justin\", \"age\":19}") > jsonPath <- tempfile(pattern = "sparkr-test", fileext = ".tmp") > writeLines(mockLines, jsonPath) > df <- read.df(jsonPath, "json", "name STRING, age DOUBLE") > collect(df) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-21224) Support a DDL-formatted string as schema in reading for R
[ https://issues.apache.org/jira/browse/SPARK-21224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-21224: - Description: This might have to be a followup for SPARK-20431 but I just decided to make this separate for R specifically as many PRs might be confusing. Please refer the discussion in the PR and SPARK-20431. In a simple view, this JIRA describes the support for a DDL-formetted string as schema as below: {code} mockLines <- c("{\"name\":\"Michael\"}", "{\"name\":\"Andy\", \"age\":30}", "{\"name\":\"Justin\", \"age\":19}") jsonPath <- tempfile(pattern = "sparkr-test", fileext = ".tmp") writeLines(mockLines, jsonPath) df <- read.df(jsonPath, "json", "name STRING, age DOUBLE") collect(df) {code} was: This might have to be a followup for SPARK-20431 but I just decided to make this separate for R specifically as it might be confusing. Please refer the discussion in the PR and SPARK-20431. In a simple view, this JIRA describes the support for a DDL-formetted string as schema as below: {code} mockLines <- c("{\"name\":\"Michael\"}", "{\"name\":\"Andy\", \"age\":30}", "{\"name\":\"Justin\", \"age\":19}") jsonPath <- tempfile(pattern = "sparkr-test", fileext = ".tmp") writeLines(mockLines, jsonPath) df <- read.df(jsonPath, "json", "name STRING, age DOUBLE") collect(df) {code} > Support a DDL-formatted string as schema in reading for R > - > > Key: SPARK-21224 > URL: https://issues.apache.org/jira/browse/SPARK-21224 > Project: Spark > Issue Type: Improvement > Components: SparkR >Affects Versions: 2.2.0 >Reporter: Hyukjin Kwon >Priority: Minor > > This might have to be a followup for SPARK-20431 but I just decided to make > this separate for R specifically as many PRs might be confusing. > Please refer the discussion in the PR and SPARK-20431. > In a simple view, this JIRA describes the support for a DDL-formetted string > as schema as below: > {code} > mockLines <- c("{\"name\":\"Michael\"}", >"{\"name\":\"Andy\", \"age\":30}", >"{\"name\":\"Justin\", \"age\":19}") > jsonPath <- tempfile(pattern = "sparkr-test", fileext = ".tmp") > writeLines(mockLines, jsonPath) > df <- read.df(jsonPath, "json", "name STRING, age DOUBLE") > collect(df) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-21224) Support a DDL-formatted string as schema in reading for R
[ https://issues.apache.org/jira/browse/SPARK-21224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21224: Assignee: Apache Spark > Support a DDL-formatted string as schema in reading for R > - > > Key: SPARK-21224 > URL: https://issues.apache.org/jira/browse/SPARK-21224 > Project: Spark > Issue Type: Improvement > Components: SparkR >Affects Versions: 2.2.0 >Reporter: Hyukjin Kwon >Assignee: Apache Spark >Priority: Minor > > This might have to be a followup for SPARK-20431 but I just decided to make > this separate for R specifically as many PRs might be confusing. > Please refer the discussion in the PR and SPARK-20431. > In a simple view, this JIRA describes the support for a DDL-formetted string > as schema as below: > {code} > mockLines <- c("{\"name\":\"Michael\"}", >"{\"name\":\"Andy\", \"age\":30}", >"{\"name\":\"Justin\", \"age\":19}") > jsonPath <- tempfile(pattern = "sparkr-test", fileext = ".tmp") > writeLines(mockLines, jsonPath) > df <- read.df(jsonPath, "json", "name STRING, age DOUBLE") > collect(df) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-21224) Support a DDL-formatted string as schema in reading for R
[ https://issues.apache.org/jira/browse/SPARK-21224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21224: Assignee: (was: Apache Spark) > Support a DDL-formatted string as schema in reading for R > - > > Key: SPARK-21224 > URL: https://issues.apache.org/jira/browse/SPARK-21224 > Project: Spark > Issue Type: Improvement > Components: SparkR >Affects Versions: 2.2.0 >Reporter: Hyukjin Kwon >Priority: Minor > > This might have to be a followup for SPARK-20431 but I just decided to make > this separate for R specifically as many PRs might be confusing. > Please refer the discussion in the PR and SPARK-20431. > In a simple view, this JIRA describes the support for a DDL-formetted string > as schema as below: > {code} > mockLines <- c("{\"name\":\"Michael\"}", >"{\"name\":\"Andy\", \"age\":30}", >"{\"name\":\"Justin\", \"age\":19}") > jsonPath <- tempfile(pattern = "sparkr-test", fileext = ".tmp") > writeLines(mockLines, jsonPath) > df <- read.df(jsonPath, "json", "name STRING, age DOUBLE") > collect(df) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-20977) NPE in CollectionAccumulator
[ https://issues.apache.org/jira/browse/SPARK-20977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sharkd tu updated SPARK-20977: -- Description: {code:java} 17/06/03 13:39:31 ERROR Utils: Uncaught exception in thread heartbeat-receiver-event-loop-thread java.lang.NullPointerException at org.apache.spark.util.CollectionAccumulator.value(AccumulatorV2.scala:464) at org.apache.spark.util.CollectionAccumulator.value(AccumulatorV2.scala:439) at org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$6$$anonfun$7.apply(TaskSchedulerImpl.scala:408) at org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$6$$anonfun$7.apply(TaskSchedulerImpl.scala:408) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.Iterator$class.foreach(Iterator.scala:893) at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.AbstractTraversable.map(Traversable.scala:104) at org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$6.apply(TaskSchedulerImpl.scala:408) at org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$6.apply(TaskSchedulerImpl.scala:407) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241) at scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:186) at org.apache.spark.scheduler.TaskSchedulerImpl.executorHeartbeatReceived(TaskSchedulerImpl.scala:407) at org.apache.spark.HeartbeatReceiver$$anonfun$receiveAndReply$1$$anon$2$$anonfun$run$2.apply$mcV$sp(HeartbeatReceiver.scala:129) at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1283) at org.apache.spark.HeartbeatReceiver$$anonfun$receiveAndReply$1$$anon$2.run(HeartbeatReceiver.scala:128) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) {code} Is that the bug of openjdk or spark? Has anybody ever hit the problem? was: 17/06/03 13:39:31 ERROR Utils: Uncaught exception in thread heartbeat-receiver-event-loop-thread java.lang.NullPointerException at org.apache.spark.util.CollectionAccumulator.value(AccumulatorV2.scala:464) at org.apache.spark.util.CollectionAccumulator.value(AccumulatorV2.scala:439) at org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$6$$anonfun$7.apply(TaskSchedulerImpl.scala:408) at org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$6$$anonfun$7.apply(TaskSchedulerImpl.scala:408) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.Iterator$class.foreach(Iterator.scala:893) at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.AbstractTraversable.map(Traversable.scala:104) at org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$6.apply(TaskSchedulerImpl.scala:408) at org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$6.apply(TaskSchedulerImpl.scala:407) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.Array
[jira] [Updated] (SPARK-20977) NPE in CollectionAccumulator
[ https://issues.apache.org/jira/browse/SPARK-20977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sharkd tu updated SPARK-20977: -- Environment: (was: OpenJDK 64-Bit Server VM (25.71-b00) for linux-ppc64 JRE (1.8.0-internal-centos_2017_04_25_01_11-b00), built on Apr 25 2017 01:24:21 by "centos" with gcc 6.3.1 20170110 (Advance-Toolchain-at10.0) IBM AT 10 branch) > NPE in CollectionAccumulator > > > Key: SPARK-20977 > URL: https://issues.apache.org/jira/browse/SPARK-20977 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.1.1 >Reporter: sharkd tu > > {code:java} > 17/06/03 13:39:31 ERROR Utils: Uncaught exception in thread > heartbeat-receiver-event-loop-thread > java.lang.NullPointerException > at > org.apache.spark.util.CollectionAccumulator.value(AccumulatorV2.scala:464) > at > org.apache.spark.util.CollectionAccumulator.value(AccumulatorV2.scala:439) > at > org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$6$$anonfun$7.apply(TaskSchedulerImpl.scala:408) > at > org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$6$$anonfun$7.apply(TaskSchedulerImpl.scala:408) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at scala.collection.Iterator$class.foreach(Iterator.scala:893) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) > at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > at scala.collection.AbstractIterable.foreach(Iterable.scala:54) > at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) > at scala.collection.AbstractTraversable.map(Traversable.scala:104) > at > org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$6.apply(TaskSchedulerImpl.scala:408) > at > org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$6.apply(TaskSchedulerImpl.scala:407) > at > scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) > at > scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) > at > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) > at > scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241) > at scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:186) > at > org.apache.spark.scheduler.TaskSchedulerImpl.executorHeartbeatReceived(TaskSchedulerImpl.scala:407) > at > org.apache.spark.HeartbeatReceiver$$anonfun$receiveAndReply$1$$anon$2$$anonfun$run$2.apply$mcV$sp(HeartbeatReceiver.scala:129) > at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1283) > at > org.apache.spark.HeartbeatReceiver$$anonfun$receiveAndReply$1$$anon$2.run(HeartbeatReceiver.scala:128) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > Is that the bug of openjdk or spark? Has anybody ever hit the problem? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-20977) NPE in CollectionAccumulator
[ https://issues.apache.org/jira/browse/SPARK-20977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sharkd tu updated SPARK-20977: -- Description: {code:java} 17/06/03 13:39:31 ERROR Utils: Uncaught exception in thread heartbeat-receiver-event-loop-thread java.lang.NullPointerException at org.apache.spark.util.CollectionAccumulator.value(AccumulatorV2.scala:464) at org.apache.spark.util.CollectionAccumulator.value(AccumulatorV2.scala:439) at org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$6$$anonfun$7.apply(TaskSchedulerImpl.scala:408) at org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$6$$anonfun$7.apply(TaskSchedulerImpl.scala:408) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.Iterator$class.foreach(Iterator.scala:893) at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.AbstractTraversable.map(Traversable.scala:104) at org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$6.apply(TaskSchedulerImpl.scala:408) at org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$6.apply(TaskSchedulerImpl.scala:407) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241) at scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:186) at org.apache.spark.scheduler.TaskSchedulerImpl.executorHeartbeatReceived(TaskSchedulerImpl.scala:407) at org.apache.spark.HeartbeatReceiver$$anonfun$receiveAndReply$1$$anon$2$$anonfun$run$2.apply$mcV$sp(HeartbeatReceiver.scala:129) at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1283) at org.apache.spark.HeartbeatReceiver$$anonfun$receiveAndReply$1$$anon$2.run(HeartbeatReceiver.scala:128) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) {code} Is that the bug of spark? Has anybody ever hit the problem? was: {code:java} 17/06/03 13:39:31 ERROR Utils: Uncaught exception in thread heartbeat-receiver-event-loop-thread java.lang.NullPointerException at org.apache.spark.util.CollectionAccumulator.value(AccumulatorV2.scala:464) at org.apache.spark.util.CollectionAccumulator.value(AccumulatorV2.scala:439) at org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$6$$anonfun$7.apply(TaskSchedulerImpl.scala:408) at org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$6$$anonfun$7.apply(TaskSchedulerImpl.scala:408) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.Iterator$class.foreach(Iterator.scala:893) at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.AbstractTraversable.map(Traversable.scala:104) at org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$6.apply(TaskSchedulerImpl.scala:408) at org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$6.apply(TaskSchedulerImpl.scala:407) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.Arra
[jira] [Commented] (SPARK-21063) Spark return an empty result from remote hadoop cluster
[ https://issues.apache.org/jira/browse/SPARK-21063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064469#comment-16064469 ] Peter Bykov commented on SPARK-21063: - [~q79969786] i have both spark hive-thriftserver and hive-thriftserver, but nothing works. I have several spark installations, including HDP 2.6 with Spark2 and launched spark thrift server. And the same behavior is observed on each cluster in 100% scenarios. I definitely confused with this and even don't know where i should look. > Spark return an empty result from remote hadoop cluster > --- > > Key: SPARK-21063 > URL: https://issues.apache.org/jira/browse/SPARK-21063 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 2.1.0, 2.1.1 >Reporter: Peter Bykov > > Spark returning empty result from when querying remote hadoop cluster. > All firewall settings removed. > Querying using JDBC working properly using hive-jdbc driver from version 1.1.1 > Code snippet is: > {code:java} > val spark = SparkSession.builder > .appName("RemoteSparkTest") > .master("local") > .getOrCreate() > val df = spark.read > .option("url", "jdbc:hive2://remote.hive.local:1/default") > .option("user", "user") > .option("password", "pass") > .option("dbtable", "test_table") > .option("driver", "org.apache.hive.jdbc.HiveDriver") > .format("jdbc") > .load() > > df.show() > {code} > Result: > {noformat} > +---+ > |test_table.test_col| > +---+ > +---+ > {noformat} > All manipulations like: > {code:java} > df.select(*).show() > {code} > returns empty result too. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21063) Spark return an empty result from remote hadoop cluster
[ https://issues.apache.org/jira/browse/SPARK-21063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064481#comment-16064481 ] Peter Bykov commented on SPARK-21063: - [~q79969786] also, i can receive data using JDBC connection in Scala without Spark API > Spark return an empty result from remote hadoop cluster > --- > > Key: SPARK-21063 > URL: https://issues.apache.org/jira/browse/SPARK-21063 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 2.1.0, 2.1.1 >Reporter: Peter Bykov > > Spark returning empty result from when querying remote hadoop cluster. > All firewall settings removed. > Querying using JDBC working properly using hive-jdbc driver from version 1.1.1 > Code snippet is: > {code:java} > val spark = SparkSession.builder > .appName("RemoteSparkTest") > .master("local") > .getOrCreate() > val df = spark.read > .option("url", "jdbc:hive2://remote.hive.local:1/default") > .option("user", "user") > .option("password", "pass") > .option("dbtable", "test_table") > .option("driver", "org.apache.hive.jdbc.HiveDriver") > .format("jdbc") > .load() > > df.show() > {code} > Result: > {noformat} > +---+ > |test_table.test_col| > +---+ > +---+ > {noformat} > All manipulations like: > {code:java} > df.select(*).show() > {code} > returns empty result too. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-21196) Split codegen info of query plan into sequence
[ https://issues.apache.org/jira/browse/SPARK-21196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-21196: --- Assignee: Gengliang Wang > Split codegen info of query plan into sequence > -- > > Key: SPARK-21196 > URL: https://issues.apache.org/jira/browse/SPARK-21196 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.2.0 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Minor > Fix For: 2.3.0 > > > codegen info of query plan can be very long. > In debugging console / web page, it would be more readable if the subtrees > and corresponding codegen are split into sequence. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-21196) Split codegen info of query plan into sequence
[ https://issues.apache.org/jira/browse/SPARK-21196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21196. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18409 [https://github.com/apache/spark/pull/18409] > Split codegen info of query plan into sequence > -- > > Key: SPARK-21196 > URL: https://issues.apache.org/jira/browse/SPARK-21196 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.2.0 >Reporter: Gengliang Wang >Priority: Minor > Fix For: 2.3.0 > > > codegen info of query plan can be very long. > In debugging console / web page, it would be more readable if the subtrees > and corresponding codegen are split into sequence. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-21225) decrease the Mem using for variable 'tasks' in function resourceOffers
yangZhiguo created SPARK-21225: -- Summary: decrease the Mem using for variable 'tasks' in function resourceOffers Key: SPARK-21225 URL: https://issues.apache.org/jira/browse/SPARK-21225 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 2.1.1, 2.1.0 Reporter: yangZhiguo Priority: Minor In the function 'resourceOffers', It declare a variable 'tasks' for storage the tasks which have allocated a executor. It declared like this: *{color:#d04437}val tasks = shuffledOffers.map(o => new ArrayBuffer[TaskDescription](o.cores)){color}* But, I think this code only conside a situation for that one task per core. If the user config the "spark.task.cpus" as 2 or 3, It really don't need so much space. I think It can motify as follow: val tasks = shuffledOffers.map(o => new ArrayBuffer[TaskDescription](Math.ceil(o.cores*1.0/CPUS_PER_TASK).toInt)) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-21218) Convert IN predicate to equivalent Parquet filter
[ https://issues.apache.org/jira/browse/SPARK-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Styles updated SPARK-21218: --- Attachment: Starscream Console on OTT---Michael-Styles---MBP-15-inch-Mid-2015 - Details for Query 1 2017-06-27 06-02-53.png Starscream Console on OTT---Michael-Styles---MBP-15-inch-Mid-2015 - Details for Query 0 2017-06-27 06-03-29.png > Convert IN predicate to equivalent Parquet filter > - > > Key: SPARK-21218 > URL: https://issues.apache.org/jira/browse/SPARK-21218 > Project: Spark > Issue Type: Improvement > Components: Optimizer >Affects Versions: 2.1.1 >Reporter: Michael Styles > Attachments: Starscream Console on > OTT---Michael-Styles---MBP-15-inch-Mid-2015 - Details for Query 0 2017-06-27 > 06-03-29.png, Starscream Console on > OTT---Michael-Styles---MBP-15-inch-Mid-2015 - Details for Query 1 2017-06-27 > 06-02-53.png > > > Convert IN predicate to equivalent expression involving equality conditions > to allow the filter to be pushed down to Parquet. > For instance, > C1 IN (10, 20) is rewritten as (C1 = 10) OR (C1 = 20) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-21218) Convert IN predicate to equivalent Parquet filter
[ https://issues.apache.org/jira/browse/SPARK-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Styles updated SPARK-21218: --- Attachment: IN Predicate.png OR Predicate.png > Convert IN predicate to equivalent Parquet filter > - > > Key: SPARK-21218 > URL: https://issues.apache.org/jira/browse/SPARK-21218 > Project: Spark > Issue Type: Improvement > Components: Optimizer >Affects Versions: 2.1.1 >Reporter: Michael Styles > Attachments: IN Predicate.png, OR Predicate.png, Starscream Console > on OTT---Michael-Styles---MBP-15-inch-Mid-2015 - Details for Query 0 > 2017-06-27 06-03-29.png, Starscream Console on > OTT---Michael-Styles---MBP-15-inch-Mid-2015 - Details for Query 1 2017-06-27 > 06-02-53.png > > > Convert IN predicate to equivalent expression involving equality conditions > to allow the filter to be pushed down to Parquet. > For instance, > C1 IN (10, 20) is rewritten as (C1 = 10) OR (C1 = 20) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-21218) Convert IN predicate to equivalent Parquet filter
[ https://issues.apache.org/jira/browse/SPARK-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Styles updated SPARK-21218: --- Attachment: (was: Starscream Console on OTT---Michael-Styles---MBP-15-inch-Mid-2015 - Details for Query 0 2017-06-27 06-03-29.png) > Convert IN predicate to equivalent Parquet filter > - > > Key: SPARK-21218 > URL: https://issues.apache.org/jira/browse/SPARK-21218 > Project: Spark > Issue Type: Improvement > Components: Optimizer >Affects Versions: 2.1.1 >Reporter: Michael Styles > Attachments: IN Predicate.png, OR Predicate.png > > > Convert IN predicate to equivalent expression involving equality conditions > to allow the filter to be pushed down to Parquet. > For instance, > C1 IN (10, 20) is rewritten as (C1 = 10) OR (C1 = 20) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-21218) Convert IN predicate to equivalent Parquet filter
[ https://issues.apache.org/jira/browse/SPARK-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Styles updated SPARK-21218: --- Attachment: (was: Starscream Console on OTT---Michael-Styles---MBP-15-inch-Mid-2015 - Details for Query 1 2017-06-27 06-02-53.png) > Convert IN predicate to equivalent Parquet filter > - > > Key: SPARK-21218 > URL: https://issues.apache.org/jira/browse/SPARK-21218 > Project: Spark > Issue Type: Improvement > Components: Optimizer >Affects Versions: 2.1.1 >Reporter: Michael Styles > Attachments: IN Predicate.png, OR Predicate.png > > > Convert IN predicate to equivalent expression involving equality conditions > to allow the filter to be pushed down to Parquet. > For instance, > C1 IN (10, 20) is rewritten as (C1 = 10) OR (C1 = 20) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21218) Convert IN predicate to equivalent Parquet filter
[ https://issues.apache.org/jira/browse/SPARK-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064608#comment-16064608 ] Michael Styles commented on SPARK-21218: By not pushing the filter to Parquet, are we not preventing Parquet from skipping blocks during read operations? I have tests that show big improvements when applying this transformation. For instance, I have a Parquet file with 162,456,394 rows which is sorted on column C1. *IN Predicate* {noformat} df.filter[df['C1'].isin([42, 139])).collect() {noformat} !IN Predicate.png|thumbnail! *OR Predicate* {noformat} df.filter((df['C1'] == 42) | (df['C1'] == 139)).collect() {noformat} !OR Pedicate.png|thumbnail! Notice the difference in the number of output rows for the scan. > Convert IN predicate to equivalent Parquet filter > - > > Key: SPARK-21218 > URL: https://issues.apache.org/jira/browse/SPARK-21218 > Project: Spark > Issue Type: Improvement > Components: Optimizer >Affects Versions: 2.1.1 >Reporter: Michael Styles > Attachments: IN Predicate.png, OR Predicate.png > > > Convert IN predicate to equivalent expression involving equality conditions > to allow the filter to be pushed down to Parquet. > For instance, > C1 IN (10, 20) is rewritten as (C1 = 10) OR (C1 = 20) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-21218) Convert IN predicate to equivalent Parquet filter
[ https://issues.apache.org/jira/browse/SPARK-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064608#comment-16064608 ] Michael Styles edited comment on SPARK-21218 at 6/27/17 10:25 AM: -- By not pushing the filter to Parquet, are we not preventing Parquet from skipping blocks during read operations? I have tests that show big improvements when applying this transformation. For instance, I have a Parquet file with 162,456,394 rows which is sorted on column C1. *IN Predicate* {noformat} df.filter[df['C1'].isin([42, 139])).collect() {noformat} !IN Predicate.png|thumbnail! *OR Predicate* {noformat} df.filter((df['C1'] == 42) | (df['C1'] == 139)).collect() {noformat} !OR Predicate.png|thumbnail! Notice the difference in the number of output rows for the scan. was (Author: ptkool): By not pushing the filter to Parquet, are we not preventing Parquet from skipping blocks during read operations? I have tests that show big improvements when applying this transformation. For instance, I have a Parquet file with 162,456,394 rows which is sorted on column C1. *IN Predicate* {noformat} df.filter[df['C1'].isin([42, 139])).collect() {noformat} !IN Predicate.png|thumbnail! *OR Predicate* {noformat} df.filter((df['C1'] == 42) | (df['C1'] == 139)).collect() {noformat} !OR Pedicate.png|thumbnail! Notice the difference in the number of output rows for the scan. > Convert IN predicate to equivalent Parquet filter > - > > Key: SPARK-21218 > URL: https://issues.apache.org/jira/browse/SPARK-21218 > Project: Spark > Issue Type: Improvement > Components: Optimizer >Affects Versions: 2.1.1 >Reporter: Michael Styles > Attachments: IN Predicate.png, OR Predicate.png > > > Convert IN predicate to equivalent expression involving equality conditions > to allow the filter to be pushed down to Parquet. > For instance, > C1 IN (10, 20) is rewritten as (C1 = 10) OR (C1 = 20) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-21218) Convert IN predicate to equivalent Parquet filter
[ https://issues.apache.org/jira/browse/SPARK-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064608#comment-16064608 ] Michael Styles edited comment on SPARK-21218 at 6/27/17 10:30 AM: -- By not pushing the filter to Parquet, are we not preventing Parquet from skipping blocks during read operations? I have tests that show big improvements when applying this transformation. For instance, I have a Parquet file with 162,456,394 rows which is sorted on column C1. *IN Predicate* {noformat} df.filter[df['C1'].isin([42, 139])).collect() {noformat} *OR Predicate* {noformat} df.filter((df['C1'] == 42) | (df['C1'] == 139)).collect() {noformat} Notice the difference in the number of output rows for the scans (see attachments). Also, the IN predicate test took about 1.1 minutes, while the OR predicate test took about 16 seconds. was (Author: ptkool): By not pushing the filter to Parquet, are we not preventing Parquet from skipping blocks during read operations? I have tests that show big improvements when applying this transformation. For instance, I have a Parquet file with 162,456,394 rows which is sorted on column C1. *IN Predicate* {noformat} df.filter[df['C1'].isin([42, 139])).collect() {noformat} !IN Predicate.png|thumbnail! *OR Predicate* {noformat} df.filter((df['C1'] == 42) | (df['C1'] == 139)).collect() {noformat} !OR Predicate.png|thumbnail! Notice the difference in the number of output rows for the scan. > Convert IN predicate to equivalent Parquet filter > - > > Key: SPARK-21218 > URL: https://issues.apache.org/jira/browse/SPARK-21218 > Project: Spark > Issue Type: Improvement > Components: Optimizer >Affects Versions: 2.1.1 >Reporter: Michael Styles > Attachments: IN Predicate.png, OR Predicate.png > > > Convert IN predicate to equivalent expression involving equality conditions > to allow the filter to be pushed down to Parquet. > For instance, > C1 IN (10, 20) is rewritten as (C1 = 10) OR (C1 = 20) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-21226) Save empty dataframe in pyspark prints nothing
Carlos M. Casas created SPARK-21226: --- Summary: Save empty dataframe in pyspark prints nothing Key: SPARK-21226 URL: https://issues.apache.org/jira/browse/SPARK-21226 Project: Spark Issue Type: Bug Components: PySpark Affects Versions: 2.1.0, 2.0.0 Reporter: Carlos M. Casas I try the following: schema = whatever schema you want df1 = sqlContext.createDataFrame(sc.emptyRDD(), schema) df1.write.parquet("as1") and I just get a directory as1 with a _SUCCESS file in it. If I try to read that file, I get an exception. On the other hand, if I run: schema = whatever schema you want df2 = sqlContext.createDataFrame([], schema) df2.write.parquet("as2") I get a directory as2 with some files on it (representing field type information?). If I try to read it, it works: it read an empty df with the proper schema. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-21225) decrease the Mem using for variable 'tasks' in function resourceOffers
[ https://issues.apache.org/jira/browse/SPARK-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yangZhiguo updated SPARK-21225: --- Description: In the function 'resourceOffers', It declare a variable 'tasks' for storage the tasks which have allocated a executor. It declared like this: *{color:#d04437}val tasks = shuffledOffers.map(o => new ArrayBuffer[TaskDescription](o.cores)){color}* But, I think this code only conside a situation for that one task per core. If the user config the "spark.task.cpus" as 2 or 3, It really don't need so much space. I think It can motify as follow: {color:#14892c}*val tasks = shuffledOffers.map(o => new ArrayBuffer[TaskDescription](Math.ceil(o.cores*1.0/CPUS_PER_TASK).toInt))*{color} was: In the function 'resourceOffers', It declare a variable 'tasks' for storage the tasks which have allocated a executor. It declared like this: *{color:#d04437}val tasks = shuffledOffers.map(o => new ArrayBuffer[TaskDescription](o.cores)){color}* But, I think this code only conside a situation for that one task per core. If the user config the "spark.task.cpus" as 2 or 3, It really don't need so much space. I think It can motify as follow: val tasks = shuffledOffers.map(o => new ArrayBuffer[TaskDescription](Math.ceil(o.cores*1.0/CPUS_PER_TASK).toInt)) > decrease the Mem using for variable 'tasks' in function resourceOffers > -- > > Key: SPARK-21225 > URL: https://issues.apache.org/jira/browse/SPARK-21225 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 2.1.0, 2.1.1 >Reporter: yangZhiguo >Priority: Minor > Original Estimate: 1h > Remaining Estimate: 1h > > In the function 'resourceOffers', It declare a variable 'tasks' for > storage the tasks which have allocated a executor. It declared like this: > *{color:#d04437}val tasks = shuffledOffers.map(o => new > ArrayBuffer[TaskDescription](o.cores)){color}* > But, I think this code only conside a situation for that one task per core. > If the user config the "spark.task.cpus" as 2 or 3, It really don't need so > much space. I think It can motify as follow: > {color:#14892c}*val tasks = shuffledOffers.map(o => new > ArrayBuffer[TaskDescription](Math.ceil(o.cores*1.0/CPUS_PER_TASK).toInt))*{color} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21225) decrease the Mem using for variable 'tasks' in function resourceOffers
[ https://issues.apache.org/jira/browse/SPARK-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064658#comment-16064658 ] Apache Spark commented on SPARK-21225: -- User 'JackYangzg' has created a pull request for this issue: https://github.com/apache/spark/pull/18434 > decrease the Mem using for variable 'tasks' in function resourceOffers > -- > > Key: SPARK-21225 > URL: https://issues.apache.org/jira/browse/SPARK-21225 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 2.1.0, 2.1.1 >Reporter: yangZhiguo >Priority: Minor > Original Estimate: 1h > Remaining Estimate: 1h > > In the function 'resourceOffers', It declare a variable 'tasks' for > storage the tasks which have allocated a executor. It declared like this: > *{color:#d04437}val tasks = shuffledOffers.map(o => new > ArrayBuffer[TaskDescription](o.cores)){color}* > But, I think this code only conside a situation for that one task per core. > If the user config the "spark.task.cpus" as 2 or 3, It really don't need so > much space. I think It can motify as follow: > {color:#14892c}*val tasks = shuffledOffers.map(o => new > ArrayBuffer[TaskDescription](Math.ceil(o.cores*1.0/CPUS_PER_TASK).toInt))*{color} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-21225) decrease the Mem using for variable 'tasks' in function resourceOffers
[ https://issues.apache.org/jira/browse/SPARK-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21225: Assignee: Apache Spark > decrease the Mem using for variable 'tasks' in function resourceOffers > -- > > Key: SPARK-21225 > URL: https://issues.apache.org/jira/browse/SPARK-21225 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 2.1.0, 2.1.1 >Reporter: yangZhiguo >Assignee: Apache Spark >Priority: Minor > Original Estimate: 1h > Remaining Estimate: 1h > > In the function 'resourceOffers', It declare a variable 'tasks' for > storage the tasks which have allocated a executor. It declared like this: > *{color:#d04437}val tasks = shuffledOffers.map(o => new > ArrayBuffer[TaskDescription](o.cores)){color}* > But, I think this code only conside a situation for that one task per core. > If the user config the "spark.task.cpus" as 2 or 3, It really don't need so > much space. I think It can motify as follow: > {color:#14892c}*val tasks = shuffledOffers.map(o => new > ArrayBuffer[TaskDescription](Math.ceil(o.cores*1.0/CPUS_PER_TASK).toInt))*{color} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-21225) decrease the Mem using for variable 'tasks' in function resourceOffers
[ https://issues.apache.org/jira/browse/SPARK-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21225: Assignee: (was: Apache Spark) > decrease the Mem using for variable 'tasks' in function resourceOffers > -- > > Key: SPARK-21225 > URL: https://issues.apache.org/jira/browse/SPARK-21225 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 2.1.0, 2.1.1 >Reporter: yangZhiguo >Priority: Minor > Original Estimate: 1h > Remaining Estimate: 1h > > In the function 'resourceOffers', It declare a variable 'tasks' for > storage the tasks which have allocated a executor. It declared like this: > *{color:#d04437}val tasks = shuffledOffers.map(o => new > ArrayBuffer[TaskDescription](o.cores)){color}* > But, I think this code only conside a situation for that one task per core. > If the user config the "spark.task.cpus" as 2 or 3, It really don't need so > much space. I think It can motify as follow: > {color:#14892c}*val tasks = shuffledOffers.map(o => new > ArrayBuffer[TaskDescription](Math.ceil(o.cores*1.0/CPUS_PER_TASK).toInt))*{color} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-21226) Save empty dataframe in pyspark prints nothing
[ https://issues.apache.org/jira/browse/SPARK-21226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21226: -- Priority: Minor (was: Major) What is the error? > Save empty dataframe in pyspark prints nothing > -- > > Key: SPARK-21226 > URL: https://issues.apache.org/jira/browse/SPARK-21226 > Project: Spark > Issue Type: Bug > Components: PySpark >Affects Versions: 2.0.0, 2.1.0 >Reporter: Carlos M. Casas >Priority: Minor > > I try the following: > schema = whatever schema you want > df1 = sqlContext.createDataFrame(sc.emptyRDD(), schema) > df1.write.parquet("as1") > and I just get a directory as1 with a _SUCCESS file in it. If I try to read > that file, I get an exception. > On the other hand, if I run: > schema = whatever schema you want > df2 = sqlContext.createDataFrame([], schema) > df2.write.parquet("as2") > I get a directory as2 with some files on it (representing field type > information?). If I try to read it, it works: it read an empty df with the > proper schema. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21223) Thread-safety issue in FsHistoryProvider
[ https://issues.apache.org/jira/browse/SPARK-21223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064676#comment-16064676 ] Sean Owen commented on SPARK-21223: --- [~gostop_zlx] this overlaps a lot with SPARK-21078. Can you look at that one and possibly address the comment there too? > Thread-safety issue in FsHistoryProvider > - > > Key: SPARK-21223 > URL: https://issues.apache.org/jira/browse/SPARK-21223 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.1.1 >Reporter: zenglinxi > > Currently, Spark HistoryServer use a HashMap named fileToAppInfo in class > FsHistoryProvider to store the map of eventlog path and attemptInfo. > When use ThreadPool to Replay the log files in the list and merge the list of > old applications with new ones, multi thread may update fileToAppInfo at the > same time, which may cause Thread-safety issues. > {code:java} > for (file <- logInfos) { >tasks += replayExecutor.submit(new Runnable { > override def run(): Unit = mergeApplicationListing(file) > }) > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21225) decrease the Mem using for variable 'tasks' in function resourceOffers
[ https://issues.apache.org/jira/browse/SPARK-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064682#comment-16064682 ] Apache Spark commented on SPARK-21225: -- User 'JackYangzg' has created a pull request for this issue: https://github.com/apache/spark/pull/18435 > decrease the Mem using for variable 'tasks' in function resourceOffers > -- > > Key: SPARK-21225 > URL: https://issues.apache.org/jira/browse/SPARK-21225 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 2.1.0, 2.1.1 >Reporter: yangZhiguo >Priority: Minor > Original Estimate: 1h > Remaining Estimate: 1h > > In the function 'resourceOffers', It declare a variable 'tasks' for > storage the tasks which have allocated a executor. It declared like this: > *{color:#d04437}val tasks = shuffledOffers.map(o => new > ArrayBuffer[TaskDescription](o.cores)){color}* > But, I think this code only conside a situation for that one task per core. > If the user config the "spark.task.cpus" as 2 or 3, It really don't need so > much space. I think It can motify as follow: > {color:#14892c}*val tasks = shuffledOffers.map(o => new > ArrayBuffer[TaskDescription](Math.ceil(o.cores*1.0/CPUS_PER_TASK).toInt))*{color} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21226) Save empty dataframe in pyspark prints nothing
[ https://issues.apache.org/jira/browse/SPARK-21226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064706#comment-16064706 ] Carlos M. Casas commented on SPARK-21226: - The error is a different way of writing what apparently are two similar (empty) dataframes. The problem is that, depending on how we create the empty dataframe, the spark.write.parquet method creates or doesn't create the actual .parquet files. In any case, it writes the _SUCCESS file. If written, they can be read; if not written, they can't be read. Why does this method depend on how we created the dataframe? Although irrelevant, the exception I see when reading the parquet folder with just the _SUCCESS file is: pyspark.sql.utils.AnalysisException: u'Unable to infer schema for Parquet. It must be specified manually.;' > Save empty dataframe in pyspark prints nothing > -- > > Key: SPARK-21226 > URL: https://issues.apache.org/jira/browse/SPARK-21226 > Project: Spark > Issue Type: Bug > Components: PySpark >Affects Versions: 2.0.0, 2.1.0 >Reporter: Carlos M. Casas >Priority: Minor > > I try the following: > schema = whatever schema you want > df1 = sqlContext.createDataFrame(sc.emptyRDD(), schema) > df1.write.parquet("as1") > and I just get a directory as1 with a _SUCCESS file in it. If I try to read > that file, I get an exception. > On the other hand, if I run: > schema = whatever schema you want > df2 = sqlContext.createDataFrame([], schema) > df2.write.parquet("as2") > I get a directory as2 with some files on it (representing field type > information?). If I try to read it, it works: it read an empty df with the > proper schema. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20073) Unexpected Cartesian product when using eqNullSafe in join with a derived table
[ https://issues.apache.org/jira/browse/SPARK-20073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064729#comment-16064729 ] Apache Spark commented on SPARK-20073: -- User 'maropu' has created a pull request for this issue: https://github.com/apache/spark/pull/18436 > Unexpected Cartesian product when using eqNullSafe in join with a derived > table > --- > > Key: SPARK-20073 > URL: https://issues.apache.org/jira/browse/SPARK-20073 > Project: Spark > Issue Type: Bug > Components: Optimizer >Affects Versions: 2.0.2, 2.1.0 >Reporter: Everett Anderson > Labels: correctness > > It appears that if you try to join tables A and B when B is derived from A > and you use the eqNullSafe / <=> operator for the join condition, Spark > performs a Cartesian product. > However, if you perform the join on tables of the same data when they don't > have a relationship, the expected non-Cartesian product join occurs. > {noformat} > // Create some fake data. > import org.apache.spark.sql.Row > import org.apache.spark.sql.Dataset > import org.apache.spark.sql.types._ > import org.apache.spark.sql.functions > val peopleRowsRDD = sc.parallelize(Seq( > Row("Fred", 8, 1), > Row("Fred", 8, 2), > Row(null, 10, 3), > Row(null, 10, 4), > Row("Amy", 12, 5), > Row("Amy", 12, 6))) > > val peopleSchema = StructType(Seq( > StructField("name", StringType, nullable = true), > StructField("group", IntegerType, nullable = true), > StructField("data", IntegerType, nullable = true))) > > val people = spark.createDataFrame(peopleRowsRDD, peopleSchema) > people.createOrReplaceTempView("people") > scala> people.show > ++-++ > |name|group|data| > ++-++ > |Fred|8| 1| > |Fred|8| 2| > |null| 10| 3| > |null| 10| 4| > | Amy| 12| 5| > | Amy| 12| 6| > ++-++ > // Now create a derived table from that table. It doesn't matter much what. > val variantCounts = spark.sql("select name, count(distinct(name, group, > data)) as variant_count from people group by name having variant_count > 1") > variantCounts.show > ++-+ > > |name|variant_count| > ++-+ > |Fred|2| > |null|2| > | Amy|2| > ++-+ > // Now try an inner join using the regular equalTo that drops nulls. This > works fine. > val innerJoinEqualTo = variantCounts.join(people, > variantCounts("name").equalTo(people("name"))) > innerJoinEqualTo.show > ++-++-++ > > |name|variant_count|name|group|data| > ++-++-++ > |Fred|2|Fred|8| 1| > |Fred|2|Fred|8| 2| > | Amy|2| Amy| 12| 5| > | Amy|2| Amy| 12| 6| > ++-++-++ > // Okay now lets switch to the <=> operator > // > // If you haven't set spark.sql.crossJoin.enabled=true, you'll get an error > like > // "Cartesian joins could be prohibitively expensive and are disabled by > default. To explicitly enable them, please set spark.sql.crossJoin.enabled = > true;" > // > // if you have enabled them, you'll get the table below. > // > // However, we really don't want or expect a Cartesian product! > val innerJoinSqlNullSafeEqOp = variantCounts.join(people, > variantCounts("name")<=>(people("name"))) > innerJoinSqlNullSafeEqOp.show > ++-++-++ > > |name|variant_count|name|group|data| > ++-++-++ > |Fred|2|Fred|8| 1| > |Fred|2|Fred|8| 2| > |Fred|2|null| 10| 3| > |Fred|2|null| 10| 4| > |Fred|2| Amy| 12| 5| > |Fred|2| Amy| 12| 6| > |null|2|Fred|8| 1| > |null|2|Fred|8| 2| > |null|2|null| 10| 3| > |null|2|null| 10| 4| > |null|2| Amy| 12| 5| > |null|2| Amy| 12| 6| > | Amy|2|Fred|8| 1| > | Amy|2|Fred|8| 2| > | Amy|2|null| 10| 3| > | Amy|2|null| 10| 4| > | Amy|2| Amy| 12| 5| > | Amy|2| Amy| 12| 6| > ++-++-++ > // Okay, let's try to construct the exact same variantCount table manually > // so it has no relationship to the original. > val variantCountRowsRDD = sc.parallelize(Seq( > Row("Fred", 2), > Row(null, 2), > Row("Amy", 2))) > > val variantCountSchema = StructType(Seq( > StructField("name", StringType, nullable = true), > StructField("
[jira] [Assigned] (SPARK-20073) Unexpected Cartesian product when using eqNullSafe in join with a derived table
[ https://issues.apache.org/jira/browse/SPARK-20073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20073: Assignee: (was: Apache Spark) > Unexpected Cartesian product when using eqNullSafe in join with a derived > table > --- > > Key: SPARK-20073 > URL: https://issues.apache.org/jira/browse/SPARK-20073 > Project: Spark > Issue Type: Bug > Components: Optimizer >Affects Versions: 2.0.2, 2.1.0 >Reporter: Everett Anderson > Labels: correctness > > It appears that if you try to join tables A and B when B is derived from A > and you use the eqNullSafe / <=> operator for the join condition, Spark > performs a Cartesian product. > However, if you perform the join on tables of the same data when they don't > have a relationship, the expected non-Cartesian product join occurs. > {noformat} > // Create some fake data. > import org.apache.spark.sql.Row > import org.apache.spark.sql.Dataset > import org.apache.spark.sql.types._ > import org.apache.spark.sql.functions > val peopleRowsRDD = sc.parallelize(Seq( > Row("Fred", 8, 1), > Row("Fred", 8, 2), > Row(null, 10, 3), > Row(null, 10, 4), > Row("Amy", 12, 5), > Row("Amy", 12, 6))) > > val peopleSchema = StructType(Seq( > StructField("name", StringType, nullable = true), > StructField("group", IntegerType, nullable = true), > StructField("data", IntegerType, nullable = true))) > > val people = spark.createDataFrame(peopleRowsRDD, peopleSchema) > people.createOrReplaceTempView("people") > scala> people.show > ++-++ > |name|group|data| > ++-++ > |Fred|8| 1| > |Fred|8| 2| > |null| 10| 3| > |null| 10| 4| > | Amy| 12| 5| > | Amy| 12| 6| > ++-++ > // Now create a derived table from that table. It doesn't matter much what. > val variantCounts = spark.sql("select name, count(distinct(name, group, > data)) as variant_count from people group by name having variant_count > 1") > variantCounts.show > ++-+ > > |name|variant_count| > ++-+ > |Fred|2| > |null|2| > | Amy|2| > ++-+ > // Now try an inner join using the regular equalTo that drops nulls. This > works fine. > val innerJoinEqualTo = variantCounts.join(people, > variantCounts("name").equalTo(people("name"))) > innerJoinEqualTo.show > ++-++-++ > > |name|variant_count|name|group|data| > ++-++-++ > |Fred|2|Fred|8| 1| > |Fred|2|Fred|8| 2| > | Amy|2| Amy| 12| 5| > | Amy|2| Amy| 12| 6| > ++-++-++ > // Okay now lets switch to the <=> operator > // > // If you haven't set spark.sql.crossJoin.enabled=true, you'll get an error > like > // "Cartesian joins could be prohibitively expensive and are disabled by > default. To explicitly enable them, please set spark.sql.crossJoin.enabled = > true;" > // > // if you have enabled them, you'll get the table below. > // > // However, we really don't want or expect a Cartesian product! > val innerJoinSqlNullSafeEqOp = variantCounts.join(people, > variantCounts("name")<=>(people("name"))) > innerJoinSqlNullSafeEqOp.show > ++-++-++ > > |name|variant_count|name|group|data| > ++-++-++ > |Fred|2|Fred|8| 1| > |Fred|2|Fred|8| 2| > |Fred|2|null| 10| 3| > |Fred|2|null| 10| 4| > |Fred|2| Amy| 12| 5| > |Fred|2| Amy| 12| 6| > |null|2|Fred|8| 1| > |null|2|Fred|8| 2| > |null|2|null| 10| 3| > |null|2|null| 10| 4| > |null|2| Amy| 12| 5| > |null|2| Amy| 12| 6| > | Amy|2|Fred|8| 1| > | Amy|2|Fred|8| 2| > | Amy|2|null| 10| 3| > | Amy|2|null| 10| 4| > | Amy|2| Amy| 12| 5| > | Amy|2| Amy| 12| 6| > ++-++-++ > // Okay, let's try to construct the exact same variantCount table manually > // so it has no relationship to the original. > val variantCountRowsRDD = sc.parallelize(Seq( > Row("Fred", 2), > Row(null, 2), > Row("Amy", 2))) > > val variantCountSchema = StructType(Seq( > StructField("name", StringType, nullable = true), > StructField("variant_count", IntegerType, nullable = true))) > > val manualVariantCounts = spark.createDataFrame(variantC
[jira] [Assigned] (SPARK-20073) Unexpected Cartesian product when using eqNullSafe in join with a derived table
[ https://issues.apache.org/jira/browse/SPARK-20073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20073: Assignee: Apache Spark > Unexpected Cartesian product when using eqNullSafe in join with a derived > table > --- > > Key: SPARK-20073 > URL: https://issues.apache.org/jira/browse/SPARK-20073 > Project: Spark > Issue Type: Bug > Components: Optimizer >Affects Versions: 2.0.2, 2.1.0 >Reporter: Everett Anderson >Assignee: Apache Spark > Labels: correctness > > It appears that if you try to join tables A and B when B is derived from A > and you use the eqNullSafe / <=> operator for the join condition, Spark > performs a Cartesian product. > However, if you perform the join on tables of the same data when they don't > have a relationship, the expected non-Cartesian product join occurs. > {noformat} > // Create some fake data. > import org.apache.spark.sql.Row > import org.apache.spark.sql.Dataset > import org.apache.spark.sql.types._ > import org.apache.spark.sql.functions > val peopleRowsRDD = sc.parallelize(Seq( > Row("Fred", 8, 1), > Row("Fred", 8, 2), > Row(null, 10, 3), > Row(null, 10, 4), > Row("Amy", 12, 5), > Row("Amy", 12, 6))) > > val peopleSchema = StructType(Seq( > StructField("name", StringType, nullable = true), > StructField("group", IntegerType, nullable = true), > StructField("data", IntegerType, nullable = true))) > > val people = spark.createDataFrame(peopleRowsRDD, peopleSchema) > people.createOrReplaceTempView("people") > scala> people.show > ++-++ > |name|group|data| > ++-++ > |Fred|8| 1| > |Fred|8| 2| > |null| 10| 3| > |null| 10| 4| > | Amy| 12| 5| > | Amy| 12| 6| > ++-++ > // Now create a derived table from that table. It doesn't matter much what. > val variantCounts = spark.sql("select name, count(distinct(name, group, > data)) as variant_count from people group by name having variant_count > 1") > variantCounts.show > ++-+ > > |name|variant_count| > ++-+ > |Fred|2| > |null|2| > | Amy|2| > ++-+ > // Now try an inner join using the regular equalTo that drops nulls. This > works fine. > val innerJoinEqualTo = variantCounts.join(people, > variantCounts("name").equalTo(people("name"))) > innerJoinEqualTo.show > ++-++-++ > > |name|variant_count|name|group|data| > ++-++-++ > |Fred|2|Fred|8| 1| > |Fred|2|Fred|8| 2| > | Amy|2| Amy| 12| 5| > | Amy|2| Amy| 12| 6| > ++-++-++ > // Okay now lets switch to the <=> operator > // > // If you haven't set spark.sql.crossJoin.enabled=true, you'll get an error > like > // "Cartesian joins could be prohibitively expensive and are disabled by > default. To explicitly enable them, please set spark.sql.crossJoin.enabled = > true;" > // > // if you have enabled them, you'll get the table below. > // > // However, we really don't want or expect a Cartesian product! > val innerJoinSqlNullSafeEqOp = variantCounts.join(people, > variantCounts("name")<=>(people("name"))) > innerJoinSqlNullSafeEqOp.show > ++-++-++ > > |name|variant_count|name|group|data| > ++-++-++ > |Fred|2|Fred|8| 1| > |Fred|2|Fred|8| 2| > |Fred|2|null| 10| 3| > |Fred|2|null| 10| 4| > |Fred|2| Amy| 12| 5| > |Fred|2| Amy| 12| 6| > |null|2|Fred|8| 1| > |null|2|Fred|8| 2| > |null|2|null| 10| 3| > |null|2|null| 10| 4| > |null|2| Amy| 12| 5| > |null|2| Amy| 12| 6| > | Amy|2|Fred|8| 1| > | Amy|2|Fred|8| 2| > | Amy|2|null| 10| 3| > | Amy|2|null| 10| 4| > | Amy|2| Amy| 12| 5| > | Amy|2| Amy| 12| 6| > ++-++-++ > // Okay, let's try to construct the exact same variantCount table manually > // so it has no relationship to the original. > val variantCountRowsRDD = sc.parallelize(Seq( > Row("Fred", 2), > Row(null, 2), > Row("Amy", 2))) > > val variantCountSchema = StructType(Seq( > StructField("name", StringType, nullable = true), > StructField("variant_count", IntegerType, nullable = true))) > > val manualVariantCounts = spark
[jira] [Comment Edited] (SPARK-21218) Convert IN predicate to equivalent Parquet filter
[ https://issues.apache.org/jira/browse/SPARK-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064608#comment-16064608 ] Michael Styles edited comment on SPARK-21218 at 6/27/17 12:17 PM: -- By not pushing the filter to Parquet, are we not preventing Parquet from skipping blocks during read operations? I have tests that show big improvements when applying this transformation. For instance, I have a Parquet file with 162,456,394 rows which is sorted on column C1. *IN Predicate* {noformat} df.filter[df['C1'].isin([42, 139])).collect() {noformat} *OR Predicate* {noformat} df.filter((df['C1'] == 42) | (df['C1'] == 139)).collect() {noformat} I'm seeing about a 50 -75 % improvement. was (Author: ptkool): By not pushing the filter to Parquet, are we not preventing Parquet from skipping blocks during read operations? I have tests that show big improvements when applying this transformation. For instance, I have a Parquet file with 162,456,394 rows which is sorted on column C1. *IN Predicate* {noformat} df.filter[df['C1'].isin([42, 139])).collect() {noformat} *OR Predicate* {noformat} df.filter((df['C1'] == 42) | (df['C1'] == 139)).collect() {noformat} Notice the difference in the number of output rows for the scans (see attachments). Also, the IN predicate test took about 1.1 minutes, while the OR predicate test took about 16 seconds. > Convert IN predicate to equivalent Parquet filter > - > > Key: SPARK-21218 > URL: https://issues.apache.org/jira/browse/SPARK-21218 > Project: Spark > Issue Type: Improvement > Components: Optimizer >Affects Versions: 2.1.1 >Reporter: Michael Styles > Attachments: IN Predicate.png, OR Predicate.png > > > Convert IN predicate to equivalent expression involving equality conditions > to allow the filter to be pushed down to Parquet. > For instance, > C1 IN (10, 20) is rewritten as (C1 = 10) OR (C1 = 20) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21176) Master UI hangs with spark.ui.reverseProxy=true if the master node has many CPUs
[ https://issues.apache.org/jira/browse/SPARK-21176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064747#comment-16064747 ] Apache Spark commented on SPARK-21176: -- User 'IngoSchuster' has created a pull request for this issue: https://github.com/apache/spark/pull/18437 > Master UI hangs with spark.ui.reverseProxy=true if the master node has many > CPUs > > > Key: SPARK-21176 > URL: https://issues.apache.org/jira/browse/SPARK-21176 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.1.0, 2.1.1, 2.2.0, 2.2.1 > Environment: ppc64le GNU/Linux, POWER8, only master node is reachable > externally other nodes are in an internal network >Reporter: Ingo Schuster > Labels: network, web-ui > > In reverse proxy mode, Sparks exhausts the Jetty thread pool if the master > node has too many cpus or the cluster has too many executers: > For each ProxyServlet, Jetty creates Selector threads: minimum 4, maximum > half the number of available CPUs: > {{this(Math.max(1, Runtime.getRuntime().availableProcessors() / 2));}} > (see > https://github.com/eclipse/jetty.project/blob/0c8273f2ca1f9bf2064cd9c4c939d2546443f759/jetty-client/src/main/java/org/eclipse/jetty/client/http/HttpClientTransportOverHTTP.java) > In reverse proxy mode, a proxy servlet is set up for each executor. > I have a system with 7 executors and 88 CPUs on the master node. Jetty tries > to instantiate 7*44 = 309 selector threads just for the reverse proxy > servlets, but since the QueuedThreadPool is initialized with 200 threads by > default, the UI gets stuck. > I have patched JettyUtils.scala to extend the thread pool ( {{val pool = new > QueuedThreadPool(400)}}). With this hack, the UI works. > Obviously, the Jetty defaults are meant for a real web server. If that has 88 > CPUs, you do certainly expect a lot of traffic. > For the Spark admin UI however, there will rarely be concurrent accesses for > the same application or the same executor. > I therefore propose to dramatically reduce the number of selector threads > that get instantiated - at least by default. > I will propose a fix in a pull request. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-21176) Master UI hangs with spark.ui.reverseProxy=true if the master node has many CPUs
[ https://issues.apache.org/jira/browse/SPARK-21176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21176: Assignee: (was: Apache Spark) > Master UI hangs with spark.ui.reverseProxy=true if the master node has many > CPUs > > > Key: SPARK-21176 > URL: https://issues.apache.org/jira/browse/SPARK-21176 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.1.0, 2.1.1, 2.2.0, 2.2.1 > Environment: ppc64le GNU/Linux, POWER8, only master node is reachable > externally other nodes are in an internal network >Reporter: Ingo Schuster > Labels: network, web-ui > > In reverse proxy mode, Sparks exhausts the Jetty thread pool if the master > node has too many cpus or the cluster has too many executers: > For each ProxyServlet, Jetty creates Selector threads: minimum 4, maximum > half the number of available CPUs: > {{this(Math.max(1, Runtime.getRuntime().availableProcessors() / 2));}} > (see > https://github.com/eclipse/jetty.project/blob/0c8273f2ca1f9bf2064cd9c4c939d2546443f759/jetty-client/src/main/java/org/eclipse/jetty/client/http/HttpClientTransportOverHTTP.java) > In reverse proxy mode, a proxy servlet is set up for each executor. > I have a system with 7 executors and 88 CPUs on the master node. Jetty tries > to instantiate 7*44 = 309 selector threads just for the reverse proxy > servlets, but since the QueuedThreadPool is initialized with 200 threads by > default, the UI gets stuck. > I have patched JettyUtils.scala to extend the thread pool ( {{val pool = new > QueuedThreadPool(400)}}). With this hack, the UI works. > Obviously, the Jetty defaults are meant for a real web server. If that has 88 > CPUs, you do certainly expect a lot of traffic. > For the Spark admin UI however, there will rarely be concurrent accesses for > the same application or the same executor. > I therefore propose to dramatically reduce the number of selector threads > that get instantiated - at least by default. > I will propose a fix in a pull request. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-21176) Master UI hangs with spark.ui.reverseProxy=true if the master node has many CPUs
[ https://issues.apache.org/jira/browse/SPARK-21176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21176: Assignee: Apache Spark > Master UI hangs with spark.ui.reverseProxy=true if the master node has many > CPUs > > > Key: SPARK-21176 > URL: https://issues.apache.org/jira/browse/SPARK-21176 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.1.0, 2.1.1, 2.2.0, 2.2.1 > Environment: ppc64le GNU/Linux, POWER8, only master node is reachable > externally other nodes are in an internal network >Reporter: Ingo Schuster >Assignee: Apache Spark > Labels: network, web-ui > > In reverse proxy mode, Sparks exhausts the Jetty thread pool if the master > node has too many cpus or the cluster has too many executers: > For each ProxyServlet, Jetty creates Selector threads: minimum 4, maximum > half the number of available CPUs: > {{this(Math.max(1, Runtime.getRuntime().availableProcessors() / 2));}} > (see > https://github.com/eclipse/jetty.project/blob/0c8273f2ca1f9bf2064cd9c4c939d2546443f759/jetty-client/src/main/java/org/eclipse/jetty/client/http/HttpClientTransportOverHTTP.java) > In reverse proxy mode, a proxy servlet is set up for each executor. > I have a system with 7 executors and 88 CPUs on the master node. Jetty tries > to instantiate 7*44 = 309 selector threads just for the reverse proxy > servlets, but since the QueuedThreadPool is initialized with 200 threads by > default, the UI gets stuck. > I have patched JettyUtils.scala to extend the thread pool ( {{val pool = new > QueuedThreadPool(400)}}). With this hack, the UI works. > Obviously, the Jetty defaults are meant for a real web server. If that has 88 > CPUs, you do certainly expect a lot of traffic. > For the Spark admin UI however, there will rarely be concurrent accesses for > the same application or the same executor. > I therefore propose to dramatically reduce the number of selector threads > that get instantiated - at least by default. > I will propose a fix in a pull request. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-21227) Unicode in Json field causes AnalysisException when selecting from Dataframe
Seydou Dia created SPARK-21227: -- Summary: Unicode in Json field causes AnalysisException when selecting from Dataframe Key: SPARK-21227 URL: https://issues.apache.org/jira/browse/SPARK-21227 Project: Spark Issue Type: Bug Components: PySpark Affects Versions: 2.1.0 Reporter: Seydou Dia Hi, please find below the step to reproduce the issue I am facing, $ pyspark {code:python} Python 3.4.3 (default, Sep 1 2016, 23:33:38) [GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux Type "help", "copyright", "credits" or "license" for more information. Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 17/06/27 12:29:05 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041. 17/06/27 12:29:05 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042. 17/06/27 12:29:08 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. Welcome to __ / __/__ ___ _/ /__ _\ \/ _ \/ _ `/ __/ '_/ /__ / .__/\_,_/_/ /_/\_\ version 2.1.0 /_/ Using Python version 3.4.3 (default, Sep 1 2016 23:33:38) SparkSession available as 'spark'. >>> sc=spark.sparkContext >>> js = ['{"city_name": "paris"}' ... , '{"city_name": "rome"}' ... , '{"city_name": "berlin"}' ... , '{"cıty_name": "new-york"}' ... , '{"cıty_name": "toronto"}' ... , '{"cıty_name": "chicago"}' ... , '{"cıty_name": "dubai"}'] >>> myRDD = sc.parallelize(js) >>> myDF = spark.read.json(myRDD) >>> myDF.printSchema() root |-- city_name: string (nullable = true) |-- cıty_name: string (nullable = true) >>> myDF.select(myDF['city_name']) Traceback (most recent call last): File "/usr/lib/spark/python/pyspark/sql/utils.py", line 63, in deco return f(*a, **kw) File "/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling o226.apply. : org.apache.spark.sql.AnalysisException: Reference 'city_name' is ambiguous, could be: city_name#29, city_name#30.; at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolve(LogicalPlan.scala:264) at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveQuoted(LogicalPlan.scala:168) at org.apache.spark.sql.Dataset.resolve(Dataset.scala:217) at org.apache.spark.sql.Dataset.col(Dataset.scala:1073) at org.apache.spark.sql.Dataset.apply(Dataset.scala:1059) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:280) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:214) at java.lang.Thread.run(Thread.java:745) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "", line 1, in File "/usr/lib/spark/python/pyspark/sql/dataframe.py", line 943, in __getitem__ jc = self._jdf.apply(item) File "/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__ File "/usr/lib/spark/python/pyspark/sql/utils.py", line 69, in deco raise AnalysisException(s.split(': ', 1)[1], stackTrace) pyspark.sql.utils.AnalysisException: "Reference 'city_name' is ambiguous, could be: city_name#29, city_name#30.;" {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-21227) Unicode in Json field causes AnalysisException when selecting from Dataframe
[ https://issues.apache.org/jira/browse/SPARK-21227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seydou Dia updated SPARK-21227: --- Description: Hi, please find below the step to reproduce the issue I am facing, {code:python} $ pyspark Python 3.4.3 (default, Sep 1 2016, 23:33:38) [GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux Type "help", "copyright", "credits" or "license" for more information. Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 17/06/27 12:29:05 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041. 17/06/27 12:29:05 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042. 17/06/27 12:29:08 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. Welcome to __ / __/__ ___ _/ /__ _\ \/ _ \/ _ `/ __/ '_/ /__ / .__/\_,_/_/ /_/\_\ version 2.1.0 /_/ Using Python version 3.4.3 (default, Sep 1 2016 23:33:38) SparkSession available as 'spark'. >>> sc=spark.sparkContext >>> js = ['{"city_name": "paris"}' ... , '{"city_name": "rome"}' ... , '{"city_name": "berlin"}' ... , '{"cıty_name": "new-york"}' ... , '{"cıty_name": "toronto"}' ... , '{"cıty_name": "chicago"}' ... , '{"cıty_name": "dubai"}'] >>> myRDD = sc.parallelize(js) >>> myDF = spark.read.json(myRDD) >>> myDF.printSchema() root |-- city_name: string (nullable = true) |-- cıty_name: string (nullable = true) >>> myDF.select(myDF['city_name']) Traceback (most recent call last): File "/usr/lib/spark/python/pyspark/sql/utils.py", line 63, in deco return f(*a, **kw) File "/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling o226.apply. : org.apache.spark.sql.AnalysisException: Reference 'city_name' is ambiguous, could be: city_name#29, city_name#30.; at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolve(LogicalPlan.scala:264) at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveQuoted(LogicalPlan.scala:168) at org.apache.spark.sql.Dataset.resolve(Dataset.scala:217) at org.apache.spark.sql.Dataset.col(Dataset.scala:1073) at org.apache.spark.sql.Dataset.apply(Dataset.scala:1059) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:280) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:214) at java.lang.Thread.run(Thread.java:745) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "", line 1, in File "/usr/lib/spark/python/pyspark/sql/dataframe.py", line 943, in __getitem__ jc = self._jdf.apply(item) File "/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__ File "/usr/lib/spark/python/pyspark/sql/utils.py", line 69, in deco raise AnalysisException(s.split(': ', 1)[1], stackTrace) pyspark.sql.utils.AnalysisException: "Reference 'city_name' is ambiguous, could be: city_name#29, city_name#30.;" {code} was: Hi, please find below the step to reproduce the issue I am facing, $ pyspark {code:python} Python 3.4.3 (default, Sep 1 2016, 23:33:38) [GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux Type "help", "copyright", "credits" or "license" for more information. Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 17/06/27 12:29:05 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041. 17/06/27 12:29:05 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042. 17/06/27 12:29:08 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. Welcome to __ / __/__ ___ _/ /__ _\ \/ _ \/ _ `/ __/ '_/ /__ / .__/\_,_/_/ /_/\_\ version 2.1.0 /_/ Using Python version 3.4.3 (default, Sep 1 2016 23:33:38) SparkSession available as 'spark'. >>> sc=spark.sparkContext >>> js = ['{"city_name": "paris"}' ... ,
[jira] [Updated] (SPARK-21227) Unicode in Json field causes AnalysisException when selecting from Dataframe
[ https://issues.apache.org/jira/browse/SPARK-21227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seydou Dia updated SPARK-21227: --- Description: Hi, please find below the step to reproduce the issue I am facing. First I create a json with 2 fields: * city_name * cıty_name The first one is valid ascii, while the second contains a unicode (ı, i without dot ). When I try to select from the dataframe I have an {noformat} AnalysisException {noformat}. {code:python} $ pyspark Python 3.4.3 (default, Sep 1 2016, 23:33:38) [GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux Type "help", "copyright", "credits" or "license" for more information. Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 17/06/27 12:29:05 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041. 17/06/27 12:29:05 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042. 17/06/27 12:29:08 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. Welcome to __ / __/__ ___ _/ /__ _\ \/ _ \/ _ `/ __/ '_/ /__ / .__/\_,_/_/ /_/\_\ version 2.1.0 /_/ Using Python version 3.4.3 (default, Sep 1 2016 23:33:38) SparkSession available as 'spark'. >>> sc=spark.sparkContext >>> js = ['{"city_name": "paris"}' ... , '{"city_name": "rome"}' ... , '{"city_name": "berlin"}' ... , '{"cıty_name": "new-york"}' ... , '{"cıty_name": "toronto"}' ... , '{"cıty_name": "chicago"}' ... , '{"cıty_name": "dubai"}'] >>> myRDD = sc.parallelize(js) >>> myDF = spark.read.json(myRDD) >>> myDF.printSchema() root |-- city_name: string (nullable = true) |-- cıty_name: string (nullable = true) >>> myDF.select(myDF['city_name']) Traceback (most recent call last): File "/usr/lib/spark/python/pyspark/sql/utils.py", line 63, in deco return f(*a, **kw) File "/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling o226.apply. : org.apache.spark.sql.AnalysisException: Reference 'city_name' is ambiguous, could be: city_name#29, city_name#30.; at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolve(LogicalPlan.scala:264) at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveQuoted(LogicalPlan.scala:168) at org.apache.spark.sql.Dataset.resolve(Dataset.scala:217) at org.apache.spark.sql.Dataset.col(Dataset.scala:1073) at org.apache.spark.sql.Dataset.apply(Dataset.scala:1059) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:280) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:214) at java.lang.Thread.run(Thread.java:745) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "", line 1, in File "/usr/lib/spark/python/pyspark/sql/dataframe.py", line 943, in __getitem__ jc = self._jdf.apply(item) File "/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__ File "/usr/lib/spark/python/pyspark/sql/utils.py", line 69, in deco raise AnalysisException(s.split(': ', 1)[1], stackTrace) pyspark.sql.utils.AnalysisException: "Reference 'city_name' is ambiguous, could be: city_name#29, city_name#30.;" {code} was: Hi, please find below the step to reproduce the issue I am facing, {code:python} $ pyspark Python 3.4.3 (default, Sep 1 2016, 23:33:38) [GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux Type "help", "copyright", "credits" or "license" for more information. Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 17/06/27 12:29:05 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041. 17/06/27 12:29:05 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042. 17/06/27 12:29:08 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. Welcome to __ / __/__ ___ _/ /__
[jira] [Commented] (SPARK-21067) Thrift Server - CTAS fail with Unable to move source
[ https://issues.apache.org/jira/browse/SPARK-21067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064833#comment-16064833 ] Dominic Ricard commented on SPARK-21067: [~q79969786], yes. As stated in the description, ours is set to "/tmp/hive-staging/{user.name}" > Thrift Server - CTAS fail with Unable to move source > > > Key: SPARK-21067 > URL: https://issues.apache.org/jira/browse/SPARK-21067 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.1 > Environment: Yarn > Hive MetaStore > HDFS (HA) >Reporter: Dominic Ricard > > After upgrading our Thrift cluster to 2.1.1, we ran into an issue where CTAS > would fail, sometimes... > Most of the time, the CTAS would work only once, after starting the thrift > server. After that, dropping the table and re-issuing the same CTAS would > fail with the following message (Sometime, it fails right away, sometime it > work for a long period of time): > {noformat} > Error: org.apache.spark.sql.AnalysisException: > org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source > hdfs://nameservice1//tmp/hive-staging/thrift_hive_2017-06-12_16-56-18_464_7598877199323198104-31/-ext-1/part-0 > to destination > hdfs://nameservice1/user/hive/warehouse/dricard.db/test/part-0; > (state=,code=0) > {noformat} > We have already found the following Jira > (https://issues.apache.org/jira/browse/SPARK-11021) which state that the > {{hive.exec.stagingdir}} had to be added in order for Spark to be able to > handle CREATE TABLE properly as of 2.0. As you can see in the error, we have > ours set to "/tmp/hive-staging/\{user.name\}" > Same issue with INSERT statements: > {noformat} > CREATE TABLE IF NOT EXISTS dricard.test (col1 int); INSERT INTO TABLE > dricard.test SELECT 1; > Error: org.apache.spark.sql.AnalysisException: > org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source > hdfs://nameservice1/tmp/hive-staging/thrift_hive_2017-06-12_20-41-12_964_3086448130033637241-16/-ext-1/part-0 > to destination > hdfs://nameservice1/user/hive/warehouse/dricard.db/test/part-0; > (state=,code=0) > {noformat} > This worked fine in 1.6.2, which we currently run in our Production > Environment but since 2.0+, we haven't been able to CREATE TABLE consistently > on the cluster. > SQL to reproduce issue: > {noformat} > DROP SCHEMA IF EXISTS dricard CASCADE; > CREATE SCHEMA dricard; > CREATE TABLE dricard.test (col1 int); > INSERT INTO TABLE dricard.test SELECT 1; > SELECT * from dricard.test; > DROP TABLE dricard.test; > CREATE TABLE dricard.test AS select 1 as `col1`; > SELECT * from dricard.test > {noformat} > Thrift server usually fails at INSERT... > Tried the same procedure in a spark context using spark.sql() and didn't > encounter the same issue. > Full stack Trace: > {noformat} > 17/06/14 14:52:18 ERROR thriftserver.SparkExecuteStatementOperation: Error > executing query, currentState RUNNING, > org.apache.spark.sql.AnalysisException: > org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source > hdfs://nameservice1/tmp/hive-staging/thrift_hive_2017-06-14_14-52-18_521_5906917519254880890-5/-ext-1/part-0 > to desti > nation hdfs://nameservice1/user/hive/warehouse/dricard.db/test/part-0; > at > org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106) > at > org.apache.spark.sql.hive.HiveExternalCatalog.loadTable(HiveExternalCatalog.scala:766) > at > org.apache.spark.sql.hive.execution.InsertIntoHiveTable.sideEffectResult$lzycompute(InsertIntoHiveTable.scala:374) > at > org.apache.spark.sql.hive.execution.InsertIntoHiveTable.sideEffectResult(InsertIntoHiveTable.scala:221) > at > org.apache.spark.sql.hive.execution.InsertIntoHiveTable.doExecute(InsertIntoHiveTable.scala:407) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135) > at > org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) > at > org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132) > at > org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113) > at > org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92) > at > org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92) > at org.apache.spark.sql.Dataset.(Dataset.scala:185) > at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala
[jira] [Comment Edited] (SPARK-21067) Thrift Server - CTAS fail with Unable to move source
[ https://issues.apache.org/jira/browse/SPARK-21067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064833#comment-16064833 ] Dominic Ricard edited comment on SPARK-21067 at 6/27/17 1:31 PM: - [~q79969786], yes. As stated in the description, ours is set to "/tmp/hive-staging/\{user.name\}" was (Author: dricard): [~q79969786], yes. As stated in the description, ours is set to "/tmp/hive-staging/{user.name}" > Thrift Server - CTAS fail with Unable to move source > > > Key: SPARK-21067 > URL: https://issues.apache.org/jira/browse/SPARK-21067 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.1 > Environment: Yarn > Hive MetaStore > HDFS (HA) >Reporter: Dominic Ricard > > After upgrading our Thrift cluster to 2.1.1, we ran into an issue where CTAS > would fail, sometimes... > Most of the time, the CTAS would work only once, after starting the thrift > server. After that, dropping the table and re-issuing the same CTAS would > fail with the following message (Sometime, it fails right away, sometime it > work for a long period of time): > {noformat} > Error: org.apache.spark.sql.AnalysisException: > org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source > hdfs://nameservice1//tmp/hive-staging/thrift_hive_2017-06-12_16-56-18_464_7598877199323198104-31/-ext-1/part-0 > to destination > hdfs://nameservice1/user/hive/warehouse/dricard.db/test/part-0; > (state=,code=0) > {noformat} > We have already found the following Jira > (https://issues.apache.org/jira/browse/SPARK-11021) which state that the > {{hive.exec.stagingdir}} had to be added in order for Spark to be able to > handle CREATE TABLE properly as of 2.0. As you can see in the error, we have > ours set to "/tmp/hive-staging/\{user.name\}" > Same issue with INSERT statements: > {noformat} > CREATE TABLE IF NOT EXISTS dricard.test (col1 int); INSERT INTO TABLE > dricard.test SELECT 1; > Error: org.apache.spark.sql.AnalysisException: > org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source > hdfs://nameservice1/tmp/hive-staging/thrift_hive_2017-06-12_20-41-12_964_3086448130033637241-16/-ext-1/part-0 > to destination > hdfs://nameservice1/user/hive/warehouse/dricard.db/test/part-0; > (state=,code=0) > {noformat} > This worked fine in 1.6.2, which we currently run in our Production > Environment but since 2.0+, we haven't been able to CREATE TABLE consistently > on the cluster. > SQL to reproduce issue: > {noformat} > DROP SCHEMA IF EXISTS dricard CASCADE; > CREATE SCHEMA dricard; > CREATE TABLE dricard.test (col1 int); > INSERT INTO TABLE dricard.test SELECT 1; > SELECT * from dricard.test; > DROP TABLE dricard.test; > CREATE TABLE dricard.test AS select 1 as `col1`; > SELECT * from dricard.test > {noformat} > Thrift server usually fails at INSERT... > Tried the same procedure in a spark context using spark.sql() and didn't > encounter the same issue. > Full stack Trace: > {noformat} > 17/06/14 14:52:18 ERROR thriftserver.SparkExecuteStatementOperation: Error > executing query, currentState RUNNING, > org.apache.spark.sql.AnalysisException: > org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source > hdfs://nameservice1/tmp/hive-staging/thrift_hive_2017-06-14_14-52-18_521_5906917519254880890-5/-ext-1/part-0 > to desti > nation hdfs://nameservice1/user/hive/warehouse/dricard.db/test/part-0; > at > org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106) > at > org.apache.spark.sql.hive.HiveExternalCatalog.loadTable(HiveExternalCatalog.scala:766) > at > org.apache.spark.sql.hive.execution.InsertIntoHiveTable.sideEffectResult$lzycompute(InsertIntoHiveTable.scala:374) > at > org.apache.spark.sql.hive.execution.InsertIntoHiveTable.sideEffectResult(InsertIntoHiveTable.scala:221) > at > org.apache.spark.sql.hive.execution.InsertIntoHiveTable.doExecute(InsertIntoHiveTable.scala:407) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135) > at > org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) > at > org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132) > at > org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113) > at > org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92) > at > org.apache.spark.sql.executio
[jira] [Created] (SPARK-21228) InSet.doCodeGen incorrect handling of structs
Bogdan Raducanu created SPARK-21228: --- Summary: InSet.doCodeGen incorrect handling of structs Key: SPARK-21228 URL: https://issues.apache.org/jira/browse/SPARK-21228 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.2.0 Reporter: Bogdan Raducanu In InSet it's possible that hset contains GenericInternalRows while child returns UnsafeRows (and vice versa). InSet.doCodeGen uses hset.contains which will always be false in this case. The following code reproduces the problem: ``` spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "2") // the default is 10 which requires a longer query text to repro spark.range(1, 10).selectExpr("named_struct('a', id, 'b', id) as a").createOrReplaceTempView("A") sql("select * from (select min(a) as minA from A) A where minA in (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', 2L),named_struct('a', 3L, 'b', 3L))").show ++ |minA| ++ ++ ``` In.doCodeGen appears to be correct: ``` spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "3") // now it will not use InSet +-+ | minA| +-+ |[1,1]| +-+ ``` Solution could be either to do safe<->unsafe conversion in InSet.doCodeGen or not trigger InSet optimization at all in this case. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-21228) InSet.doCodeGen incorrect handling of structs
[ https://issues.apache.org/jira/browse/SPARK-21228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bogdan Raducanu updated SPARK-21228: Description: In InSet it's possible that hset contains GenericInternalRows while child returns UnsafeRows (and vice versa). InSet.doCodeGen uses hset.contains which will always be false in this case. The following code reproduces the problem: {code} spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "2") // the default is 10 which requires a longer query text to repro spark.range(1, 10).selectExpr("named_struct('a', id, 'b', id) as a").createOrReplaceTempView("A") sql("select * from (select min(a) as minA from A) A where minA in (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', 2L),named_struct('a', 3L, 'b', 3L))").show ++ |minA| ++ ++ {code} In.doCodeGen appears to be correct: {code} spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "3") // now it will not use InSet +-+ | minA| +-+ |[1,1]| +-+ {code} Solution could be either to do safe<->unsafe conversion in InSet.doCodeGen or not trigger InSet optimization at all in this case. was: In InSet it's possible that hset contains GenericInternalRows while child returns UnsafeRows (and vice versa). InSet.doCodeGen uses hset.contains which will always be false in this case. The following code reproduces the problem: ``` spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "2") // the default is 10 which requires a longer query text to repro spark.range(1, 10).selectExpr("named_struct('a', id, 'b', id) as a").createOrReplaceTempView("A") sql("select * from (select min(a) as minA from A) A where minA in (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', 2L),named_struct('a', 3L, 'b', 3L))").show ++ |minA| ++ ++ ``` In.doCodeGen appears to be correct: ``` spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "3") // now it will not use InSet +-+ | minA| +-+ |[1,1]| +-+ ``` Solution could be either to do safe<->unsafe conversion in InSet.doCodeGen or not trigger InSet optimization at all in this case. > InSet.doCodeGen incorrect handling of structs > - > > Key: SPARK-21228 > URL: https://issues.apache.org/jira/browse/SPARK-21228 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.2.0 >Reporter: Bogdan Raducanu > > In InSet it's possible that hset contains GenericInternalRows while child > returns UnsafeRows (and vice versa). InSet.doCodeGen uses hset.contains which > will always be false in this case. > The following code reproduces the problem: > {code} > spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "2") // the > default is 10 which requires a longer query text to repro > spark.range(1, 10).selectExpr("named_struct('a', id, 'b', id) as > a").createOrReplaceTempView("A") > sql("select * from (select min(a) as minA from A) A where minA in > (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', > 2L),named_struct('a', 3L, 'b', 3L))").show > ++ > |minA| > ++ > ++ > {code} > In.doCodeGen appears to be correct: > {code} > spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "3") // now it > will not use InSet > +-+ > | minA| > +-+ > |[1,1]| > +-+ > {code} > Solution could be either to do safe<->unsafe conversion in InSet.doCodeGen or > not trigger InSet optimization at all in this case. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-21228) InSet.doCodeGen incorrect handling of structs
[ https://issues.apache.org/jira/browse/SPARK-21228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bogdan Raducanu updated SPARK-21228: Description: In InSet it's possible that hset contains GenericInternalRows while child returns UnsafeRows (and vice versa). InSet.doCodeGen uses hset.contains which will always be false in this case. The following code reproduces the problem: {code} spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "2") // the default is 10 which requires a longer query text to repro spark.range(1, 10).selectExpr("named_struct('a', id, 'b', id) as a").createOrReplaceTempView("A") sql("select * from (select min(a) as minA from A) A where minA in (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', 2L),named_struct('a', 3L, 'b', 3L))").show ++ |minA| ++ ++ {code} In.doCodeGen appears to be correct: {code} spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "3") // now it will not use InSet sql("select * from (select min(a) as minA from A) A where minA in (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', 2L),named_struct('a', 3L, 'b', 3L))").show +-+ | minA| +-+ |[1,1]| +-+ {code} Solution could be either to do safe<->unsafe conversion in InSet.doCodeGen or not trigger InSet optimization at all in this case. was: In InSet it's possible that hset contains GenericInternalRows while child returns UnsafeRows (and vice versa). InSet.doCodeGen uses hset.contains which will always be false in this case. The following code reproduces the problem: {code} spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "2") // the default is 10 which requires a longer query text to repro spark.range(1, 10).selectExpr("named_struct('a', id, 'b', id) as a").createOrReplaceTempView("A") sql("select * from (select min(a) as minA from A) A where minA in (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', 2L),named_struct('a', 3L, 'b', 3L))").show ++ |minA| ++ ++ {code} In.doCodeGen appears to be correct: {code} spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "3") // now it will not use InSet +-+ | minA| +-+ |[1,1]| +-+ {code} Solution could be either to do safe<->unsafe conversion in InSet.doCodeGen or not trigger InSet optimization at all in this case. > InSet.doCodeGen incorrect handling of structs > - > > Key: SPARK-21228 > URL: https://issues.apache.org/jira/browse/SPARK-21228 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.2.0 >Reporter: Bogdan Raducanu > > In InSet it's possible that hset contains GenericInternalRows while child > returns UnsafeRows (and vice versa). InSet.doCodeGen uses hset.contains which > will always be false in this case. > The following code reproduces the problem: > {code} > spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "2") // the > default is 10 which requires a longer query text to repro > spark.range(1, 10).selectExpr("named_struct('a', id, 'b', id) as > a").createOrReplaceTempView("A") > sql("select * from (select min(a) as minA from A) A where minA in > (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', > 2L),named_struct('a', 3L, 'b', 3L))").show > ++ > |minA| > ++ > ++ > {code} > In.doCodeGen appears to be correct: > {code} > spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "3") // now it > will not use InSet > sql("select * from (select min(a) as minA from A) A where minA in > (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', > 2L),named_struct('a', 3L, 'b', 3L))").show > +-+ > | minA| > +-+ > |[1,1]| > +-+ > {code} > Solution could be either to do safe<->unsafe conversion in InSet.doCodeGen or > not trigger InSet optimization at all in this case. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-21228) InSet.doCodeGen incorrect handling of structs
[ https://issues.apache.org/jira/browse/SPARK-21228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bogdan Raducanu updated SPARK-21228: Description: In InSet it's possible that hset contains GenericInternalRows while child returns UnsafeRows (and vice versa). InSet.doCodeGen uses hset.contains which will always be false in this case. The following code reproduces the problem: {code} spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "2") // the default is 10 which requires a longer query text to repro spark.range(1, 10).selectExpr("named_struct('a', id, 'b', id) as a").createOrReplaceTempView("A") sql("select * from (select min(a) as minA from A) A where minA in (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', 2L),named_struct('a', 3L, 'b', 3L))").show -- the Aggregate here will return UnsafeRows while the list of structs that will become hset will be GenericInternalRows ++ |minA| ++ ++ {code} In.doCodeGen appears to be correct: {code} spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "3") // now it will not use InSet sql("select * from (select min(a) as minA from A) A where minA in (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', 2L),named_struct('a', 3L, 'b', 3L))").show +-+ | minA| +-+ |[1,1]| +-+ {code} Solution could be either to do safe<->unsafe conversion in InSet.doCodeGen or not trigger InSet optimization at all in this case. was: In InSet it's possible that hset contains GenericInternalRows while child returns UnsafeRows (and vice versa). InSet.doCodeGen uses hset.contains which will always be false in this case. The following code reproduces the problem: {code} spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "2") // the default is 10 which requires a longer query text to repro spark.range(1, 10).selectExpr("named_struct('a', id, 'b', id) as a").createOrReplaceTempView("A") sql("select * from (select min(a) as minA from A) A where minA in (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', 2L),named_struct('a', 3L, 'b', 3L))").show ++ |minA| ++ ++ {code} In.doCodeGen appears to be correct: {code} spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "3") // now it will not use InSet sql("select * from (select min(a) as minA from A) A where minA in (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', 2L),named_struct('a', 3L, 'b', 3L))").show +-+ | minA| +-+ |[1,1]| +-+ {code} Solution could be either to do safe<->unsafe conversion in InSet.doCodeGen or not trigger InSet optimization at all in this case. > InSet.doCodeGen incorrect handling of structs > - > > Key: SPARK-21228 > URL: https://issues.apache.org/jira/browse/SPARK-21228 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.2.0 >Reporter: Bogdan Raducanu > > In InSet it's possible that hset contains GenericInternalRows while child > returns UnsafeRows (and vice versa). InSet.doCodeGen uses hset.contains which > will always be false in this case. > The following code reproduces the problem: > {code} > spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "2") // the > default is 10 which requires a longer query text to repro > spark.range(1, 10).selectExpr("named_struct('a', id, 'b', id) as > a").createOrReplaceTempView("A") > sql("select * from (select min(a) as minA from A) A where minA in > (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', > 2L),named_struct('a', 3L, 'b', 3L))").show -- the Aggregate here will return > UnsafeRows while the list of structs that will become hset will be > GenericInternalRows > ++ > |minA| > ++ > ++ > {code} > In.doCodeGen appears to be correct: > {code} > spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "3") // now it > will not use InSet > sql("select * from (select min(a) as minA from A) A where minA in > (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', > 2L),named_struct('a', 3L, 'b', 3L))").show > +-+ > | minA| > +-+ > |[1,1]| > +-+ > {code} > Solution could be either to do safe<->unsafe conversion in InSet.doCodeGen or > not trigger InSet optimization at all in this case. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-21228) InSet.doCodeGen incorrect handling of structs
[ https://issues.apache.org/jira/browse/SPARK-21228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bogdan Raducanu updated SPARK-21228: Description: In InSet it's possible that hset contains GenericInternalRows while child returns UnsafeRows (and vice versa). InSet.doCodeGen uses hset.contains which will always be false in this case. The following code reproduces the problem: {code} spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "2") // the default is 10 which requires a longer query text to repro spark.range(1, 10).selectExpr("named_struct('a', id, 'b', id) as a").createOrReplaceTempView("A") sql("select * from (select min(a) as minA from A) A where minA in (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', 2L),named_struct('a', 3L, 'b', 3L))").show // the Aggregate here will return UnsafeRows while the list of structs that will become hset will be GenericInternalRows ++ |minA| ++ ++ {code} In.doCodeGen appears to be correct: {code} spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "3") // now it will not use InSet sql("select * from (select min(a) as minA from A) A where minA in (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', 2L),named_struct('a', 3L, 'b', 3L))").show +-+ | minA| +-+ |[1,1]| +-+ {code} Solution could be either to do safe<->unsafe conversion in InSet.doCodeGen or not trigger InSet optimization at all in this case. was: In InSet it's possible that hset contains GenericInternalRows while child returns UnsafeRows (and vice versa). InSet.doCodeGen uses hset.contains which will always be false in this case. The following code reproduces the problem: {code} spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "2") // the default is 10 which requires a longer query text to repro spark.range(1, 10).selectExpr("named_struct('a', id, 'b', id) as a").createOrReplaceTempView("A") sql("select * from (select min(a) as minA from A) A where minA in (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', 2L),named_struct('a', 3L, 'b', 3L))").show -- the Aggregate here will return UnsafeRows while the list of structs that will become hset will be GenericInternalRows ++ |minA| ++ ++ {code} In.doCodeGen appears to be correct: {code} spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "3") // now it will not use InSet sql("select * from (select min(a) as minA from A) A where minA in (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', 2L),named_struct('a', 3L, 'b', 3L))").show +-+ | minA| +-+ |[1,1]| +-+ {code} Solution could be either to do safe<->unsafe conversion in InSet.doCodeGen or not trigger InSet optimization at all in this case. > InSet.doCodeGen incorrect handling of structs > - > > Key: SPARK-21228 > URL: https://issues.apache.org/jira/browse/SPARK-21228 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.2.0 >Reporter: Bogdan Raducanu > > In InSet it's possible that hset contains GenericInternalRows while child > returns UnsafeRows (and vice versa). InSet.doCodeGen uses hset.contains which > will always be false in this case. > The following code reproduces the problem: > {code} > spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "2") // the > default is 10 which requires a longer query text to repro > spark.range(1, 10).selectExpr("named_struct('a', id, 'b', id) as > a").createOrReplaceTempView("A") > sql("select * from (select min(a) as minA from A) A where minA in > (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', > 2L),named_struct('a', 3L, 'b', 3L))").show // the Aggregate here will return > UnsafeRows while the list of structs that will become hset will be > GenericInternalRows > ++ > |minA| > ++ > ++ > {code} > In.doCodeGen appears to be correct: > {code} > spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "3") // now it > will not use InSet > sql("select * from (select min(a) as minA from A) A where minA in > (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', > 2L),named_struct('a', 3L, 'b', 3L))").show > +-+ > | minA| > +-+ > |[1,1]| > +-+ > {code} > Solution could be either to do safe<->unsafe conversion in InSet.doCodeGen or > not trigger InSet optimization at all in this case. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-21228) InSet.doCodeGen incorrect handling of structs
[ https://issues.apache.org/jira/browse/SPARK-21228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bogdan Raducanu updated SPARK-21228: Description: In InSet it's possible that hset contains GenericInternalRows while child returns UnsafeRows (and vice versa). InSet.doCodeGen uses hset.contains which will always be false in this case. The following code reproduces the problem: {code} spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "2") // the default is 10 which requires a longer query text to repro spark.range(1, 10).selectExpr("named_struct('a', id, 'b', id) as a").createOrReplaceTempView("A") sql("select * from (select min(a) as minA from A) A where minA in (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', 2L),named_struct('a', 3L, 'b', 3L))").show // the Aggregate here will return UnsafeRows while the list of structs that will become hset will be GenericInternalRows ++ |minA| ++ ++ {code} In.doCodeGen appears to be correct: {code} spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "3") // now it will not use InSet sql("select * from (select min(a) as minA from A) A where minA in (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', 2L),named_struct('a', 3L, 'b', 3L))").show +-+ | minA| +-+ |[1,1]| +-+ {code} Solution could be either to do safe<->unsafe conversion in InSet.doCodeGen or not trigger InSet optimization at all in this case. was: In InSet it's possible that hset contains GenericInternalRows while child returns UnsafeRows (and vice versa). InSet.doCodeGen uses hset.contains which will always be false in this case. The following code reproduces the problem: {code} spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "2") // the default is 10 which requires a longer query text to repro spark.range(1, 10).selectExpr("named_struct('a', id, 'b', id) as a").createOrReplaceTempView("A") sql("select * from (select min(a) as minA from A) A where minA in (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', 2L),named_struct('a', 3L, 'b', 3L))").show // the Aggregate here will return UnsafeRows while the list of structs that will become hset will be GenericInternalRows ++ |minA| ++ ++ {code} In.doCodeGen appears to be correct: {code} spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "3") // now it will not use InSet sql("select * from (select min(a) as minA from A) A where minA in (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', 2L),named_struct('a', 3L, 'b', 3L))").show +-+ | minA| +-+ |[1,1]| +-+ {code} Solution could be either to do safe<->unsafe conversion in InSet.doCodeGen or not trigger InSet optimization at all in this case. > InSet.doCodeGen incorrect handling of structs > - > > Key: SPARK-21228 > URL: https://issues.apache.org/jira/browse/SPARK-21228 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.2.0 >Reporter: Bogdan Raducanu > > In InSet it's possible that hset contains GenericInternalRows while child > returns UnsafeRows (and vice versa). InSet.doCodeGen uses hset.contains which > will always be false in this case. > The following code reproduces the problem: > {code} > spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "2") // the > default is 10 which requires a longer query text to repro > spark.range(1, 10).selectExpr("named_struct('a', id, 'b', id) as > a").createOrReplaceTempView("A") > sql("select * from (select min(a) as minA from A) A where minA in > (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', > 2L),named_struct('a', 3L, 'b', 3L))").show // the Aggregate here will return > UnsafeRows while the list of structs that will become hset will be > GenericInternalRows > ++ > |minA| > ++ > ++ > {code} > In.doCodeGen appears to be correct: > {code} > spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "3") // now it > will not use InSet > sql("select * from (select min(a) as minA from A) A where minA in > (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', > 2L),named_struct('a', 3L, 'b', 3L))").show > +-+ > | minA| > +-+ > |[1,1]| > +-+ > {code} > Solution could be either to do safe<->unsafe conversion in InSet.doCodeGen or > not trigger InSet optimization at all in this case. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-21228) InSet incorrect handling of structs
[ https://issues.apache.org/jira/browse/SPARK-21228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bogdan Raducanu updated SPARK-21228: Summary: InSet incorrect handling of structs (was: InSet.doCodeGen incorrect handling of structs) > InSet incorrect handling of structs > --- > > Key: SPARK-21228 > URL: https://issues.apache.org/jira/browse/SPARK-21228 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.2.0 >Reporter: Bogdan Raducanu > > In InSet it's possible that hset contains GenericInternalRows while child > returns UnsafeRows (and vice versa). InSet.doCodeGen uses hset.contains which > will always be false in this case. > The following code reproduces the problem: > {code} > spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "2") // the > default is 10 which requires a longer query text to repro > spark.range(1, 10).selectExpr("named_struct('a', id, 'b', id) as > a").createOrReplaceTempView("A") > sql("select * from (select min(a) as minA from A) A where minA in > (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', > 2L),named_struct('a', 3L, 'b', 3L))").show // the Aggregate here will return > UnsafeRows while the list of structs that will become hset will be > GenericInternalRows > ++ > |minA| > ++ > ++ > {code} > In.doCodeGen appears to be correct: > {code} > spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "3") // now it > will not use InSet > sql("select * from (select min(a) as minA from A) A where minA in > (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', > 2L),named_struct('a', 3L, 'b', 3L))").show > +-+ > | minA| > +-+ > |[1,1]| > +-+ > {code} > Solution could be either to do safe<->unsafe conversion in InSet.doCodeGen or > not trigger InSet optimization at all in this case. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-21228) InSet incorrect handling of structs
[ https://issues.apache.org/jira/browse/SPARK-21228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bogdan Raducanu updated SPARK-21228: Description: In InSet it's possible that hset contains GenericInternalRows while child returns UnsafeRows (and vice versa). InSet uses hset.contains (both in doCodeGen and eval) which will always be false in this case. The following code reproduces the problem: {code} spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "2") // the default is 10 which requires a longer query text to repro spark.range(1, 10).selectExpr("named_struct('a', id, 'b', id) as a").createOrReplaceTempView("A") sql("select * from (select min(a) as minA from A) A where minA in (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', 2L),named_struct('a', 3L, 'b', 3L))").show // the Aggregate here will return UnsafeRows while the list of structs that will become hset will be GenericInternalRows ++ |minA| ++ ++ {code} In.doCodeGen uses compareStructs and seems to work. In.eval might not work but not sure how to reproduce. {code} spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "3") // now it will not use InSet sql("select * from (select min(a) as minA from A) A where minA in (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', 2L),named_struct('a', 3L, 'b', 3L))").show +-+ | minA| +-+ |[1,1]| +-+ {code} Solution could be either to do safe<->unsafe conversion in InSet or not trigger InSet optimization at all in this case. Need to investigate if In.eval is affected. was: In InSet it's possible that hset contains GenericInternalRows while child returns UnsafeRows (and vice versa). InSet.doCodeGen uses hset.contains which will always be false in this case. The following code reproduces the problem: {code} spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "2") // the default is 10 which requires a longer query text to repro spark.range(1, 10).selectExpr("named_struct('a', id, 'b', id) as a").createOrReplaceTempView("A") sql("select * from (select min(a) as minA from A) A where minA in (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', 2L),named_struct('a', 3L, 'b', 3L))").show // the Aggregate here will return UnsafeRows while the list of structs that will become hset will be GenericInternalRows ++ |minA| ++ ++ {code} In.doCodeGen appears to be correct: {code} spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "3") // now it will not use InSet sql("select * from (select min(a) as minA from A) A where minA in (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', 2L),named_struct('a', 3L, 'b', 3L))").show +-+ | minA| +-+ |[1,1]| +-+ {code} Solution could be either to do safe<->unsafe conversion in InSet.doCodeGen or not trigger InSet optimization at all in this case. > InSet incorrect handling of structs > --- > > Key: SPARK-21228 > URL: https://issues.apache.org/jira/browse/SPARK-21228 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.2.0 >Reporter: Bogdan Raducanu > > In InSet it's possible that hset contains GenericInternalRows while child > returns UnsafeRows (and vice versa). InSet uses hset.contains (both in > doCodeGen and eval) which will always be false in this case. > The following code reproduces the problem: > {code} > spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "2") // the > default is 10 which requires a longer query text to repro > spark.range(1, 10).selectExpr("named_struct('a', id, 'b', id) as > a").createOrReplaceTempView("A") > sql("select * from (select min(a) as minA from A) A where minA in > (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', > 2L),named_struct('a', 3L, 'b', 3L))").show // the Aggregate here will return > UnsafeRows while the list of structs that will become hset will be > GenericInternalRows > ++ > |minA| > ++ > ++ > {code} > In.doCodeGen uses compareStructs and seems to work. In.eval might not work > but not sure how to reproduce. > {code} > spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "3") // now it > will not use InSet > sql("select * from (select min(a) as minA from A) A where minA in > (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', > 2L),named_struct('a', 3L, 'b', 3L))").show > +-+ > | minA| > +-+ > |[1,1]| > +-+ > {code} > Solution could be either to do safe<->unsafe conversion in InSet or not > trigger InSet optimization at all in this case. > Need to investigate if In.eval is affected. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: is
[jira] [Commented] (SPARK-21218) Convert IN predicate to equivalent Parquet filter
[ https://issues.apache.org/jira/browse/SPARK-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064880#comment-16064880 ] Hyukjin Kwon commented on SPARK-21218: -- Yea, I support this for what it worth. Let's resolve this as a duplicate BTW. You could link that JIRA in your PR at your PR title. > Convert IN predicate to equivalent Parquet filter > - > > Key: SPARK-21218 > URL: https://issues.apache.org/jira/browse/SPARK-21218 > Project: Spark > Issue Type: Improvement > Components: Optimizer >Affects Versions: 2.1.1 >Reporter: Michael Styles > Attachments: IN Predicate.png, OR Predicate.png > > > Convert IN predicate to equivalent expression involving equality conditions > to allow the filter to be pushed down to Parquet. > For instance, > C1 IN (10, 20) is rewritten as (C1 = 10) OR (C1 = 20) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21228) InSet incorrect handling of structs
[ https://issues.apache.org/jira/browse/SPARK-21228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064945#comment-16064945 ] Bogdan Raducanu commented on SPARK-21228: - I tested manually (since there is no flag to disable codegen for expressions) that In.eval also fails, so only In.doCodeGen appears correct. > InSet incorrect handling of structs > --- > > Key: SPARK-21228 > URL: https://issues.apache.org/jira/browse/SPARK-21228 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.2.0 >Reporter: Bogdan Raducanu > > In InSet it's possible that hset contains GenericInternalRows while child > returns UnsafeRows (and vice versa). InSet uses hset.contains (both in > doCodeGen and eval) which will always be false in this case. > The following code reproduces the problem: > {code} > spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "2") // the > default is 10 which requires a longer query text to repro > spark.range(1, 10).selectExpr("named_struct('a', id, 'b', id) as > a").createOrReplaceTempView("A") > sql("select * from (select min(a) as minA from A) A where minA in > (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', > 2L),named_struct('a', 3L, 'b', 3L))").show // the Aggregate here will return > UnsafeRows while the list of structs that will become hset will be > GenericInternalRows > ++ > |minA| > ++ > ++ > {code} > In.doCodeGen uses compareStructs and seems to work. In.eval might not work > but not sure how to reproduce. > {code} > spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "3") // now it > will not use InSet > sql("select * from (select min(a) as minA from A) A where minA in > (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', > 2L),named_struct('a', 3L, 'b', 3L))").show > +-+ > | minA| > +-+ > |[1,1]| > +-+ > {code} > Solution could be either to do safe<->unsafe conversion in InSet or not > trigger InSet optimization at all in this case. > Need to investigate if In.eval is affected. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20002) Add support for unions between streaming and batch datasets
[ https://issues.apache.org/jira/browse/SPARK-20002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064947#comment-16064947 ] Leon Pham commented on SPARK-20002: --- We're actually reading data from two different sources and one of them is a batch source that doesn't make sense as a streaming one. We're trying to concatenate the batch data to be analyzed with the streaming datasets. > Add support for unions between streaming and batch datasets > --- > > Key: SPARK-20002 > URL: https://issues.apache.org/jira/browse/SPARK-20002 > Project: Spark > Issue Type: Improvement > Components: SQL, Structured Streaming >Affects Versions: 2.0.2 >Reporter: Leon Pham > > Currently unions between streaming datasets and batch datasets are not > supported. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21228) InSet incorrect handling of structs
[ https://issues.apache.org/jira/browse/SPARK-21228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064970#comment-16064970 ] Bogdan Raducanu commented on SPARK-21228: - InSubquery.doCodeGen is using InSet directly (although InSubquery itself is never used) so a fix should consider this too. > InSet incorrect handling of structs > --- > > Key: SPARK-21228 > URL: https://issues.apache.org/jira/browse/SPARK-21228 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.2.0 >Reporter: Bogdan Raducanu > > In InSet it's possible that hset contains GenericInternalRows while child > returns UnsafeRows (and vice versa). InSet uses hset.contains (both in > doCodeGen and eval) which will always be false in this case. > The following code reproduces the problem: > {code} > spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "2") // the > default is 10 which requires a longer query text to repro > spark.range(1, 10).selectExpr("named_struct('a', id, 'b', id) as > a").createOrReplaceTempView("A") > sql("select * from (select min(a) as minA from A) A where minA in > (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', > 2L),named_struct('a', 3L, 'b', 3L))").show // the Aggregate here will return > UnsafeRows while the list of structs that will become hset will be > GenericInternalRows > ++ > |minA| > ++ > ++ > {code} > In.doCodeGen uses compareStructs and seems to work. In.eval might not work > but not sure how to reproduce. > {code} > spark.conf.set("spark.sql.optimizer.inSetConversionThreshold", "3") // now it > will not use InSet > sql("select * from (select min(a) as minA from A) A where minA in > (named_struct('a', 1L, 'b', 1L),named_struct('a', 2L, 'b', > 2L),named_struct('a', 3L, 'b', 3L))").show > +-+ > | minA| > +-+ > |[1,1]| > +-+ > {code} > Solution could be either to do safe<->unsafe conversion in InSet or not > trigger InSet optimization at all in this case. > Need to investigate if In.eval is affected. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20226) Call to sqlContext.cacheTable takes an incredibly long time in some cases
[ https://issues.apache.org/jira/browse/SPARK-20226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064972#comment-16064972 ] Barry Becker commented on SPARK-20226: -- Calling cache() on the dataframe on the after the addColumn used to make this run fast. But around the time that we upgraded to spark 2.1.1 it got very slow again. Calling cache on the dataframe does not seem to help any more. If I hardcode the addColumn column expression to be {code} (((CAST(Plate AS STRING) + CAST(State AS STRING)) + CAST(License Type AS STRING)) + CAST(Violation Time AS STRING)) + CAST(Violation AS STRING)) + CAST(Judgment Entry Date AS STRING)) + CAST(Issue Date AS STRING)) + CAST(Summons Number AS STRING)) + CAST(Fine Amount AS STRING)) + CAST(Penalty Amount AS STRING)) + CAST(Interest Amount AS STRING)) + CAST(Violation AS STRING)) {code} instead of {code} CAST(UDF(UDF(UDF(UDF(UDF(UDF(UDF(UDF(UDF(UDF(UDF(Plate, State), License Type), Violation Time), Violation), UDF(Judgment Entry Date)), UDF(Issue Date)), UDF(Summons Number)), UDF(Fine Amount)), UDF(Penalty Amount)), UDF(Interest Amount)), Violation) AS STRING) {code} which is what is generated by our expression parser, then the time goes from 70 seconds down to 10 seconds. Still slow, but not nearly as slow. > Call to sqlContext.cacheTable takes an incredibly long time in some cases > - > > Key: SPARK-20226 > URL: https://issues.apache.org/jira/browse/SPARK-20226 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.1.0 > Environment: linux or windows >Reporter: Barry Becker > Labels: cache > Attachments: profile_indexer2.PNG, xyzzy.csv > > > I have a case where the call to sqlContext.cacheTable can take an arbitrarily > long time depending on the number of columns that are referenced in a > withColumn expression applied to a dataframe. > The dataset is small (20 columns 7861 rows). The sequence to reproduce is the > following: > 1) add a new column that references 8 - 14 of the columns in the dataset. >- If I add 8 columns, then the call to cacheTable is fast - like *5 > seconds* >- If I add 11 columns, then it is slow - like *60 seconds* >- and if I add 14 columns, then it basically *takes forever* - I gave up > after 10 minutes or so. > The Column expression that is added, is basically just concatenating > the columns together in a single string. If a number is concatenated on a > string (or vice versa) the number is first converted to a string. > The expression looks something like this: > {code} > `Plate` + `State` + `License Type` + `Summons Number` + `Issue Date` + > `Violation Time` + `Violation` + `Judgment Entry Date` + `Fine Amount` + > `Penalty Amount` + `Interest Amount` > {code} > which we then convert to a Column expression that looks like this: > {code} > UDF(UDF(UDF(UDF(UDF(UDF(UDF(UDF(UDF(UDF('Plate, 'State), 'License Type), > UDF('Summons Number)), UDF('Issue Date)), 'Violation Time), 'Violation), > UDF('Judgment Entry Date)), UDF('Fine Amount)), UDF('Penalty Amount)), > UDF('Interest Amount)) > {code} >where the UDFs are very simple functions that basically call toString > and + as needed. > 2) apply a pipeline that includes some transformers that was saved earlier. > Here are the steps of the pipeline (extracted from parquet) > - > {code}{"class":"org.apache.spark.ml.feature.StringIndexerModel","timestamp":1491333200603,"sparkVersion":"2.1.0","uid":"strIdx_aeb04d2777cc","paramMap":{"handleInvalid":"skip","outputCol":"State_IDX__","inputCol":"State_CLEANED__"}}{code} > - > {code}{"class":"org.apache.spark.ml.feature.StringIndexerModel","timestamp":1491333200837,"sparkVersion":"2.1.0","uid":"strIdx_0164c4c13979","paramMap":{"inputCol":"License > Type_CLEANED__","handleInvalid":"skip","outputCol":"License > Type_IDX__"}}{code} > - > {code}{"class":"org.apache.spark.ml.feature.StringIndexerModel","timestamp":1491333201068,"sparkVersion":"2.1.0","uid":"strIdx_25b6cbd02751","paramMap":{"inputCol":"Violation_CLEANED__","handleInvalid":"skip","outputCol":"Violation_IDX__"}}{code} > - > {code}{"class":"org.apache.spark.ml.feature.StringIndexerModel","timestamp":1491333201282,"sparkVersion":"2.1.0","uid":"strIdx_aa12df0354d9","paramMap":{"handleInvalid":"skip","inputCol":"County_CLEANED__","outputCol":"County_IDX__"}}{code} > - > {code}{"class":"org.apache.spark.ml.feature.StringIndexerModel","timestamp":1491333201552,"sparkVersion":"2.1.0","uid":"strIdx_babb120f3cc1","paramMap":{"handleInvalid":"skip","outputCol":"Issuing > Agency_IDX__","inputCol":"Issuing Agency_CLEANED__"}}{code} > - > {code}{"class":"org.apache.spark.ml.feature.StringIndexerModel","timestamp":1491333201759
[jira] [Commented] (SPARK-18294) Implement commit protocol to support `mapred` package's committer
[ https://issues.apache.org/jira/browse/SPARK-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065023#comment-16065023 ] Apache Spark commented on SPARK-18294: -- User 'jiangxb1987' has created a pull request for this issue: https://github.com/apache/spark/pull/18438 > Implement commit protocol to support `mapred` package's committer > - > > Key: SPARK-18294 > URL: https://issues.apache.org/jira/browse/SPARK-18294 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Reporter: Jiang Xingbo > > Current `FileCommitProtocol` is based on `mapreduce` package, we should > implement a `HadoopMapRedCommitProtocol` that supports the older mapred > package's commiter. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-21229) remove QueryPlan.preCanonicalized
Wenchen Fan created SPARK-21229: --- Summary: remove QueryPlan.preCanonicalized Key: SPARK-21229 URL: https://issues.apache.org/jira/browse/SPARK-21229 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 2.3.0 Reporter: Wenchen Fan Assignee: Wenchen Fan Priority: Minor -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21229) remove QueryPlan.preCanonicalized
[ https://issues.apache.org/jira/browse/SPARK-21229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065118#comment-16065118 ] Apache Spark commented on SPARK-21229: -- User 'cloud-fan' has created a pull request for this issue: https://github.com/apache/spark/pull/18440 > remove QueryPlan.preCanonicalized > - > > Key: SPARK-21229 > URL: https://issues.apache.org/jira/browse/SPARK-21229 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.3.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-21229) remove QueryPlan.preCanonicalized
[ https://issues.apache.org/jira/browse/SPARK-21229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21229: Assignee: Apache Spark (was: Wenchen Fan) > remove QueryPlan.preCanonicalized > - > > Key: SPARK-21229 > URL: https://issues.apache.org/jira/browse/SPARK-21229 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.3.0 >Reporter: Wenchen Fan >Assignee: Apache Spark >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-21229) remove QueryPlan.preCanonicalized
[ https://issues.apache.org/jira/browse/SPARK-21229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21229: Assignee: Wenchen Fan (was: Apache Spark) > remove QueryPlan.preCanonicalized > - > > Key: SPARK-21229 > URL: https://issues.apache.org/jira/browse/SPARK-21229 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.3.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-19104) CompileException with Map and Case Class in Spark 2.1.0
[ https://issues.apache.org/jira/browse/SPARK-19104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-19104. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 18418 [https://github.com/apache/spark/pull/18418] > CompileException with Map and Case Class in Spark 2.1.0 > > > Key: SPARK-19104 > URL: https://issues.apache.org/jira/browse/SPARK-19104 > Project: Spark > Issue Type: Bug > Components: Optimizer, SQL >Affects Versions: 2.1.0, 2.2.0 >Reporter: Nils Grabbert > Fix For: 2.2.0 > > > The following code will run with Spark 2.0.2 but not with Spark 2.1.0: > {code} > case class InnerData(name: String, value: Int) > case class Data(id: Int, param: Map[String, InnerData]) > val data = Seq.tabulate(10)(i => Data(1, Map("key" -> InnerData("name", i + > 100 > val ds = spark.createDataset(data) > {code} > Exception: > {code} > Caused by: org.codehaus.commons.compiler.CompileException: File > 'generated.java', Line 63, Column 46: Expression > "ExternalMapToCatalyst_value_isNull1" is not an rvalue > at org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:11004) > at > org.codehaus.janino.UnitCompiler.toRvalueOrCompileException(UnitCompiler.java:6639) > > at > org.codehaus.janino.UnitCompiler.getConstantValue2(UnitCompiler.java:5001) > at org.codehaus.janino.UnitCompiler.access$10500(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$13.visitAmbiguousName(UnitCompiler.java:4984) > > at org.codehaus.janino.Java$AmbiguousName.accept(Java.java:3633) > at org.codehaus.janino.Java$Lvalue.accept(Java.java:3563) > at > org.codehaus.janino.UnitCompiler.getConstantValue(UnitCompiler.java:4956) > at org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:4925) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3189) > at org.codehaus.janino.UnitCompiler.access$5100(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$9.visitAssignment(UnitCompiler.java:3143) > at > org.codehaus.janino.UnitCompiler$9.visitAssignment(UnitCompiler.java:3139) > at org.codehaus.janino.Java$Assignment.accept(Java.java:3847) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:3139) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2112) > at org.codehaus.janino.UnitCompiler.access$1700(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1377) > > at > org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1370) > > at org.codehaus.janino.Java$ExpressionStatement.accept(Java.java:2558) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370) > at > org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1450) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:2811) > at > org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1262) > > at > org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1234) > > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:538) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:890) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:894) > at org.codehaus.janino.UnitCompiler.access$600(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:377) > > at > org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:369) > > at org.codehaus.janino.Java$MemberClassDeclaration.accept(Java.java:1128) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369) > at > org.codehaus.janino.UnitCompiler.compileDeclaredMemberTypes(UnitCompiler.java:1209) > > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:564) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:420) > at org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:374) > > at > org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:369) > > at > org.codehaus.janino.Java$AbstractPackageMemberClassDeclaration.accept(Java.java:1309) > > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369) > at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:345) > at > org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:396) > > at > org.codehaus.janino.ClassBodyEvaluator.compileToClass(ClassBodyEvaluator.java:311) > > at org.codehaus.janino.ClassBodyEvaluator.cook(ClassBodyE
[jira] [Assigned] (SPARK-19104) CompileException with Map and Case Class in Spark 2.1.0
[ https://issues.apache.org/jira/browse/SPARK-19104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-19104: --- Assignee: Liang-Chi Hsieh > CompileException with Map and Case Class in Spark 2.1.0 > > > Key: SPARK-19104 > URL: https://issues.apache.org/jira/browse/SPARK-19104 > Project: Spark > Issue Type: Bug > Components: Optimizer, SQL >Affects Versions: 2.1.0, 2.2.0 >Reporter: Nils Grabbert >Assignee: Liang-Chi Hsieh > Fix For: 2.2.0 > > > The following code will run with Spark 2.0.2 but not with Spark 2.1.0: > {code} > case class InnerData(name: String, value: Int) > case class Data(id: Int, param: Map[String, InnerData]) > val data = Seq.tabulate(10)(i => Data(1, Map("key" -> InnerData("name", i + > 100 > val ds = spark.createDataset(data) > {code} > Exception: > {code} > Caused by: org.codehaus.commons.compiler.CompileException: File > 'generated.java', Line 63, Column 46: Expression > "ExternalMapToCatalyst_value_isNull1" is not an rvalue > at org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:11004) > at > org.codehaus.janino.UnitCompiler.toRvalueOrCompileException(UnitCompiler.java:6639) > > at > org.codehaus.janino.UnitCompiler.getConstantValue2(UnitCompiler.java:5001) > at org.codehaus.janino.UnitCompiler.access$10500(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$13.visitAmbiguousName(UnitCompiler.java:4984) > > at org.codehaus.janino.Java$AmbiguousName.accept(Java.java:3633) > at org.codehaus.janino.Java$Lvalue.accept(Java.java:3563) > at > org.codehaus.janino.UnitCompiler.getConstantValue(UnitCompiler.java:4956) > at org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:4925) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3189) > at org.codehaus.janino.UnitCompiler.access$5100(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$9.visitAssignment(UnitCompiler.java:3143) > at > org.codehaus.janino.UnitCompiler$9.visitAssignment(UnitCompiler.java:3139) > at org.codehaus.janino.Java$Assignment.accept(Java.java:3847) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:3139) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2112) > at org.codehaus.janino.UnitCompiler.access$1700(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1377) > > at > org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1370) > > at org.codehaus.janino.Java$ExpressionStatement.accept(Java.java:2558) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370) > at > org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1450) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:2811) > at > org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1262) > > at > org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1234) > > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:538) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:890) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:894) > at org.codehaus.janino.UnitCompiler.access$600(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:377) > > at > org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:369) > > at org.codehaus.janino.Java$MemberClassDeclaration.accept(Java.java:1128) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369) > at > org.codehaus.janino.UnitCompiler.compileDeclaredMemberTypes(UnitCompiler.java:1209) > > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:564) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:420) > at org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:374) > > at > org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:369) > > at > org.codehaus.janino.Java$AbstractPackageMemberClassDeclaration.accept(Java.java:1309) > > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369) > at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:345) > at > org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:396) > > at > org.codehaus.janino.ClassBodyEvaluator.compileToClass(ClassBodyEvaluator.java:311) > > at org.codehaus.janino.ClassBodyEvaluator.cook(ClassBodyEvaluator.java:229) > at org.codehaus.janino.SimpleCompil
[jira] [Commented] (SPARK-21218) Convert IN predicate to equivalent Parquet filter
[ https://issues.apache.org/jira/browse/SPARK-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065142#comment-16065142 ] Michael Styles commented on SPARK-21218: [~hyukjin.kwon] Not sure I understand what you want me to do with my PR, assuming this JIRA is resolved as a duplicate? > Convert IN predicate to equivalent Parquet filter > - > > Key: SPARK-21218 > URL: https://issues.apache.org/jira/browse/SPARK-21218 > Project: Spark > Issue Type: Improvement > Components: Optimizer >Affects Versions: 2.1.1 >Reporter: Michael Styles > Attachments: IN Predicate.png, OR Predicate.png > > > Convert IN predicate to equivalent expression involving equality conditions > to allow the filter to be pushed down to Parquet. > For instance, > C1 IN (10, 20) is rewritten as (C1 = 10) OR (C1 = 20) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21218) Convert IN predicate to equivalent Parquet filter
[ https://issues.apache.org/jira/browse/SPARK-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065177#comment-16065177 ] Andrew Duffy commented on SPARK-21218: -- Curious, I wonder what the previous benchmarks were lacking. Have you tried disjunction push-down on other datatypes, e.g. strings? In any case, I'm also on board with this change, if it is in fact useful. I think [~hyukjin.kwon] was saying we should close this as a dupe, and rename the PR with the original ticket number (#17091). You can re-open that issue and say that you've done tests where this now looks like it will be a big improvement. > Convert IN predicate to equivalent Parquet filter > - > > Key: SPARK-21218 > URL: https://issues.apache.org/jira/browse/SPARK-21218 > Project: Spark > Issue Type: Improvement > Components: Optimizer >Affects Versions: 2.1.1 >Reporter: Michael Styles > Attachments: IN Predicate.png, OR Predicate.png > > > Convert IN predicate to equivalent expression involving equality conditions > to allow the filter to be pushed down to Parquet. > For instance, > C1 IN (10, 20) is rewritten as (C1 = 10) OR (C1 = 20) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-21218) Convert IN predicate to equivalent Parquet filter
[ https://issues.apache.org/jira/browse/SPARK-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065177#comment-16065177 ] Andrew Duffy edited comment on SPARK-21218 at 6/27/17 5:39 PM: --- Curious, I wonder what the previous benchmarks were lacking. Have you tried disjunction push-down on other datatypes, e.g. strings? In any case, I'm also on board with this change, if it is in fact useful. I think [~hyukjin.kwon] was saying we should close this as a dupe, and rename the PR with the original ticket number [17091|https://issues.apache.org/jira/browse/SPARK-17091]. You can re-open that issue and say that you've done tests where this now looks like it will be a big improvement. was (Author: andreweduffy): Curious, I wonder what the previous benchmarks were lacking. Have you tried disjunction push-down on other datatypes, e.g. strings? In any case, I'm also on board with this change, if it is in fact useful. I think [~hyukjin.kwon] was saying we should close this as a dupe, and rename the PR with the original ticket number (#17091). You can re-open that issue and say that you've done tests where this now looks like it will be a big improvement. > Convert IN predicate to equivalent Parquet filter > - > > Key: SPARK-21218 > URL: https://issues.apache.org/jira/browse/SPARK-21218 > Project: Spark > Issue Type: Improvement > Components: Optimizer >Affects Versions: 2.1.1 >Reporter: Michael Styles > Attachments: IN Predicate.png, OR Predicate.png > > > Convert IN predicate to equivalent expression involving equality conditions > to allow the filter to be pushed down to Parquet. > For instance, > C1 IN (10, 20) is rewritten as (C1 = 10) OR (C1 = 20) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21218) Convert IN predicate to equivalent Parquet filter
[ https://issues.apache.org/jira/browse/SPARK-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065210#comment-16065210 ] Michael Styles commented on SPARK-21218: In Parquet 1.7, there as a bug involving corrupt statistics on binary columns (https://issues.apache.org/jira/browse/PARQUET-251). This bug prevented earlier versions of Spark from generating Parquet filters on any string columns. Spark 2.1 has moved up to Parquet 1.8.2, so this issue no longer exists. > Convert IN predicate to equivalent Parquet filter > - > > Key: SPARK-21218 > URL: https://issues.apache.org/jira/browse/SPARK-21218 > Project: Spark > Issue Type: Improvement > Components: Optimizer >Affects Versions: 2.1.1 >Reporter: Michael Styles > Attachments: IN Predicate.png, OR Predicate.png > > > Convert IN predicate to equivalent expression involving equality conditions > to allow the filter to be pushed down to Parquet. > For instance, > C1 IN (10, 20) is rewritten as (C1 = 10) OR (C1 = 20) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21215) Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve
[ https://issues.apache.org/jira/browse/SPARK-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065329#comment-16065329 ] Michael Kunkel commented on SPARK-21215: The "resolution for this by [~sowen] was to put this on the spark mailing list. But I am sure this is just a scam to toss questions because the mailing list is no longer accepting emails. When sending a email to the mailing list, a reply is given Hello, This employee can no longer access email on this account. Your email will not be forwarded. So, this is not resolved. > Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot > resolve > - > > Key: SPARK-21215 > URL: https://issues.apache.org/jira/browse/SPARK-21215 > Project: Spark > Issue Type: Bug > Components: Java API, SQL >Affects Versions: 2.1.1 > Environment: macOSX >Reporter: Michael Kunkel > > First Spark project. > I have a Java method that returns a Dataset. I want to convert this to a > Dataset, where the Object is named StatusChangeDB. I have created a > POJO StatusChangeDB.java and coded it with all the query objects found in the > mySQL table. > I then create a Encoder and convert the Dataset to a > Dataset. However when I try to .show() the values of the > Dataset I receive the error > bq. > bq. Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot > resolve '`hvpinid_quad`' given input columns: [status_change_type, > superLayer, loclayer, sector, locwire]; > bq. at > org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42) > bq. at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:86) > bq. at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:83) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:290) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:290) > bq. at > org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:289) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:307) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:188) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:305) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4$$anonfun$apply$10.apply(TreeNode.scala:324) > bq. at > scala.collection.MapLike$MappedValues$$anonfun$iterator$3.apply(MapLike.scala:246) > bq. at > scala.collection.MapLike$MappedValues$$anonfun$iterator$3.apply(MapLike.scala:246) > bq. at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) > bq. at scala.collection.Iterator$class.foreach(Iterator.scala:893) > bq. at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) > bq. at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > bq. at scala.collection.IterableLike$$anon$1.foreach(IterableLike.scala:311) > bq. at > scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59) > bq. at > scala.collection.mutable.MapBuilder.$plus$plus$eq(MapBuilder.scala:25) > bq. at > scala.collection.TraversableViewLike$class.force(TraversableViewLike.scala:88) > bq. at scala.collection.IterableLike$$anon$1.force(IterableLike.scala:311) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:332) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:188) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:305) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:255) > bq. at > org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:255) > bq. at > org.apache.spark.sql.catalyst.plans.QueryPlan.transform
[jira] [Created] (SPARK-21230) Spark Encoder with mysql Enum and data truncated Error
Michael Kunkel created SPARK-21230: -- Summary: Spark Encoder with mysql Enum and data truncated Error Key: SPARK-21230 URL: https://issues.apache.org/jira/browse/SPARK-21230 Project: Spark Issue Type: Bug Components: Java API Affects Versions: 2.1.1 Environment: macosX Reporter: Michael Kunkel I am using Spark via Java for a MYSQL/ML(machine learning) project. In the mysql database, I have a column "status_change_type" of type enum = {broke, fixed} in a table called "status_change" in a DB called "test". I have an object StatusChangeDB that constructs the needed structure for the table, however for the "status_change_type", I constructed it as a String. I know the bytes from MYSQL enum to Java string are much different, but I am using Spark, so the encoder does not recognize enums properly. However when I try to set the value of the enum via a Java string, I receive the "data truncated" error h5. org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 4.0 failed 1 times, most recent failure: Lost task 0.0 in stage 4.0 (TID 9, localhost, executor driver): java.sql.BatchUpdateException: Data truncated for column 'status_change_type' at row 1 at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:2055) I have tried to use enum for "status_change_type", however it fails with a stack trace of h5. Exception in thread "AWT-EventQueue-0" java.lang.NullPointerException at org.spark_project.guava.reflect.TypeToken.method(TypeToken.java:465) at org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:126) at org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:125) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186) at org.apache.spark.sql.catalyst.JavaTypeInference$.org$apache$spark$sql$catalyst$JavaTypeInference$$inferDataType(JavaTypeInference.scala:125) at org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:127) at org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:125) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186) at org.apache.spark.sql.catalyst.JavaTypeInference$.org$apache$spark$sql$catalyst$JavaTypeInference$$inferDataType(JavaTypeInference.scala:125) at org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:127) at org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:125) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186) ... ... h5. I have tried to use the jdbc setting "jdbcCompliantTruncation=false" but this does nothing as I get the same error of "data truncated" as first stated. Here are my jdbc options map, in case I am using the "jdbcCompliantTruncation=false" incorrectly. public static Map jdbcOptions() { Map jdbcOptions = new HashMap(); jdbcOptions.put("url", "jdbc:mysql://localhost:3306/test?jdbcCompliantTruncation=false"); jdbcOptions.put("driver", "com.mysql.jdbc.Driver"); jdbcOptions.put("dbtable", "status_change"); jdbcOptions.put("user", "root"); jdbcOptions.put("password", ""); return jdbcOptions; } Here is the Spark method for inserting into the mysql DB private void insertMYSQLQuery(Dataset changeDF) { try { changeDF.write().mode(SaveMode.Append).jdbc(SparkManager.jdbcAppendOptions(), "status_change", new java.util.Properties()); } catch (Exception e) { System.out.println(e); } } where jdbcAppendOptions uses the jdbcOptions m
[jira] [Commented] (SPARK-21215) Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve
[ https://issues.apache.org/jira/browse/SPARK-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065336#comment-16065336 ] Sean Owen commented on SPARK-21215: --- I'm not sure what you're referring to. The user@ list works fine. sometimes you get weird auto-responses from somebody's account on the list. But, it has nothing to do with the list. > Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot > resolve > - > > Key: SPARK-21215 > URL: https://issues.apache.org/jira/browse/SPARK-21215 > Project: Spark > Issue Type: Bug > Components: Java API, SQL >Affects Versions: 2.1.1 > Environment: macOSX >Reporter: Michael Kunkel > > First Spark project. > I have a Java method that returns a Dataset. I want to convert this to a > Dataset, where the Object is named StatusChangeDB. I have created a > POJO StatusChangeDB.java and coded it with all the query objects found in the > mySQL table. > I then create a Encoder and convert the Dataset to a > Dataset. However when I try to .show() the values of the > Dataset I receive the error > bq. > bq. Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot > resolve '`hvpinid_quad`' given input columns: [status_change_type, > superLayer, loclayer, sector, locwire]; > bq. at > org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42) > bq. at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:86) > bq. at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:83) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:290) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:290) > bq. at > org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:289) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:307) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:188) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:305) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4$$anonfun$apply$10.apply(TreeNode.scala:324) > bq. at > scala.collection.MapLike$MappedValues$$anonfun$iterator$3.apply(MapLike.scala:246) > bq. at > scala.collection.MapLike$MappedValues$$anonfun$iterator$3.apply(MapLike.scala:246) > bq. at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) > bq. at scala.collection.Iterator$class.foreach(Iterator.scala:893) > bq. at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) > bq. at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > bq. at scala.collection.IterableLike$$anon$1.foreach(IterableLike.scala:311) > bq. at > scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59) > bq. at > scala.collection.mutable.MapBuilder.$plus$plus$eq(MapBuilder.scala:25) > bq. at > scala.collection.TraversableViewLike$class.force(TraversableViewLike.scala:88) > bq. at scala.collection.IterableLike$$anon$1.force(IterableLike.scala:311) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:332) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:188) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:305) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:255) > bq. at > org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:255) > bq. at > org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:266) > bq. at > org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(QueryPlan.scala:276) > bq. at > org.apac
[jira] [Commented] (SPARK-21230) Spark Encoder with mysql Enum and data truncated Error
[ https://issues.apache.org/jira/browse/SPARK-21230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065338#comment-16065338 ] Sean Owen commented on SPARK-21230: --- This does also not look like a useful JIRA. It looks like a question about using MySQL and JDBC. Until it's narrowed down to a Spark issue, we'd generally close this. > Spark Encoder with mysql Enum and data truncated Error > -- > > Key: SPARK-21230 > URL: https://issues.apache.org/jira/browse/SPARK-21230 > Project: Spark > Issue Type: Bug > Components: Java API >Affects Versions: 2.1.1 > Environment: macosX >Reporter: Michael Kunkel > > I am using Spark via Java for a MYSQL/ML(machine learning) project. > In the mysql database, I have a column "status_change_type" of type enum = > {broke, fixed} in a table called "status_change" in a DB called "test". > I have an object StatusChangeDB that constructs the needed structure for the > table, however for the "status_change_type", I constructed it as a String. I > know the bytes from MYSQL enum to Java string are much different, but I am > using Spark, so the encoder does not recognize enums properly. However when I > try to set the value of the enum via a Java string, I receive the "data > truncated" error > h5. org.apache.spark.SparkException: Job aborted due to stage failure: > Task 0 in stage 4.0 failed 1 times, most recent failure: Lost task 0.0 in > stage 4.0 (TID 9, localhost, executor driver): java.sql.BatchUpdateException: > Data truncated for column 'status_change_type' at row 1 at > com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:2055) > I have tried to use enum for "status_change_type", however it fails with a > stack trace of > h5. Exception in thread "AWT-EventQueue-0" java.lang.NullPointerException > at org.spark_project.guava.reflect.TypeToken.method(TypeToken.java:465) at > org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:126) > at > org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:125) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) at > scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at > scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186) at > org.apache.spark.sql.catalyst.JavaTypeInference$.org$apache$spark$sql$catalyst$JavaTypeInference$$inferDataType(JavaTypeInference.scala:125) > at > org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:127) > at > org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:125) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) at > scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at > scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186) at > org.apache.spark.sql.catalyst.JavaTypeInference$.org$apache$spark$sql$catalyst$JavaTypeInference$$inferDataType(JavaTypeInference.scala:125) > at > org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:127) > at > org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:125) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) at > scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at > scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186) ... ... > h5. > I have tried to use the jdbc setting "jdbcCompliantTruncation=false" but this > does nothing as I get the same error of "data truncated" as first stated. > Here are my jdbc options map, in case I am using the > "jdbcCompliantTruncation=false" incorrectly. > public static Map jdbcOptions() { > Map jdbcOptions = new HashMap(); > jdbcOptions.put("url", > "jdbc:mysql://localhost:3306/test?jdbcCompliantTruncation=false"); > jdbcOptions.put("driver", "com.mysql.jdbc.Driver"); > jdbcOpti
[jira] [Commented] (SPARK-21215) Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve
[ https://issues.apache.org/jira/browse/SPARK-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065345#comment-16065345 ] Michael Kunkel commented on SPARK-21215: I looked at a few months worth of posts, and it seems that the mailing list is not accepting new mails. This is because the mailing list is no longer valid according to the auto-reply from the yahoo-mailing server. > Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot > resolve > - > > Key: SPARK-21215 > URL: https://issues.apache.org/jira/browse/SPARK-21215 > Project: Spark > Issue Type: Bug > Components: Java API, SQL >Affects Versions: 2.1.1 > Environment: macOSX >Reporter: Michael Kunkel > > First Spark project. > I have a Java method that returns a Dataset. I want to convert this to a > Dataset, where the Object is named StatusChangeDB. I have created a > POJO StatusChangeDB.java and coded it with all the query objects found in the > mySQL table. > I then create a Encoder and convert the Dataset to a > Dataset. However when I try to .show() the values of the > Dataset I receive the error > bq. > bq. Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot > resolve '`hvpinid_quad`' given input columns: [status_change_type, > superLayer, loclayer, sector, locwire]; > bq. at > org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42) > bq. at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:86) > bq. at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:83) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:290) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:290) > bq. at > org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:289) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:307) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:188) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:305) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4$$anonfun$apply$10.apply(TreeNode.scala:324) > bq. at > scala.collection.MapLike$MappedValues$$anonfun$iterator$3.apply(MapLike.scala:246) > bq. at > scala.collection.MapLike$MappedValues$$anonfun$iterator$3.apply(MapLike.scala:246) > bq. at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) > bq. at scala.collection.Iterator$class.foreach(Iterator.scala:893) > bq. at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) > bq. at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > bq. at scala.collection.IterableLike$$anon$1.foreach(IterableLike.scala:311) > bq. at > scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59) > bq. at > scala.collection.mutable.MapBuilder.$plus$plus$eq(MapBuilder.scala:25) > bq. at > scala.collection.TraversableViewLike$class.force(TraversableViewLike.scala:88) > bq. at scala.collection.IterableLike$$anon$1.force(IterableLike.scala:311) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:332) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:188) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:305) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:255) > bq. at > org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:255) > bq. at > org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:266) > bq. at > org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(Q
[jira] [Commented] (SPARK-21215) Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve
[ https://issues.apache.org/jira/browse/SPARK-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065354#comment-16065354 ] Michael Kunkel commented on SPARK-21215: The posts go onto the list, but the owner ASF does not have access to it, as the reply states. If they have no access to it, how can they be the help? > Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot > resolve > - > > Key: SPARK-21215 > URL: https://issues.apache.org/jira/browse/SPARK-21215 > Project: Spark > Issue Type: Bug > Components: Java API, SQL >Affects Versions: 2.1.1 > Environment: macOSX >Reporter: Michael Kunkel > > First Spark project. > I have a Java method that returns a Dataset. I want to convert this to a > Dataset, where the Object is named StatusChangeDB. I have created a > POJO StatusChangeDB.java and coded it with all the query objects found in the > mySQL table. > I then create a Encoder and convert the Dataset to a > Dataset. However when I try to .show() the values of the > Dataset I receive the error > bq. > bq. Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot > resolve '`hvpinid_quad`' given input columns: [status_change_type, > superLayer, loclayer, sector, locwire]; > bq. at > org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42) > bq. at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:86) > bq. at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:83) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:290) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:290) > bq. at > org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:289) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:307) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:188) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:305) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4$$anonfun$apply$10.apply(TreeNode.scala:324) > bq. at > scala.collection.MapLike$MappedValues$$anonfun$iterator$3.apply(MapLike.scala:246) > bq. at > scala.collection.MapLike$MappedValues$$anonfun$iterator$3.apply(MapLike.scala:246) > bq. at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) > bq. at scala.collection.Iterator$class.foreach(Iterator.scala:893) > bq. at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) > bq. at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > bq. at scala.collection.IterableLike$$anon$1.foreach(IterableLike.scala:311) > bq. at > scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59) > bq. at > scala.collection.mutable.MapBuilder.$plus$plus$eq(MapBuilder.scala:25) > bq. at > scala.collection.TraversableViewLike$class.force(TraversableViewLike.scala:88) > bq. at scala.collection.IterableLike$$anon$1.force(IterableLike.scala:311) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:332) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:188) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:305) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:255) > bq. at > org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:255) > bq. at > org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:266) > bq. at > org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(QueryPlan.scala:276) > bq. at > org.apache.spark.sql.catalyst.p
[jira] [Commented] (SPARK-21230) Spark Encoder with mysql Enum and data truncated Error
[ https://issues.apache.org/jira/browse/SPARK-21230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065347#comment-16065347 ] Michael Kunkel commented on SPARK-21230: The problem is with the Spark Encoder of type enum. So the problem is spark as the post attempts to point out. > Spark Encoder with mysql Enum and data truncated Error > -- > > Key: SPARK-21230 > URL: https://issues.apache.org/jira/browse/SPARK-21230 > Project: Spark > Issue Type: Bug > Components: Java API >Affects Versions: 2.1.1 > Environment: macosX >Reporter: Michael Kunkel > > I am using Spark via Java for a MYSQL/ML(machine learning) project. > In the mysql database, I have a column "status_change_type" of type enum = > {broke, fixed} in a table called "status_change" in a DB called "test". > I have an object StatusChangeDB that constructs the needed structure for the > table, however for the "status_change_type", I constructed it as a String. I > know the bytes from MYSQL enum to Java string are much different, but I am > using Spark, so the encoder does not recognize enums properly. However when I > try to set the value of the enum via a Java string, I receive the "data > truncated" error > h5. org.apache.spark.SparkException: Job aborted due to stage failure: > Task 0 in stage 4.0 failed 1 times, most recent failure: Lost task 0.0 in > stage 4.0 (TID 9, localhost, executor driver): java.sql.BatchUpdateException: > Data truncated for column 'status_change_type' at row 1 at > com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:2055) > I have tried to use enum for "status_change_type", however it fails with a > stack trace of > h5. Exception in thread "AWT-EventQueue-0" java.lang.NullPointerException > at org.spark_project.guava.reflect.TypeToken.method(TypeToken.java:465) at > org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:126) > at > org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:125) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) at > scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at > scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186) at > org.apache.spark.sql.catalyst.JavaTypeInference$.org$apache$spark$sql$catalyst$JavaTypeInference$$inferDataType(JavaTypeInference.scala:125) > at > org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:127) > at > org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:125) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) at > scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at > scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186) at > org.apache.spark.sql.catalyst.JavaTypeInference$.org$apache$spark$sql$catalyst$JavaTypeInference$$inferDataType(JavaTypeInference.scala:125) > at > org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:127) > at > org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:125) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) at > scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at > scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186) ... ... > h5. > I have tried to use the jdbc setting "jdbcCompliantTruncation=false" but this > does nothing as I get the same error of "data truncated" as first stated. > Here are my jdbc options map, in case I am using the > "jdbcCompliantTruncation=false" incorrectly. > public static Map jdbcOptions() { > Map jdbcOptions = new HashMap(); > jdbcOptions.put("url", > "jdbc:mysql://localhost:3306/test?jdbcCompliantTruncation=false"); > jdbcOptions.put("driver", "com.mysql.jdbc.Driver"); > jdbcOptions.put("dbtable", "status_change"); > jdbc
[jira] [Comment Edited] (SPARK-21215) Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve
[ https://issues.apache.org/jira/browse/SPARK-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065354#comment-16065354 ] Michael Kunkel edited comment on SPARK-21215 at 6/27/17 7:40 PM: - The posts go onto the list, but the owner, ASF, does not have access to it, as the reply states. If they have no access to it, how can they be the help? was (Author: michaelkunkel): The posts go onto the list, but the owner ASF does not have access to it, as the reply states. If they have no access to it, how can they be the help? > Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot > resolve > - > > Key: SPARK-21215 > URL: https://issues.apache.org/jira/browse/SPARK-21215 > Project: Spark > Issue Type: Bug > Components: Java API, SQL >Affects Versions: 2.1.1 > Environment: macOSX >Reporter: Michael Kunkel > > First Spark project. > I have a Java method that returns a Dataset. I want to convert this to a > Dataset, where the Object is named StatusChangeDB. I have created a > POJO StatusChangeDB.java and coded it with all the query objects found in the > mySQL table. > I then create a Encoder and convert the Dataset to a > Dataset. However when I try to .show() the values of the > Dataset I receive the error > bq. > bq. Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot > resolve '`hvpinid_quad`' given input columns: [status_change_type, > superLayer, loclayer, sector, locwire]; > bq. at > org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42) > bq. at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:86) > bq. at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:83) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:290) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:290) > bq. at > org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:289) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:307) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:188) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:305) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4$$anonfun$apply$10.apply(TreeNode.scala:324) > bq. at > scala.collection.MapLike$MappedValues$$anonfun$iterator$3.apply(MapLike.scala:246) > bq. at > scala.collection.MapLike$MappedValues$$anonfun$iterator$3.apply(MapLike.scala:246) > bq. at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) > bq. at scala.collection.Iterator$class.foreach(Iterator.scala:893) > bq. at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) > bq. at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > bq. at scala.collection.IterableLike$$anon$1.foreach(IterableLike.scala:311) > bq. at > scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59) > bq. at > scala.collection.mutable.MapBuilder.$plus$plus$eq(MapBuilder.scala:25) > bq. at > scala.collection.TraversableViewLike$class.force(TraversableViewLike.scala:88) > bq. at scala.collection.IterableLike$$anon$1.force(IterableLike.scala:311) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:332) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:188) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:305) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:255) > bq. at > org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:255) > bq. at > org.apache.spark.sql.catalyst.plans.QueryPlan.tra
[jira] [Commented] (SPARK-21215) Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve
[ https://issues.apache.org/jira/browse/SPARK-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065351#comment-16065351 ] Michael Kunkel commented on SPARK-21215: [~sowen] I am not attempting to argue that facts. When I try to post to the ASF, I instantly get a reply from yahoo, with the subject matter exactly as the mail I sent. The reply states that the owner no longer has access to the mailing list. Try it please > Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot > resolve > - > > Key: SPARK-21215 > URL: https://issues.apache.org/jira/browse/SPARK-21215 > Project: Spark > Issue Type: Bug > Components: Java API, SQL >Affects Versions: 2.1.1 > Environment: macOSX >Reporter: Michael Kunkel > > First Spark project. > I have a Java method that returns a Dataset. I want to convert this to a > Dataset, where the Object is named StatusChangeDB. I have created a > POJO StatusChangeDB.java and coded it with all the query objects found in the > mySQL table. > I then create a Encoder and convert the Dataset to a > Dataset. However when I try to .show() the values of the > Dataset I receive the error > bq. > bq. Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot > resolve '`hvpinid_quad`' given input columns: [status_change_type, > superLayer, loclayer, sector, locwire]; > bq. at > org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42) > bq. at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:86) > bq. at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:83) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:290) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:290) > bq. at > org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:289) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:307) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:188) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:305) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4$$anonfun$apply$10.apply(TreeNode.scala:324) > bq. at > scala.collection.MapLike$MappedValues$$anonfun$iterator$3.apply(MapLike.scala:246) > bq. at > scala.collection.MapLike$MappedValues$$anonfun$iterator$3.apply(MapLike.scala:246) > bq. at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) > bq. at scala.collection.Iterator$class.foreach(Iterator.scala:893) > bq. at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) > bq. at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > bq. at scala.collection.IterableLike$$anon$1.foreach(IterableLike.scala:311) > bq. at > scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59) > bq. at > scala.collection.mutable.MapBuilder.$plus$plus$eq(MapBuilder.scala:25) > bq. at > scala.collection.TraversableViewLike$class.force(TraversableViewLike.scala:88) > bq. at scala.collection.IterableLike$$anon$1.force(IterableLike.scala:311) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:332) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:188) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:305) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:255) > bq. at > org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:255) > bq. at > org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:266) > bq. at > org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$cata
[jira] [Commented] (SPARK-21215) Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve
[ https://issues.apache.org/jira/browse/SPARK-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065348#comment-16065348 ] Sean Owen commented on SPARK-21215: --- Not sure what you're looking at, but the mailing list has posts from 10 minutes ago: http://apache-spark-user-list.1001560.n3.nabble.com/ I'm not sure what you mean about Yahoo, because nothing about this project or the ASF uses Yahoo. > Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot > resolve > - > > Key: SPARK-21215 > URL: https://issues.apache.org/jira/browse/SPARK-21215 > Project: Spark > Issue Type: Bug > Components: Java API, SQL >Affects Versions: 2.1.1 > Environment: macOSX >Reporter: Michael Kunkel > > First Spark project. > I have a Java method that returns a Dataset. I want to convert this to a > Dataset, where the Object is named StatusChangeDB. I have created a > POJO StatusChangeDB.java and coded it with all the query objects found in the > mySQL table. > I then create a Encoder and convert the Dataset to a > Dataset. However when I try to .show() the values of the > Dataset I receive the error > bq. > bq. Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot > resolve '`hvpinid_quad`' given input columns: [status_change_type, > superLayer, loclayer, sector, locwire]; > bq. at > org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42) > bq. at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:86) > bq. at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:83) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:290) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:290) > bq. at > org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:289) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:307) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:188) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:305) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4$$anonfun$apply$10.apply(TreeNode.scala:324) > bq. at > scala.collection.MapLike$MappedValues$$anonfun$iterator$3.apply(MapLike.scala:246) > bq. at > scala.collection.MapLike$MappedValues$$anonfun$iterator$3.apply(MapLike.scala:246) > bq. at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) > bq. at scala.collection.Iterator$class.foreach(Iterator.scala:893) > bq. at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) > bq. at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > bq. at scala.collection.IterableLike$$anon$1.foreach(IterableLike.scala:311) > bq. at > scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59) > bq. at > scala.collection.mutable.MapBuilder.$plus$plus$eq(MapBuilder.scala:25) > bq. at > scala.collection.TraversableViewLike$class.force(TraversableViewLike.scala:88) > bq. at scala.collection.IterableLike$$anon$1.force(IterableLike.scala:311) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:332) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:188) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:305) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:255) > bq. at > org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:255) > bq. at > org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:266) > bq. at > org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveT
[jira] [Comment Edited] (SPARK-21215) Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve
[ https://issues.apache.org/jira/browse/SPARK-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065351#comment-16065351 ] Michael Kunkel edited comment on SPARK-21215 at 6/27/17 7:37 PM: - [~sowen] I am not attempting to argue the facts. When I try to post to the ASF, I instantly get a reply from yahoo, with the subject matter exactly as the mail I sent. The reply states that the owner no longer has access to the mailing list. Try it please was (Author: michaelkunkel): [~sowen] I am not attempting to argue that facts. When I try to post to the ASF, I instantly get a reply from yahoo, with the subject matter exactly as the mail I sent. The reply states that the owner no longer has access to the mailing list. Try it please > Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot > resolve > - > > Key: SPARK-21215 > URL: https://issues.apache.org/jira/browse/SPARK-21215 > Project: Spark > Issue Type: Bug > Components: Java API, SQL >Affects Versions: 2.1.1 > Environment: macOSX >Reporter: Michael Kunkel > > First Spark project. > I have a Java method that returns a Dataset. I want to convert this to a > Dataset, where the Object is named StatusChangeDB. I have created a > POJO StatusChangeDB.java and coded it with all the query objects found in the > mySQL table. > I then create a Encoder and convert the Dataset to a > Dataset. However when I try to .show() the values of the > Dataset I receive the error > bq. > bq. Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot > resolve '`hvpinid_quad`' given input columns: [status_change_type, > superLayer, loclayer, sector, locwire]; > bq. at > org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42) > bq. at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:86) > bq. at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:83) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:290) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:290) > bq. at > org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:289) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:307) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:188) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:305) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4$$anonfun$apply$10.apply(TreeNode.scala:324) > bq. at > scala.collection.MapLike$MappedValues$$anonfun$iterator$3.apply(MapLike.scala:246) > bq. at > scala.collection.MapLike$MappedValues$$anonfun$iterator$3.apply(MapLike.scala:246) > bq. at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) > bq. at scala.collection.Iterator$class.foreach(Iterator.scala:893) > bq. at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) > bq. at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > bq. at scala.collection.IterableLike$$anon$1.foreach(IterableLike.scala:311) > bq. at > scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59) > bq. at > scala.collection.mutable.MapBuilder.$plus$plus$eq(MapBuilder.scala:25) > bq. at > scala.collection.TraversableViewLike$class.force(TraversableViewLike.scala:88) > bq. at scala.collection.IterableLike$$anon$1.force(IterableLike.scala:311) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:332) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:188) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:305) > bq. at > org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:287) > bq. at > org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsU
[jira] [Reopened] (SPARK-17091) ParquetFilters rewrite IN to OR of Eq
[ https://issues.apache.org/jira/browse/SPARK-17091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Styles reopened SPARK-17091: > ParquetFilters rewrite IN to OR of Eq > - > > Key: SPARK-17091 > URL: https://issues.apache.org/jira/browse/SPARK-17091 > Project: Spark > Issue Type: Bug >Reporter: Andrew Duffy > > Past attempts at pushing down the InSet operation for Parquet relied on > user-defined predicates. It would be simpler to rewrite an IN clause into the > corresponding OR union of a set of equality conditions. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17091) ParquetFilters rewrite IN to OR of Eq
[ https://issues.apache.org/jira/browse/SPARK-17091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065413#comment-16065413 ] Michael Styles commented on SPARK-17091: By not pushing the filter to Parquet, are we not preventing Parquet from skipping blocks during read operations? I have tests that show big improvements when applying this transformation. For instance, I have a Parquet file with 162,456,394 rows which is sorted on column C1. *IN Predicate* {noformat} df.filter[df['C1'].isin([42, 139])).collect() {noformat} *OR Predicate* {noformat} df.filter((df['C1'] == 42) | (df['C1'] == 139)).collect() {noformat} I'm seeing about a 50 -75 % improvement. > ParquetFilters rewrite IN to OR of Eq > - > > Key: SPARK-17091 > URL: https://issues.apache.org/jira/browse/SPARK-17091 > Project: Spark > Issue Type: Bug >Reporter: Andrew Duffy > > Past attempts at pushing down the InSet operation for Parquet relied on > user-defined predicates. It would be simpler to rewrite an IN clause into the > corresponding OR union of a set of equality conditions. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-17091) ParquetFilters rewrite IN to OR of Eq
[ https://issues.apache.org/jira/browse/SPARK-17091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Styles updated SPARK-17091: --- Attachment: IN Predicate.png OR Predicate.png > ParquetFilters rewrite IN to OR of Eq > - > > Key: SPARK-17091 > URL: https://issues.apache.org/jira/browse/SPARK-17091 > Project: Spark > Issue Type: Bug >Reporter: Andrew Duffy > Attachments: IN Predicate.png, OR Predicate.png > > > Past attempts at pushing down the InSet operation for Parquet relied on > user-defined predicates. It would be simpler to rewrite an IN clause into the > corresponding OR union of a set of equality conditions. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-17091) Convert IN predicate to equivalent Parquet filter
[ https://issues.apache.org/jira/browse/SPARK-17091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Styles updated SPARK-17091: --- Summary: Convert IN predicate to equivalent Parquet filter (was: ParquetFilters rewrite IN to OR of Eq) > Convert IN predicate to equivalent Parquet filter > - > > Key: SPARK-17091 > URL: https://issues.apache.org/jira/browse/SPARK-17091 > Project: Spark > Issue Type: Bug >Reporter: Andrew Duffy > Attachments: IN Predicate.png, OR Predicate.png > > > Past attempts at pushing down the InSet operation for Parquet relied on > user-defined predicates. It would be simpler to rewrite an IN clause into the > corresponding OR union of a set of equality conditions. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-17091) Convert IN predicate to equivalent Parquet filter
[ https://issues.apache.org/jira/browse/SPARK-17091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065413#comment-16065413 ] Michael Styles edited comment on SPARK-17091 at 6/27/17 8:26 PM: - By not pushing the filter to Parquet, are we not preventing Parquet from skipping blocks during read operations? I have tests that show big improvements when applying this transformation. For instance, I have a Parquet file with 162,456,394 rows which is sorted on column C1. *IN Predicate* {noformat} df.filter[df['C1'].isin([42, 139])).collect() {noformat} *OR Predicate* {noformat} df.filter((df['C1'] == 42) | (df['C1'] == 139)).collect() {noformat} I'm seeing about a 50 -75 % improvement. See attachments. was (Author: ptkool): By not pushing the filter to Parquet, are we not preventing Parquet from skipping blocks during read operations? I have tests that show big improvements when applying this transformation. For instance, I have a Parquet file with 162,456,394 rows which is sorted on column C1. *IN Predicate* {noformat} df.filter[df['C1'].isin([42, 139])).collect() {noformat} *OR Predicate* {noformat} df.filter((df['C1'] == 42) | (df['C1'] == 139)).collect() {noformat} I'm seeing about a 50 -75 % improvement. > Convert IN predicate to equivalent Parquet filter > - > > Key: SPARK-17091 > URL: https://issues.apache.org/jira/browse/SPARK-17091 > Project: Spark > Issue Type: Bug >Reporter: Andrew Duffy > Attachments: IN Predicate.png, OR Predicate.png > > > Past attempts at pushing down the InSet operation for Parquet relied on > user-defined predicates. It would be simpler to rewrite an IN clause into the > corresponding OR union of a set of equality conditions. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-21218) Convert IN predicate to equivalent Parquet filter
[ https://issues.apache.org/jira/browse/SPARK-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Styles resolved SPARK-21218. Resolution: Duplicate > Convert IN predicate to equivalent Parquet filter > - > > Key: SPARK-21218 > URL: https://issues.apache.org/jira/browse/SPARK-21218 > Project: Spark > Issue Type: Improvement > Components: Optimizer >Affects Versions: 2.1.1 >Reporter: Michael Styles > Attachments: IN Predicate.png, OR Predicate.png > > > Convert IN predicate to equivalent expression involving equality conditions > to allow the filter to be pushed down to Parquet. > For instance, > C1 IN (10, 20) is rewritten as (C1 = 10) OR (C1 = 20) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21137) Spark reads many small files slowly
[ https://issues.apache.org/jira/browse/SPARK-21137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065423#comment-16065423 ] Steve Loughran commented on SPARK-21137: Looking at this. something is trying to get the permissions for every file, which is being dealt with by an exec & all the overheads of that. Looking at the code, it's in the constructor of {{LocatedFileStatus}}, which is building it from another {{FileStatus}}. Which normally is just a simple copy of a field (fast, efficient). Looks like on RawLocalFileSystem, it actually triggers an on demand execution. Been around for a long time (HADOOP-2288), surfacing here because you're working with the local FS. For all other filesystems it's a quick operation. I think this is an issue: I don't think anybody thought this would be a problem, as it's just viewed as a marshalling of a LocatedFileStatus, which is what you get back from {{FileSystem.listLocatedStatus}}. Normally that's the higher performing one, not just on object stores, but because it scales better, being able to incrementally send back data in batches, rather than needing to enumerate an entire directory of files (possibly in the millions) and then send them around as arrays of FileStatus. Here, it's clearly not. What to do? I think we could consider whether it'd be possible to add this to the hadoop native libs & so make a fast API call. There's also the option of "allowing us to completely disable permissions entirely". That one appeals to me more from a windows perspective, where you could get rid of the hadoop native lib and still have (most) things work there...but as its an incomplete "most" it's probably an optimistic goal. > Spark reads many small files slowly > --- > > Key: SPARK-21137 > URL: https://issues.apache.org/jira/browse/SPARK-21137 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 2.1.1 >Reporter: sam >Priority: Minor > > A very common use case in big data is to read a large number of small files. > For example the Enron email dataset has 1,227,645 small files. > When one tries to read this data using Spark one will hit many issues. > Firstly, even if the data is small (each file only say 1K) any job can take a > very long time (I have a simple job that has been running for 3 hours and has > not yet got to the point of starting any tasks, I doubt if it will ever > finish). > It seems all the code in Spark that manages file listing is single threaded > and not well optimised. When I hand crank the code and don't use Spark, my > job runs much faster. > Is it possible that I'm missing some configuration option? It seems kinda > surprising to me that Spark cannot read Enron data given that it's such a > quintessential example. > So it takes 1 hour to output a line "1,227,645 input paths to process", it > then takes another hour to output the same line. Then it outputs a CSV of all > the input paths (so creates a text storm). > Now it's been stuck on the following: > {code} > 17/06/19 09:31:07 INFO LzoCodec: Successfully loaded & initialized native-lzo > library [hadoop-lzo rev 154f1ef53e2d6ed126b0957d7995e0a610947608] > {code} > for 2.5 hours. > So I've provided full reproduce steps here (including code and cluster setup) > https://github.com/samthebest/scenron, scroll down to "Bug In Spark". You can > easily just clone, and follow the README to reproduce exactly! -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21137) Spark reads many small files slowly
[ https://issues.apache.org/jira/browse/SPARK-21137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065428#comment-16065428 ] Steve Loughran commented on SPARK-21137: ps, for now, do it in parallel: {{mapreduce.input.fileinputformat.list-status.num-threads}} . There though, the fact that every thread will be exec()ing code can make it expensive (looks like a full Posix spawn on a mac. > Spark reads many small files slowly > --- > > Key: SPARK-21137 > URL: https://issues.apache.org/jira/browse/SPARK-21137 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 2.1.1 >Reporter: sam >Priority: Minor > > A very common use case in big data is to read a large number of small files. > For example the Enron email dataset has 1,227,645 small files. > When one tries to read this data using Spark one will hit many issues. > Firstly, even if the data is small (each file only say 1K) any job can take a > very long time (I have a simple job that has been running for 3 hours and has > not yet got to the point of starting any tasks, I doubt if it will ever > finish). > It seems all the code in Spark that manages file listing is single threaded > and not well optimised. When I hand crank the code and don't use Spark, my > job runs much faster. > Is it possible that I'm missing some configuration option? It seems kinda > surprising to me that Spark cannot read Enron data given that it's such a > quintessential example. > So it takes 1 hour to output a line "1,227,645 input paths to process", it > then takes another hour to output the same line. Then it outputs a CSV of all > the input paths (so creates a text storm). > Now it's been stuck on the following: > {code} > 17/06/19 09:31:07 INFO LzoCodec: Successfully loaded & initialized native-lzo > library [hadoop-lzo rev 154f1ef53e2d6ed126b0957d7995e0a610947608] > {code} > for 2.5 hours. > So I've provided full reproduce steps here (including code and cluster setup) > https://github.com/samthebest/scenron, scroll down to "Bug In Spark". You can > easily just clone, and follow the README to reproduce exactly! -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21137) Spark reads many small files slowly
[ https://issues.apache.org/jira/browse/SPARK-21137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065445#comment-16065445 ] Steve Loughran commented on SPARK-21137: Filed HADOOP-14600. Looks like a v. old codepath that's not been looked at...and since then the native lib fstat call should be able to do this, just retain the old code for when {{NativeCodeLoader.isNativeCodeLoaded() == false}}. Sam, you said bq. it's likely that the underlying Hadoop APIs have some yucky code that does something silly, I have delved down their before and my stomach cannot handle it oh, it's not so bad. At least you don't have to go near the assembly code bit. That we are all scared of. Or worse, Kerberos. Anyway, patches welcome there, with tests > Spark reads many small files slowly > --- > > Key: SPARK-21137 > URL: https://issues.apache.org/jira/browse/SPARK-21137 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 2.1.1 >Reporter: sam >Priority: Minor > > A very common use case in big data is to read a large number of small files. > For example the Enron email dataset has 1,227,645 small files. > When one tries to read this data using Spark one will hit many issues. > Firstly, even if the data is small (each file only say 1K) any job can take a > very long time (I have a simple job that has been running for 3 hours and has > not yet got to the point of starting any tasks, I doubt if it will ever > finish). > It seems all the code in Spark that manages file listing is single threaded > and not well optimised. When I hand crank the code and don't use Spark, my > job runs much faster. > Is it possible that I'm missing some configuration option? It seems kinda > surprising to me that Spark cannot read Enron data given that it's such a > quintessential example. > So it takes 1 hour to output a line "1,227,645 input paths to process", it > then takes another hour to output the same line. Then it outputs a CSV of all > the input paths (so creates a text storm). > Now it's been stuck on the following: > {code} > 17/06/19 09:31:07 INFO LzoCodec: Successfully loaded & initialized native-lzo > library [hadoop-lzo rev 154f1ef53e2d6ed126b0957d7995e0a610947608] > {code} > for 2.5 hours. > So I've provided full reproduce steps here (including code and cluster setup) > https://github.com/samthebest/scenron, scroll down to "Bug In Spark". You can > easily just clone, and follow the README to reproduce exactly! -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12868) ADD JAR via sparkSQL JDBC will fail when using a HDFS URL
[ https://issues.apache.org/jira/browse/SPARK-12868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065448#comment-16065448 ] Steve Loughran commented on SPARK-12868: I think this is the case of HADOOP-14598: once the FS has been set to {{FsUrlStreamHandlerFactory}} in {{org.apache.spark.sql.internal.SharedState}}, you can't talk to Azure. > ADD JAR via sparkSQL JDBC will fail when using a HDFS URL > - > > Key: SPARK-12868 > URL: https://issues.apache.org/jira/browse/SPARK-12868 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.6.0 >Reporter: Trystan Leftwich >Assignee: Weiqing Yang > Fix For: 2.2.0 > > > When trying to add a jar with a HDFS URI, i.E > {code:sql} > ADD JAR hdfs:///tmp/foo.jar > {code} > Via the spark sql JDBC interface it will fail with: > {code:sql} > java.net.MalformedURLException: unknown protocol: hdfs > at java.net.URL.(URL.java:593) > at java.net.URL.(URL.java:483) > at java.net.URL.(URL.java:432) > at java.net.URI.toURL(URI.java:1089) > at > org.apache.spark.sql.hive.client.ClientWrapper.addJar(ClientWrapper.scala:578) > at org.apache.spark.sql.hive.HiveContext.addJar(HiveContext.scala:652) > at org.apache.spark.sql.hive.execution.AddJar.run(commands.scala:89) > at > org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:58) > at > org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:56) > at > org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:70) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:132) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:130) > at > org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150) > at > org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130) > at > org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:55) > at > org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:55) > at org.apache.spark.sql.DataFrame.(DataFrame.scala:145) > at org.apache.spark.sql.DataFrame.(DataFrame.scala:130) > at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:52) > at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:817) > at > org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:211) > at > org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:154) > at > org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:151) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at > org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1.run(SparkExecuteStatementOperation.scala:164) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-21137) Spark reads many small files slowly
[ https://issues.apache.org/jira/browse/SPARK-21137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21137: Assignee: Apache Spark > Spark reads many small files slowly > --- > > Key: SPARK-21137 > URL: https://issues.apache.org/jira/browse/SPARK-21137 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 2.1.1 >Reporter: sam >Assignee: Apache Spark >Priority: Minor > > A very common use case in big data is to read a large number of small files. > For example the Enron email dataset has 1,227,645 small files. > When one tries to read this data using Spark one will hit many issues. > Firstly, even if the data is small (each file only say 1K) any job can take a > very long time (I have a simple job that has been running for 3 hours and has > not yet got to the point of starting any tasks, I doubt if it will ever > finish). > It seems all the code in Spark that manages file listing is single threaded > and not well optimised. When I hand crank the code and don't use Spark, my > job runs much faster. > Is it possible that I'm missing some configuration option? It seems kinda > surprising to me that Spark cannot read Enron data given that it's such a > quintessential example. > So it takes 1 hour to output a line "1,227,645 input paths to process", it > then takes another hour to output the same line. Then it outputs a CSV of all > the input paths (so creates a text storm). > Now it's been stuck on the following: > {code} > 17/06/19 09:31:07 INFO LzoCodec: Successfully loaded & initialized native-lzo > library [hadoop-lzo rev 154f1ef53e2d6ed126b0957d7995e0a610947608] > {code} > for 2.5 hours. > So I've provided full reproduce steps here (including code and cluster setup) > https://github.com/samthebest/scenron, scroll down to "Bug In Spark". You can > easily just clone, and follow the README to reproduce exactly! -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21137) Spark reads many small files slowly
[ https://issues.apache.org/jira/browse/SPARK-21137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065476#comment-16065476 ] Apache Spark commented on SPARK-21137: -- User 'srowen' has created a pull request for this issue: https://github.com/apache/spark/pull/18441 > Spark reads many small files slowly > --- > > Key: SPARK-21137 > URL: https://issues.apache.org/jira/browse/SPARK-21137 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 2.1.1 >Reporter: sam >Priority: Minor > > A very common use case in big data is to read a large number of small files. > For example the Enron email dataset has 1,227,645 small files. > When one tries to read this data using Spark one will hit many issues. > Firstly, even if the data is small (each file only say 1K) any job can take a > very long time (I have a simple job that has been running for 3 hours and has > not yet got to the point of starting any tasks, I doubt if it will ever > finish). > It seems all the code in Spark that manages file listing is single threaded > and not well optimised. When I hand crank the code and don't use Spark, my > job runs much faster. > Is it possible that I'm missing some configuration option? It seems kinda > surprising to me that Spark cannot read Enron data given that it's such a > quintessential example. > So it takes 1 hour to output a line "1,227,645 input paths to process", it > then takes another hour to output the same line. Then it outputs a CSV of all > the input paths (so creates a text storm). > Now it's been stuck on the following: > {code} > 17/06/19 09:31:07 INFO LzoCodec: Successfully loaded & initialized native-lzo > library [hadoop-lzo rev 154f1ef53e2d6ed126b0957d7995e0a610947608] > {code} > for 2.5 hours. > So I've provided full reproduce steps here (including code and cluster setup) > https://github.com/samthebest/scenron, scroll down to "Bug In Spark". You can > easily just clone, and follow the README to reproduce exactly! -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21137) Spark reads many small files slowly
[ https://issues.apache.org/jira/browse/SPARK-21137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065475#comment-16065475 ] Sean Owen commented on SPARK-21137: --- OK, so it is something that could be optimized in the Hadoop API, and seems somewhat specific to a local filesystem. I opened the change we both mention here in this thread as a PR. > Spark reads many small files slowly > --- > > Key: SPARK-21137 > URL: https://issues.apache.org/jira/browse/SPARK-21137 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 2.1.1 >Reporter: sam >Priority: Minor > > A very common use case in big data is to read a large number of small files. > For example the Enron email dataset has 1,227,645 small files. > When one tries to read this data using Spark one will hit many issues. > Firstly, even if the data is small (each file only say 1K) any job can take a > very long time (I have a simple job that has been running for 3 hours and has > not yet got to the point of starting any tasks, I doubt if it will ever > finish). > It seems all the code in Spark that manages file listing is single threaded > and not well optimised. When I hand crank the code and don't use Spark, my > job runs much faster. > Is it possible that I'm missing some configuration option? It seems kinda > surprising to me that Spark cannot read Enron data given that it's such a > quintessential example. > So it takes 1 hour to output a line "1,227,645 input paths to process", it > then takes another hour to output the same line. Then it outputs a CSV of all > the input paths (so creates a text storm). > Now it's been stuck on the following: > {code} > 17/06/19 09:31:07 INFO LzoCodec: Successfully loaded & initialized native-lzo > library [hadoop-lzo rev 154f1ef53e2d6ed126b0957d7995e0a610947608] > {code} > for 2.5 hours. > So I've provided full reproduce steps here (including code and cluster setup) > https://github.com/samthebest/scenron, scroll down to "Bug In Spark". You can > easily just clone, and follow the README to reproduce exactly! -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org