[jira] [Commented] (SPARK-16296) add null check for key when create map data in encoder
[ https://issues.apache.org/jira/browse/SPARK-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355130#comment-15355130 ] Apache Spark commented on SPARK-16296: -- User 'cloud-fan' has created a pull request for this issue: https://github.com/apache/spark/pull/13974 > add null check for key when create map data in encoder > -- > > Key: SPARK-16296 > URL: https://issues.apache.org/jira/browse/SPARK-16296 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16296) add null check for key when create map data in encoder
[ https://issues.apache.org/jira/browse/SPARK-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16296: Assignee: Wenchen Fan (was: Apache Spark) > add null check for key when create map data in encoder > -- > > Key: SPARK-16296 > URL: https://issues.apache.org/jira/browse/SPARK-16296 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16296) add null check for key when create map data in encoder
[ https://issues.apache.org/jira/browse/SPARK-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16296: Assignee: Apache Spark (was: Wenchen Fan) > add null check for key when create map data in encoder > -- > > Key: SPARK-16296 > URL: https://issues.apache.org/jira/browse/SPARK-16296 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.0 >Reporter: Wenchen Fan >Assignee: Apache Spark > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16297) Mapping Boolean and string to BIT and NVARCHAR(MAX) for SQL Server jdbc dialect
[ https://issues.apache.org/jira/browse/SPARK-16297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16297: Assignee: (was: Apache Spark) > Mapping Boolean and string to BIT and NVARCHAR(MAX) for SQL Server jdbc > dialect > > > Key: SPARK-16297 > URL: https://issues.apache.org/jira/browse/SPARK-16297 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.6.0 >Reporter: Oussama Mekni > Labels: patch > Original Estimate: 72h > Remaining Estimate: 72h > > Tested with SQLServer 2012 and SQLServer Express: > - Fix mapping of StringType to NVARCHAR(MAX) > - Fix mapping of BooleanTypeto BIT -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16297) Mapping Boolean and string to BIT and NVARCHAR(MAX) for SQL Server jdbc dialect
[ https://issues.apache.org/jira/browse/SPARK-16297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355227#comment-15355227 ] Apache Spark commented on SPARK-16297: -- User 'meknio' has created a pull request for this issue: https://github.com/apache/spark/pull/13944 > Mapping Boolean and string to BIT and NVARCHAR(MAX) for SQL Server jdbc > dialect > > > Key: SPARK-16297 > URL: https://issues.apache.org/jira/browse/SPARK-16297 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.6.0 >Reporter: Oussama Mekni > Labels: patch > Original Estimate: 72h > Remaining Estimate: 72h > > Tested with SQLServer 2012 and SQLServer Express: > - Fix mapping of StringType to NVARCHAR(MAX) > - Fix mapping of BooleanTypeto BIT -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16297) Mapping Boolean and string to BIT and NVARCHAR(MAX) for SQL Server jdbc dialect
[ https://issues.apache.org/jira/browse/SPARK-16297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16297: Assignee: Apache Spark > Mapping Boolean and string to BIT and NVARCHAR(MAX) for SQL Server jdbc > dialect > > > Key: SPARK-16297 > URL: https://issues.apache.org/jira/browse/SPARK-16297 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.6.0 >Reporter: Oussama Mekni >Assignee: Apache Spark > Labels: patch > Original Estimate: 72h > Remaining Estimate: 72h > > Tested with SQLServer 2012 and SQLServer Express: > - Fix mapping of StringType to NVARCHAR(MAX) > - Fix mapping of BooleanTypeto BIT -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16299) Capture errors from R workers in daemon.R to avoid deletion of R session temporary directory
[ https://issues.apache.org/jira/browse/SPARK-16299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355462#comment-15355462 ] Apache Spark commented on SPARK-16299: -- User 'sun-rui' has created a pull request for this issue: https://github.com/apache/spark/pull/13975 > Capture errors from R workers in daemon.R to avoid deletion of R session > temporary directory > > > Key: SPARK-16299 > URL: https://issues.apache.org/jira/browse/SPARK-16299 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 1.6.2 >Reporter: Sun Rui > > Running SparkR unit tests randomly has the following error: > Failed > - > 1. Error: pipeRDD() on RDDs (@test_rdd.R#428) > -- > org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in > stage 792.0 failed 1 times, most recent failure: Lost task 0.0 in stage 792.0 > (TID 1493, localhost): org.apache.spark.SparkException: R computation failed > with > [1] 1 > [1] 1 > [1] 2 > [1] 2 > [1] 3 > [1] 3 > [1] 2 > [1] 2 > [1] 2 > [1] 2 > [1] 2 > [1] 2 > ignoring SIGPIPE signal > Calls: source ... -> lapply -> lapply -> FUN -> writeRaw -> > writeBin > Execution halted > cannot open the connection > Calls: source ... computeFunc -> FUN -> system2 -> writeLines -> file > In addition: Warning message: > In file(con, "w") : > cannot open file '/tmp/Rtmp0Gr1aU/file2de3efc94b3': No such file or > directory > Execution halted > at org.apache.spark.api.r.RRunner.compute(RRunner.scala:108) > at org.apache.spark.api.r.BaseRRDD.compute(RRDD.scala:49) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70) > at org.apache.spark.scheduler.Task.run(Task.scala:85) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > This is related to daemon R worker mode. By default, SparkR launches an R > daemon worker per executor, and forks R workers from the daemon when > necessary. > The problem about forking R worker is that all forked R processes share a > temporary directory, as documented at > https://stat.ethz.ch/R-manual/R-devel/library/base/html/tempfile.html. > When any forked R worker exits either normally or caused by errors, the > cleanup procedure of R will delete the temporary directory. This will affect > the still-running forked R workers because any temporary files created by > them under the temporary directories will be removed together. Also all > future R workers that will be forked from the daemon will be affected if they > use tempdir() or tempfile() to get tempoaray files because they will fail to > create temporary files under the already-deleted session temporary directory. > So in order for the daemon mode to work, this problem should be circumvented. > In current dameon.R, R workers directly exits skipping the cleanup procedure > of R so that the shared temporary directory won't be deleted. > {code} > source(script) > # Set SIGUSR1 so that child can exit > tools::pskill(Sys.getpid(), tools::SIGUSR1) > parallel:::mcexit(0L) > {code} > However, this is a bug in daemon.R, that when there is any execution error in > R workers, the error handling of R will finally go into the cleanup > procedure. So try() should be used in daemon.R to catch any error in R > workers, so that R workers will directly exit. > {code} > try(source(script)) > # Set SIGUSR1 so that child can exit > tools::pskill(Sys.getpid(), tools::SIGUSR1) > parallel:::mcexit(0L) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16299) Capture errors from R workers in daemon.R to avoid deletion of R session temporary directory
[ https://issues.apache.org/jira/browse/SPARK-16299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16299: Assignee: Apache Spark > Capture errors from R workers in daemon.R to avoid deletion of R session > temporary directory > > > Key: SPARK-16299 > URL: https://issues.apache.org/jira/browse/SPARK-16299 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 1.6.2 >Reporter: Sun Rui >Assignee: Apache Spark > > Running SparkR unit tests randomly has the following error: > Failed > - > 1. Error: pipeRDD() on RDDs (@test_rdd.R#428) > -- > org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in > stage 792.0 failed 1 times, most recent failure: Lost task 0.0 in stage 792.0 > (TID 1493, localhost): org.apache.spark.SparkException: R computation failed > with > [1] 1 > [1] 1 > [1] 2 > [1] 2 > [1] 3 > [1] 3 > [1] 2 > [1] 2 > [1] 2 > [1] 2 > [1] 2 > [1] 2 > ignoring SIGPIPE signal > Calls: source ... -> lapply -> lapply -> FUN -> writeRaw -> > writeBin > Execution halted > cannot open the connection > Calls: source ... computeFunc -> FUN -> system2 -> writeLines -> file > In addition: Warning message: > In file(con, "w") : > cannot open file '/tmp/Rtmp0Gr1aU/file2de3efc94b3': No such file or > directory > Execution halted > at org.apache.spark.api.r.RRunner.compute(RRunner.scala:108) > at org.apache.spark.api.r.BaseRRDD.compute(RRDD.scala:49) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70) > at org.apache.spark.scheduler.Task.run(Task.scala:85) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > This is related to daemon R worker mode. By default, SparkR launches an R > daemon worker per executor, and forks R workers from the daemon when > necessary. > The problem about forking R worker is that all forked R processes share a > temporary directory, as documented at > https://stat.ethz.ch/R-manual/R-devel/library/base/html/tempfile.html. > When any forked R worker exits either normally or caused by errors, the > cleanup procedure of R will delete the temporary directory. This will affect > the still-running forked R workers because any temporary files created by > them under the temporary directories will be removed together. Also all > future R workers that will be forked from the daemon will be affected if they > use tempdir() or tempfile() to get tempoaray files because they will fail to > create temporary files under the already-deleted session temporary directory. > So in order for the daemon mode to work, this problem should be circumvented. > In current dameon.R, R workers directly exits skipping the cleanup procedure > of R so that the shared temporary directory won't be deleted. > {code} > source(script) > # Set SIGUSR1 so that child can exit > tools::pskill(Sys.getpid(), tools::SIGUSR1) > parallel:::mcexit(0L) > {code} > However, this is a bug in daemon.R, that when there is any execution error in > R workers, the error handling of R will finally go into the cleanup > procedure. So try() should be used in daemon.R to catch any error in R > workers, so that R workers will directly exit. > {code} > try(source(script)) > # Set SIGUSR1 so that child can exit > tools::pskill(Sys.getpid(), tools::SIGUSR1) > parallel:::mcexit(0L) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16299) Capture errors from R workers in daemon.R to avoid deletion of R session temporary directory
[ https://issues.apache.org/jira/browse/SPARK-16299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16299: Assignee: (was: Apache Spark) > Capture errors from R workers in daemon.R to avoid deletion of R session > temporary directory > > > Key: SPARK-16299 > URL: https://issues.apache.org/jira/browse/SPARK-16299 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 1.6.2 >Reporter: Sun Rui > > Running SparkR unit tests randomly has the following error: > Failed > - > 1. Error: pipeRDD() on RDDs (@test_rdd.R#428) > -- > org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in > stage 792.0 failed 1 times, most recent failure: Lost task 0.0 in stage 792.0 > (TID 1493, localhost): org.apache.spark.SparkException: R computation failed > with > [1] 1 > [1] 1 > [1] 2 > [1] 2 > [1] 3 > [1] 3 > [1] 2 > [1] 2 > [1] 2 > [1] 2 > [1] 2 > [1] 2 > ignoring SIGPIPE signal > Calls: source ... -> lapply -> lapply -> FUN -> writeRaw -> > writeBin > Execution halted > cannot open the connection > Calls: source ... computeFunc -> FUN -> system2 -> writeLines -> file > In addition: Warning message: > In file(con, "w") : > cannot open file '/tmp/Rtmp0Gr1aU/file2de3efc94b3': No such file or > directory > Execution halted > at org.apache.spark.api.r.RRunner.compute(RRunner.scala:108) > at org.apache.spark.api.r.BaseRRDD.compute(RRDD.scala:49) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70) > at org.apache.spark.scheduler.Task.run(Task.scala:85) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > This is related to daemon R worker mode. By default, SparkR launches an R > daemon worker per executor, and forks R workers from the daemon when > necessary. > The problem about forking R worker is that all forked R processes share a > temporary directory, as documented at > https://stat.ethz.ch/R-manual/R-devel/library/base/html/tempfile.html. > When any forked R worker exits either normally or caused by errors, the > cleanup procedure of R will delete the temporary directory. This will affect > the still-running forked R workers because any temporary files created by > them under the temporary directories will be removed together. Also all > future R workers that will be forked from the daemon will be affected if they > use tempdir() or tempfile() to get tempoaray files because they will fail to > create temporary files under the already-deleted session temporary directory. > So in order for the daemon mode to work, this problem should be circumvented. > In current dameon.R, R workers directly exits skipping the cleanup procedure > of R so that the shared temporary directory won't be deleted. > {code} > source(script) > # Set SIGUSR1 so that child can exit > tools::pskill(Sys.getpid(), tools::SIGUSR1) > parallel:::mcexit(0L) > {code} > However, this is a bug in daemon.R, that when there is any execution error in > R workers, the error handling of R will finally go into the cleanup > procedure. So try() should be used in daemon.R to catch any error in R > workers, so that R workers will directly exit. > {code} > try(source(script)) > # Set SIGUSR1 so that child can exit > tools::pskill(Sys.getpid(), tools::SIGUSR1) > parallel:::mcexit(0L) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16299) Capture errors from R workers in daemon.R to avoid deletion of R session temporary directory
[ https://issues.apache.org/jira/browse/SPARK-16299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16299: Assignee: Apache Spark > Capture errors from R workers in daemon.R to avoid deletion of R session > temporary directory > > > Key: SPARK-16299 > URL: https://issues.apache.org/jira/browse/SPARK-16299 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 1.6.2 >Reporter: Sun Rui >Assignee: Apache Spark > > Running SparkR unit tests randomly has the following error: > Failed > - > 1. Error: pipeRDD() on RDDs (@test_rdd.R#428) > -- > org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in > stage 792.0 failed 1 times, most recent failure: Lost task 0.0 in stage 792.0 > (TID 1493, localhost): org.apache.spark.SparkException: R computation failed > with > [1] 1 > [1] 1 > [1] 2 > [1] 2 > [1] 3 > [1] 3 > [1] 2 > [1] 2 > [1] 2 > [1] 2 > [1] 2 > [1] 2 > ignoring SIGPIPE signal > Calls: source ... -> lapply -> lapply -> FUN -> writeRaw -> > writeBin > Execution halted > cannot open the connection > Calls: source ... computeFunc -> FUN -> system2 -> writeLines -> file > In addition: Warning message: > In file(con, "w") : > cannot open file '/tmp/Rtmp0Gr1aU/file2de3efc94b3': No such file or > directory > Execution halted > at org.apache.spark.api.r.RRunner.compute(RRunner.scala:108) > at org.apache.spark.api.r.BaseRRDD.compute(RRDD.scala:49) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70) > at org.apache.spark.scheduler.Task.run(Task.scala:85) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > This is related to daemon R worker mode. By default, SparkR launches an R > daemon worker per executor, and forks R workers from the daemon when > necessary. > The problem about forking R worker is that all forked R processes share a > temporary directory, as documented at > https://stat.ethz.ch/R-manual/R-devel/library/base/html/tempfile.html. > When any forked R worker exits either normally or caused by errors, the > cleanup procedure of R will delete the temporary directory. This will affect > the still-running forked R workers because any temporary files created by > them under the temporary directories will be removed together. Also all > future R workers that will be forked from the daemon will be affected if they > use tempdir() or tempfile() to get tempoaray files because they will fail to > create temporary files under the already-deleted session temporary directory. > So in order for the daemon mode to work, this problem should be circumvented. > In current dameon.R, R workers directly exits skipping the cleanup procedure > of R so that the shared temporary directory won't be deleted. > {code} > source(script) > # Set SIGUSR1 so that child can exit > tools::pskill(Sys.getpid(), tools::SIGUSR1) > parallel:::mcexit(0L) > {code} > However, this is a bug in daemon.R, that when there is any execution error in > R workers, the error handling of R will finally go into the cleanup > procedure. So try() should be used in daemon.R to catch any error in R > workers, so that R workers will directly exit. > {code} > try(source(script)) > # Set SIGUSR1 so that child can exit > tools::pskill(Sys.getpid(), tools::SIGUSR1) > parallel:::mcexit(0L) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16299) Capture errors from R workers in daemon.R to avoid deletion of R session temporary directory
[ https://issues.apache.org/jira/browse/SPARK-16299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16299: Assignee: (was: Apache Spark) > Capture errors from R workers in daemon.R to avoid deletion of R session > temporary directory > > > Key: SPARK-16299 > URL: https://issues.apache.org/jira/browse/SPARK-16299 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 1.6.2 >Reporter: Sun Rui > > Running SparkR unit tests randomly has the following error: > Failed > - > 1. Error: pipeRDD() on RDDs (@test_rdd.R#428) > -- > org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in > stage 792.0 failed 1 times, most recent failure: Lost task 0.0 in stage 792.0 > (TID 1493, localhost): org.apache.spark.SparkException: R computation failed > with > [1] 1 > [1] 1 > [1] 2 > [1] 2 > [1] 3 > [1] 3 > [1] 2 > [1] 2 > [1] 2 > [1] 2 > [1] 2 > [1] 2 > ignoring SIGPIPE signal > Calls: source ... -> lapply -> lapply -> FUN -> writeRaw -> > writeBin > Execution halted > cannot open the connection > Calls: source ... computeFunc -> FUN -> system2 -> writeLines -> file > In addition: Warning message: > In file(con, "w") : > cannot open file '/tmp/Rtmp0Gr1aU/file2de3efc94b3': No such file or > directory > Execution halted > at org.apache.spark.api.r.RRunner.compute(RRunner.scala:108) > at org.apache.spark.api.r.BaseRRDD.compute(RRDD.scala:49) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70) > at org.apache.spark.scheduler.Task.run(Task.scala:85) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > This is related to daemon R worker mode. By default, SparkR launches an R > daemon worker per executor, and forks R workers from the daemon when > necessary. > The problem about forking R worker is that all forked R processes share a > temporary directory, as documented at > https://stat.ethz.ch/R-manual/R-devel/library/base/html/tempfile.html. > When any forked R worker exits either normally or caused by errors, the > cleanup procedure of R will delete the temporary directory. This will affect > the still-running forked R workers because any temporary files created by > them under the temporary directories will be removed together. Also all > future R workers that will be forked from the daemon will be affected if they > use tempdir() or tempfile() to get tempoaray files because they will fail to > create temporary files under the already-deleted session temporary directory. > So in order for the daemon mode to work, this problem should be circumvented. > In current dameon.R, R workers directly exits skipping the cleanup procedure > of R so that the shared temporary directory won't be deleted. > {code} > source(script) > # Set SIGUSR1 so that child can exit > tools::pskill(Sys.getpid(), tools::SIGUSR1) > parallel:::mcexit(0L) > {code} > However, this is a bug in daemon.R, that when there is any execution error in > R workers, the error handling of R will finally go into the cleanup > procedure. So try() should be used in daemon.R to catch any error in R > workers, so that R workers will directly exit. > {code} > try(source(script)) > # Set SIGUSR1 so that child can exit > tools::pskill(Sys.getpid(), tools::SIGUSR1) > parallel:::mcexit(0L) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16288) Implement inline table generating function
[ https://issues.apache.org/jira/browse/SPARK-16288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16288: Assignee: (was: Apache Spark) > Implement inline table generating function > -- > > Key: SPARK-16288 > URL: https://issues.apache.org/jira/browse/SPARK-16288 > Project: Spark > Issue Type: Sub-task > Components: SQL >Reporter: Reynold Xin > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16288) Implement inline table generating function
[ https://issues.apache.org/jira/browse/SPARK-16288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16288: Assignee: Apache Spark > Implement inline table generating function > -- > > Key: SPARK-16288 > URL: https://issues.apache.org/jira/browse/SPARK-16288 > Project: Spark > Issue Type: Sub-task > Components: SQL >Reporter: Reynold Xin >Assignee: Apache Spark > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16288) Implement inline table generating function
[ https://issues.apache.org/jira/browse/SPARK-16288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355560#comment-15355560 ] Apache Spark commented on SPARK-16288: -- User 'dongjoon-hyun' has created a pull request for this issue: https://github.com/apache/spark/pull/13976 > Implement inline table generating function > -- > > Key: SPARK-16288 > URL: https://issues.apache.org/jira/browse/SPARK-16288 > Project: Spark > Issue Type: Sub-task > Components: SQL >Reporter: Reynold Xin > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16301) Analyzer rule for resolving using joins should respect case sensitivity setting
[ https://issues.apache.org/jira/browse/SPARK-16301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355596#comment-15355596 ] Apache Spark commented on SPARK-16301: -- User 'yhuai' has created a pull request for this issue: https://github.com/apache/spark/pull/13977 > Analyzer rule for resolving using joins should respect case sensitivity > setting > --- > > Key: SPARK-16301 > URL: https://issues.apache.org/jira/browse/SPARK-16301 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: Yin Huai >Assignee: Yin Huai >Priority: Critical > > Quick repro: Passes on Spark 1.6.x, but fails on 2.0 > {code} > case class MyColumn(userId: Int, field: String) > val ds = Seq(MyColumn(1, "a")).toDF > ds.join(ds, Seq("userid")) > {code} > {code} > stacktrace: > java.util.NoSuchElementException: None.get > at scala.None$.get(Option.scala:313) > at scala.None$.get(Option.scala:311) > at > org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$88.apply(Analyzer.scala:1844) > at > org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$88.apply(Analyzer.scala:1844) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16256) Add Structured Streaming Programming Guide
[ https://issues.apache.org/jira/browse/SPARK-16256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355629#comment-15355629 ] Apache Spark commented on SPARK-16256: -- User 'tdas' has created a pull request for this issue: https://github.com/apache/spark/pull/13978 > Add Structured Streaming Programming Guide > -- > > Key: SPARK-16256 > URL: https://issues.apache.org/jira/browse/SPARK-16256 > Project: Spark > Issue Type: Sub-task > Components: SQL, Streaming >Reporter: Tathagata Das >Assignee: Tathagata Das > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16302) Set the right number of partitions for reading data from a local collection.
[ https://issues.apache.org/jira/browse/SPARK-16302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16302: Assignee: Apache Spark > Set the right number of partitions for reading data from a local collection. > > > Key: SPARK-16302 > URL: https://issues.apache.org/jira/browse/SPARK-16302 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: Lianhui Wang >Assignee: Apache Spark > > query: val df = Seq[(Int, Int)]().toDF("key", "value").count it always use > defaultParallelism tasks. So i cause run empty or small tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16302) Set the right number of partitions for reading data from a local collection.
[ https://issues.apache.org/jira/browse/SPARK-16302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355662#comment-15355662 ] Apache Spark commented on SPARK-16302: -- User 'lianhuiwang' has created a pull request for this issue: https://github.com/apache/spark/pull/13979 > Set the right number of partitions for reading data from a local collection. > > > Key: SPARK-16302 > URL: https://issues.apache.org/jira/browse/SPARK-16302 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: Lianhui Wang > > query: val df = Seq[(Int, Int)]().toDF("key", "value").count it always use > defaultParallelism tasks. So i cause run empty or small tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16302) Set the right number of partitions for reading data from a local collection.
[ https://issues.apache.org/jira/browse/SPARK-16302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16302: Assignee: (was: Apache Spark) > Set the right number of partitions for reading data from a local collection. > > > Key: SPARK-16302 > URL: https://issues.apache.org/jira/browse/SPARK-16302 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: Lianhui Wang > > query: val df = Seq[(Int, Int)]().toDF("key", "value").count it always use > defaultParallelism tasks. So i cause run empty or small tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16198) Change the access level of the predict method in spark.ml.Predictor to public
[ https://issues.apache.org/jira/browse/SPARK-16198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16198: Assignee: (was: Apache Spark) > Change the access level of the predict method in spark.ml.Predictor to public > - > > Key: SPARK-16198 > URL: https://issues.apache.org/jira/browse/SPARK-16198 > Project: Spark > Issue Type: Improvement > Components: ML >Reporter: Hussein Hazimeh >Priority: Minor > Labels: latency, performance > > h1. Summary > The transform method of predictors in spark.ml has a relatively high latency > when predicting single instances or small batches, which is mainly due to the > overhead introduced by DataFrame operations. For a text classification task > on the RCV1 datatset, changing the access level of the low-level "predict" > method from protected to public and using it to make predictions reduced the > latency of single predictions by three to four folds and that of batches by > 50%. While the transform method is flexible and sufficient for general usage, > exposing the low-level predict method to the public API can benefit many > applications that require low-latency response. > h1. Experiment > I performed an experiment to measure the latency of single instance > predictions in Spark and some other popular ML toolkits. Specifically, I'm > looking at the the time it takes to predict or classify a feature vector > residing in memory after the model is trained. > For each toolkit in the table below, logistic regression was trained on the > Reuters RCV1 dataset which contains 697,641 documents and 47,236 features > stored in LIBSVM format along with binary labels. Then the wall-clock time > required to classify each document in a sample of 100,000 documents is > measured, and the 50th, 90th, and 99th percentiles and the maximum time are > reported. > All toolkits were tested on a desktop machine with an i7-6700 processor and > 16 GB memory, running Ubuntu 14.04 and OpenBLAS. The wall clock resolution is > 80ns for Python and 20ns for Scala. > h1. Results > The table below shows the latency of predictions for single instances in > milliseconds, sorted by P90. Spark and Spark 2 refer to versions 1.6.1 and > 2.0.0-SNAPSHOT (on master), respectively. In {color:blue}Spark > (Modified){color} and {color:blue}Spark 2 (Modified){color}, I changed the > access level of the predict method from protected to public and used it to > perform the predictions instead of transform. > ||Toolkit||API||P50||P90||P99||Max|| > |Spark|MLLIB (Scala)|0.0002|0.0015|0.0028|0.0685| > |{color:blue}Spark 2 (Modified){color}|{color:blue}ML > (Scala){color}|0.0004|0.0031|0.0087|0.3979| > |{color:blue}Spark (Modified){color}|{color:blue}ML > (Scala){color}|0.0013|0.0061|0.0632|0.4924| > |Spark|MLLIB (Python)|0.0065|0.0075|0.0212|0.1579| > |Scikit-Learn|Python|0.0341|0.0460|0.0849|0.2285| > |LIBLINEAR|Python|0.0669|0.1484|0.2623|1.7322| > |{color:red}Spark{color}|{color:red}ML > (Scala){color}|2.3713|2.9577|4.6521|511.2448| > |{color:red}Spark 2{color}|{color:red}ML > (Scala){color}|8.4603|9.4352|13.2143|292.8733| > |BIDMach (CPU)|Scala|5.4061|49.1362|102.2563|12040.6773| > |BIDMach (GPU)|Scala|471.3460|477.8214|485.9805|807.4782| > The results show that spark.mllib has the lowest latency among all other > toolkits and APIs, and this can be attributed to its low-level prediction > function that operates directly on the feature vector. However, spark.ml has > a relatively high latency which is in the order of 3ms for Spark 1.6.1 and > 10ms for Spark 2.0.0. Profiling the transform method of logistic regression > in spark.ml showed that only 0.01% of the time is being spent in doing the > dot product and logit transformation, while the rest of the time is dominated > by the DataFrame operations (mostly the “withColumn” operation that appends > the predictions column(s) to the input DataFrame). The results of the > modified versions of spark.ml, which directly use the predict method, > validate this observation as the latency is reduced by three to four folds. > Since Spark splits batch predictions into a series of single-instance > predictions, reducing the latency of single predictions can lead to lower > latencies in batch predictions. I tried batch predictions in spark.ml (1.6.1) > using testing_features.map(x => model.predict( x)).collect() instead of > model.transform(testing_dataframe).select(“prediction”).collect(), and the > former had roughly 50% less latency for batches of size 1000, 10,000, and > 100,000. > Although the experiment is constrained to logistic regression, other > predictors in the classification, regression, and clustering modules can > suffer from the same problem a
[jira] [Assigned] (SPARK-16198) Change the access level of the predict method in spark.ml.Predictor to public
[ https://issues.apache.org/jira/browse/SPARK-16198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16198: Assignee: Apache Spark > Change the access level of the predict method in spark.ml.Predictor to public > - > > Key: SPARK-16198 > URL: https://issues.apache.org/jira/browse/SPARK-16198 > Project: Spark > Issue Type: Improvement > Components: ML >Reporter: Hussein Hazimeh >Assignee: Apache Spark >Priority: Minor > Labels: latency, performance > > h1. Summary > The transform method of predictors in spark.ml has a relatively high latency > when predicting single instances or small batches, which is mainly due to the > overhead introduced by DataFrame operations. For a text classification task > on the RCV1 datatset, changing the access level of the low-level "predict" > method from protected to public and using it to make predictions reduced the > latency of single predictions by three to four folds and that of batches by > 50%. While the transform method is flexible and sufficient for general usage, > exposing the low-level predict method to the public API can benefit many > applications that require low-latency response. > h1. Experiment > I performed an experiment to measure the latency of single instance > predictions in Spark and some other popular ML toolkits. Specifically, I'm > looking at the the time it takes to predict or classify a feature vector > residing in memory after the model is trained. > For each toolkit in the table below, logistic regression was trained on the > Reuters RCV1 dataset which contains 697,641 documents and 47,236 features > stored in LIBSVM format along with binary labels. Then the wall-clock time > required to classify each document in a sample of 100,000 documents is > measured, and the 50th, 90th, and 99th percentiles and the maximum time are > reported. > All toolkits were tested on a desktop machine with an i7-6700 processor and > 16 GB memory, running Ubuntu 14.04 and OpenBLAS. The wall clock resolution is > 80ns for Python and 20ns for Scala. > h1. Results > The table below shows the latency of predictions for single instances in > milliseconds, sorted by P90. Spark and Spark 2 refer to versions 1.6.1 and > 2.0.0-SNAPSHOT (on master), respectively. In {color:blue}Spark > (Modified){color} and {color:blue}Spark 2 (Modified){color}, I changed the > access level of the predict method from protected to public and used it to > perform the predictions instead of transform. > ||Toolkit||API||P50||P90||P99||Max|| > |Spark|MLLIB (Scala)|0.0002|0.0015|0.0028|0.0685| > |{color:blue}Spark 2 (Modified){color}|{color:blue}ML > (Scala){color}|0.0004|0.0031|0.0087|0.3979| > |{color:blue}Spark (Modified){color}|{color:blue}ML > (Scala){color}|0.0013|0.0061|0.0632|0.4924| > |Spark|MLLIB (Python)|0.0065|0.0075|0.0212|0.1579| > |Scikit-Learn|Python|0.0341|0.0460|0.0849|0.2285| > |LIBLINEAR|Python|0.0669|0.1484|0.2623|1.7322| > |{color:red}Spark{color}|{color:red}ML > (Scala){color}|2.3713|2.9577|4.6521|511.2448| > |{color:red}Spark 2{color}|{color:red}ML > (Scala){color}|8.4603|9.4352|13.2143|292.8733| > |BIDMach (CPU)|Scala|5.4061|49.1362|102.2563|12040.6773| > |BIDMach (GPU)|Scala|471.3460|477.8214|485.9805|807.4782| > The results show that spark.mllib has the lowest latency among all other > toolkits and APIs, and this can be attributed to its low-level prediction > function that operates directly on the feature vector. However, spark.ml has > a relatively high latency which is in the order of 3ms for Spark 1.6.1 and > 10ms for Spark 2.0.0. Profiling the transform method of logistic regression > in spark.ml showed that only 0.01% of the time is being spent in doing the > dot product and logit transformation, while the rest of the time is dominated > by the DataFrame operations (mostly the “withColumn” operation that appends > the predictions column(s) to the input DataFrame). The results of the > modified versions of spark.ml, which directly use the predict method, > validate this observation as the latency is reduced by three to four folds. > Since Spark splits batch predictions into a series of single-instance > predictions, reducing the latency of single predictions can lead to lower > latencies in batch predictions. I tried batch predictions in spark.ml (1.6.1) > using testing_features.map(x => model.predict( x)).collect() instead of > model.transform(testing_dataframe).select(“prediction”).collect(), and the > former had roughly 50% less latency for batches of size 1000, 10,000, and > 100,000. > Although the experiment is constrained to logistic regression, other > predictors in the classification, regression, and clustering modules can > suffe
[jira] [Commented] (SPARK-16198) Change the access level of the predict method in spark.ml.Predictor to public
[ https://issues.apache.org/jira/browse/SPARK-16198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355687#comment-15355687 ] Apache Spark commented on SPARK-16198: -- User 'husseinhazimeh' has created a pull request for this issue: https://github.com/apache/spark/pull/13980 > Change the access level of the predict method in spark.ml.Predictor to public > - > > Key: SPARK-16198 > URL: https://issues.apache.org/jira/browse/SPARK-16198 > Project: Spark > Issue Type: Improvement > Components: ML >Reporter: Hussein Hazimeh >Priority: Minor > Labels: latency, performance > > h1. Summary > The transform method of predictors in spark.ml has a relatively high latency > when predicting single instances or small batches, which is mainly due to the > overhead introduced by DataFrame operations. For a text classification task > on the RCV1 datatset, changing the access level of the low-level "predict" > method from protected to public and using it to make predictions reduced the > latency of single predictions by three to four folds and that of batches by > 50%. While the transform method is flexible and sufficient for general usage, > exposing the low-level predict method to the public API can benefit many > applications that require low-latency response. > h1. Experiment > I performed an experiment to measure the latency of single instance > predictions in Spark and some other popular ML toolkits. Specifically, I'm > looking at the the time it takes to predict or classify a feature vector > residing in memory after the model is trained. > For each toolkit in the table below, logistic regression was trained on the > Reuters RCV1 dataset which contains 697,641 documents and 47,236 features > stored in LIBSVM format along with binary labels. Then the wall-clock time > required to classify each document in a sample of 100,000 documents is > measured, and the 50th, 90th, and 99th percentiles and the maximum time are > reported. > All toolkits were tested on a desktop machine with an i7-6700 processor and > 16 GB memory, running Ubuntu 14.04 and OpenBLAS. The wall clock resolution is > 80ns for Python and 20ns for Scala. > h1. Results > The table below shows the latency of predictions for single instances in > milliseconds, sorted by P90. Spark and Spark 2 refer to versions 1.6.1 and > 2.0.0-SNAPSHOT (on master), respectively. In {color:blue}Spark > (Modified){color} and {color:blue}Spark 2 (Modified){color}, I changed the > access level of the predict method from protected to public and used it to > perform the predictions instead of transform. > ||Toolkit||API||P50||P90||P99||Max|| > |Spark|MLLIB (Scala)|0.0002|0.0015|0.0028|0.0685| > |{color:blue}Spark 2 (Modified){color}|{color:blue}ML > (Scala){color}|0.0004|0.0031|0.0087|0.3979| > |{color:blue}Spark (Modified){color}|{color:blue}ML > (Scala){color}|0.0013|0.0061|0.0632|0.4924| > |Spark|MLLIB (Python)|0.0065|0.0075|0.0212|0.1579| > |Scikit-Learn|Python|0.0341|0.0460|0.0849|0.2285| > |LIBLINEAR|Python|0.0669|0.1484|0.2623|1.7322| > |{color:red}Spark{color}|{color:red}ML > (Scala){color}|2.3713|2.9577|4.6521|511.2448| > |{color:red}Spark 2{color}|{color:red}ML > (Scala){color}|8.4603|9.4352|13.2143|292.8733| > |BIDMach (CPU)|Scala|5.4061|49.1362|102.2563|12040.6773| > |BIDMach (GPU)|Scala|471.3460|477.8214|485.9805|807.4782| > The results show that spark.mllib has the lowest latency among all other > toolkits and APIs, and this can be attributed to its low-level prediction > function that operates directly on the feature vector. However, spark.ml has > a relatively high latency which is in the order of 3ms for Spark 1.6.1 and > 10ms for Spark 2.0.0. Profiling the transform method of logistic regression > in spark.ml showed that only 0.01% of the time is being spent in doing the > dot product and logit transformation, while the rest of the time is dominated > by the DataFrame operations (mostly the “withColumn” operation that appends > the predictions column(s) to the input DataFrame). The results of the > modified versions of spark.ml, which directly use the predict method, > validate this observation as the latency is reduced by three to four folds. > Since Spark splits batch predictions into a series of single-instance > predictions, reducing the latency of single predictions can lead to lower > latencies in batch predictions. I tried batch predictions in spark.ml (1.6.1) > using testing_features.map(x => model.predict( x)).collect() instead of > model.transform(testing_dataframe).select(“prediction”).collect(), and the > former had roughly 50% less latency for batches of size 1000, 10,000, and > 100,000. > Although the experiment is constrained to logistic regres
[jira] [Commented] (SPARK-16307) Improve testing for DecisionTree variances
[ https://issues.apache.org/jira/browse/SPARK-16307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355896#comment-15355896 ] Apache Spark commented on SPARK-16307: -- User 'MechCoder' has created a pull request for this issue: https://github.com/apache/spark/pull/13981 > Improve testing for DecisionTree variances > -- > > Key: SPARK-16307 > URL: https://issues.apache.org/jira/browse/SPARK-16307 > Project: Spark > Issue Type: Test >Reporter: Manoj Kumar >Priority: Minor > > The current test assumes that Impurity.calculate() returns the variance > correctly. A better approach would be to test if the variance returned equals > the variance that we can manually verify on a toy data and tree. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16307) Improve testing for DecisionTree variances
[ https://issues.apache.org/jira/browse/SPARK-16307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16307: Assignee: Apache Spark > Improve testing for DecisionTree variances > -- > > Key: SPARK-16307 > URL: https://issues.apache.org/jira/browse/SPARK-16307 > Project: Spark > Issue Type: Test >Reporter: Manoj Kumar >Assignee: Apache Spark >Priority: Minor > > The current test assumes that Impurity.calculate() returns the variance > correctly. A better approach would be to test if the variance returned equals > the variance that we can manually verify on a toy data and tree. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16307) Improve testing for DecisionTree variances
[ https://issues.apache.org/jira/browse/SPARK-16307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16307: Assignee: (was: Apache Spark) > Improve testing for DecisionTree variances > -- > > Key: SPARK-16307 > URL: https://issues.apache.org/jira/browse/SPARK-16307 > Project: Spark > Issue Type: Test >Reporter: Manoj Kumar >Priority: Minor > > The current test assumes that Impurity.calculate() returns the variance > correctly. A better approach would be to test if the variance returned equals > the variance that we can manually verify on a toy data and tree. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16304) LinkageError should not crash Spark executor
[ https://issues.apache.org/jira/browse/SPARK-16304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16304: Assignee: Apache Spark > LinkageError should not crash Spark executor > > > Key: SPARK-16304 > URL: https://issues.apache.org/jira/browse/SPARK-16304 > Project: Spark > Issue Type: Bug > Components: Spark Core >Reporter: Reynold Xin >Assignee: Apache Spark > > If we have a linkage error in the user code, Spark executors get killed > immediately. This is not great for user experience. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16021) Zero out freed memory in test to help catch correctness bugs
[ https://issues.apache.org/jira/browse/SPARK-16021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355951#comment-15355951 ] Apache Spark commented on SPARK-16021: -- User 'ericl' has created a pull request for this issue: https://github.com/apache/spark/pull/13983 > Zero out freed memory in test to help catch correctness bugs > > > Key: SPARK-16021 > URL: https://issues.apache.org/jira/browse/SPARK-16021 > Project: Spark > Issue Type: Improvement >Reporter: Eric Liang > > In both on-heap and off-heap modes, it would be helpful to immediately zero > out (or otherwise fill with a sentinel value) memory when an object is > deallocated. > Currently, in on-heap mode, freed memory can be accessed without visible > error if no other consumer has written to the same space. Similarly, off-heap > memory can be accessed without fault if the allocation library has not > released the pages back to the OS. Zeroing out freed memory would make these > errors immediately visible as a correctness problem. > Since this would add some performance overhead, it would make sense to > conf-flag and enable only in test. > cc [~sameerag] [~hvanhovell] -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16304) LinkageError should not crash Spark executor
[ https://issues.apache.org/jira/browse/SPARK-16304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355949#comment-15355949 ] Apache Spark commented on SPARK-16304: -- User 'petermaxlee' has created a pull request for this issue: https://github.com/apache/spark/pull/13982 > LinkageError should not crash Spark executor > > > Key: SPARK-16304 > URL: https://issues.apache.org/jira/browse/SPARK-16304 > Project: Spark > Issue Type: Bug > Components: Spark Core >Reporter: Reynold Xin > > If we have a linkage error in the user code, Spark executors get killed > immediately. This is not great for user experience. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16021) Zero out freed memory in test to help catch correctness bugs
[ https://issues.apache.org/jira/browse/SPARK-16021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16021: Assignee: Apache Spark > Zero out freed memory in test to help catch correctness bugs > > > Key: SPARK-16021 > URL: https://issues.apache.org/jira/browse/SPARK-16021 > Project: Spark > Issue Type: Improvement >Reporter: Eric Liang >Assignee: Apache Spark > > In both on-heap and off-heap modes, it would be helpful to immediately zero > out (or otherwise fill with a sentinel value) memory when an object is > deallocated. > Currently, in on-heap mode, freed memory can be accessed without visible > error if no other consumer has written to the same space. Similarly, off-heap > memory can be accessed without fault if the allocation library has not > released the pages back to the OS. Zeroing out freed memory would make these > errors immediately visible as a correctness problem. > Since this would add some performance overhead, it would make sense to > conf-flag and enable only in test. > cc [~sameerag] [~hvanhovell] -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16021) Zero out freed memory in test to help catch correctness bugs
[ https://issues.apache.org/jira/browse/SPARK-16021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16021: Assignee: Apache Spark > Zero out freed memory in test to help catch correctness bugs > > > Key: SPARK-16021 > URL: https://issues.apache.org/jira/browse/SPARK-16021 > Project: Spark > Issue Type: Improvement >Reporter: Eric Liang >Assignee: Apache Spark > > In both on-heap and off-heap modes, it would be helpful to immediately zero > out (or otherwise fill with a sentinel value) memory when an object is > deallocated. > Currently, in on-heap mode, freed memory can be accessed without visible > error if no other consumer has written to the same space. Similarly, off-heap > memory can be accessed without fault if the allocation library has not > released the pages back to the OS. Zeroing out freed memory would make these > errors immediately visible as a correctness problem. > Since this would add some performance overhead, it would make sense to > conf-flag and enable only in test. > cc [~sameerag] [~hvanhovell] -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16310) SparkR csv source should have the same default na.string as R
[ https://issues.apache.org/jira/browse/SPARK-16310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16310: Assignee: Apache Spark > SparkR csv source should have the same default na.string as R > - > > Key: SPARK-16310 > URL: https://issues.apache.org/jira/browse/SPARK-16310 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 1.6.2 >Reporter: Felix Cheung >Assignee: Apache Spark >Priority: Minor > > https://stat.ethz.ch/R-manual/R-devel/library/utils/html/read.table.html > na.strings = "NA" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16310) SparkR csv source should have the same default na.string as R
[ https://issues.apache.org/jira/browse/SPARK-16310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16310: Assignee: (was: Apache Spark) > SparkR csv source should have the same default na.string as R > - > > Key: SPARK-16310 > URL: https://issues.apache.org/jira/browse/SPARK-16310 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 1.6.2 >Reporter: Felix Cheung >Priority: Minor > > https://stat.ethz.ch/R-manual/R-devel/library/utils/html/read.table.html > na.strings = "NA" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16310) SparkR csv source should have the same default na.string as R
[ https://issues.apache.org/jira/browse/SPARK-16310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355985#comment-15355985 ] Apache Spark commented on SPARK-16310: -- User 'felixcheung' has created a pull request for this issue: https://github.com/apache/spark/pull/13984 > SparkR csv source should have the same default na.string as R > - > > Key: SPARK-16310 > URL: https://issues.apache.org/jira/browse/SPARK-16310 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 1.6.2 >Reporter: Felix Cheung >Priority: Minor > > https://stat.ethz.ch/R-manual/R-devel/library/utils/html/read.table.html > na.strings = "NA" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16114) Add network word count example
[ https://issues.apache.org/jira/browse/SPARK-16114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356255#comment-15356255 ] Apache Spark commented on SPARK-16114: -- User 'jjthomas' has created a pull request for this issue: https://github.com/apache/spark/pull/13957 > Add network word count example > -- > > Key: SPARK-16114 > URL: https://issues.apache.org/jira/browse/SPARK-16114 > Project: Spark > Issue Type: Sub-task > Components: SQL, Streaming >Reporter: James Thomas > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16313) Spark should not silently drop exceptions in file listing
[ https://issues.apache.org/jira/browse/SPARK-16313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356364#comment-15356364 ] Apache Spark commented on SPARK-16313: -- User 'rxin' has created a pull request for this issue: https://github.com/apache/spark/pull/13987 > Spark should not silently drop exceptions in file listing > - > > Key: SPARK-16313 > URL: https://issues.apache.org/jira/browse/SPARK-16313 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: Reynold Xin >Assignee: Reynold Xin >Priority: Critical > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16313) Spark should not silently drop exceptions in file listing
[ https://issues.apache.org/jira/browse/SPARK-16313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16313: Assignee: Apache Spark (was: Reynold Xin) > Spark should not silently drop exceptions in file listing > - > > Key: SPARK-16313 > URL: https://issues.apache.org/jira/browse/SPARK-16313 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: Reynold Xin >Assignee: Apache Spark >Priority: Critical > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16313) Spark should not silently drop exceptions in file listing
[ https://issues.apache.org/jira/browse/SPARK-16313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16313: Assignee: Reynold Xin (was: Apache Spark) > Spark should not silently drop exceptions in file listing > - > > Key: SPARK-16313 > URL: https://issues.apache.org/jira/browse/SPARK-16313 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: Reynold Xin >Assignee: Reynold Xin >Priority: Critical > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16101) Refactoring CSV data source to be consistent with JSON data source
[ https://issues.apache.org/jira/browse/SPARK-16101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16101: Assignee: Apache Spark > Refactoring CSV data source to be consistent with JSON data source > -- > > Key: SPARK-16101 > URL: https://issues.apache.org/jira/browse/SPARK-16101 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 2.0.0 >Reporter: Hyukjin Kwon >Assignee: Apache Spark > > Currently, CSV data source has a pretty much different structure with JSON > data source although they can be pretty much similar. > It would be great if they have the similar structure so that some common > issues can be resolved together. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16101) Refactoring CSV data source to be consistent with JSON data source
[ https://issues.apache.org/jira/browse/SPARK-16101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16101: Assignee: (was: Apache Spark) > Refactoring CSV data source to be consistent with JSON data source > -- > > Key: SPARK-16101 > URL: https://issues.apache.org/jira/browse/SPARK-16101 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 2.0.0 >Reporter: Hyukjin Kwon > > Currently, CSV data source has a pretty much different structure with JSON > data source although they can be pretty much similar. > It would be great if they have the similar structure so that some common > issues can be resolved together. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16101) Refactoring CSV data source to be consistent with JSON data source
[ https://issues.apache.org/jira/browse/SPARK-16101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356393#comment-15356393 ] Apache Spark commented on SPARK-16101: -- User 'HyukjinKwon' has created a pull request for this issue: https://github.com/apache/spark/pull/13988 > Refactoring CSV data source to be consistent with JSON data source > -- > > Key: SPARK-16101 > URL: https://issues.apache.org/jira/browse/SPARK-16101 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 2.0.0 >Reporter: Hyukjin Kwon > > Currently, CSV data source has a pretty much different structure with JSON > data source although they can be pretty much similar. > It would be great if they have the similar structure so that some common > issues can be resolved together. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16311) Improve metadata refresh
[ https://issues.apache.org/jira/browse/SPARK-16311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356516#comment-15356516 ] Apache Spark commented on SPARK-16311: -- User 'petermaxlee' has created a pull request for this issue: https://github.com/apache/spark/pull/13989 > Improve metadata refresh > > > Key: SPARK-16311 > URL: https://issues.apache.org/jira/browse/SPARK-16311 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: Reynold Xin > > When the underlying file changes, it can be very confusing to users when they > see a FileNotFoundException. It would be great to do the following: > (1) Append a message to the FileNotFoundException that a workaround is to do > explicitly metadata refresh. > (2) Make metadata refresh work on temporary tables/views. > (3) Make metadata refresh work on Datasets/DataFrames, by introducing a > Dataset.refresh() method. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16311) Improve metadata refresh
[ https://issues.apache.org/jira/browse/SPARK-16311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16311: Assignee: (was: Apache Spark) > Improve metadata refresh > > > Key: SPARK-16311 > URL: https://issues.apache.org/jira/browse/SPARK-16311 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: Reynold Xin > > When the underlying file changes, it can be very confusing to users when they > see a FileNotFoundException. It would be great to do the following: > (1) Append a message to the FileNotFoundException that a workaround is to do > explicitly metadata refresh. > (2) Make metadata refresh work on temporary tables/views. > (3) Make metadata refresh work on Datasets/DataFrames, by introducing a > Dataset.refresh() method. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16311) Improve metadata refresh
[ https://issues.apache.org/jira/browse/SPARK-16311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16311: Assignee: Apache Spark > Improve metadata refresh > > > Key: SPARK-16311 > URL: https://issues.apache.org/jira/browse/SPARK-16311 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: Reynold Xin >Assignee: Apache Spark > > When the underlying file changes, it can be very confusing to users when they > see a FileNotFoundException. It would be great to do the following: > (1) Append a message to the FileNotFoundException that a workaround is to do > explicitly metadata refresh. > (2) Make metadata refresh work on temporary tables/views. > (3) Make metadata refresh work on Datasets/DataFrames, by introducing a > Dataset.refresh() method. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16287) Implement str_to_map SQL function
[ https://issues.apache.org/jira/browse/SPARK-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16287: Assignee: Apache Spark > Implement str_to_map SQL function > - > > Key: SPARK-16287 > URL: https://issues.apache.org/jira/browse/SPARK-16287 > Project: Spark > Issue Type: Sub-task > Components: SQL >Reporter: Reynold Xin >Assignee: Apache Spark > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16287) Implement str_to_map SQL function
[ https://issues.apache.org/jira/browse/SPARK-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356524#comment-15356524 ] Apache Spark commented on SPARK-16287: -- User 'techaddict' has created a pull request for this issue: https://github.com/apache/spark/pull/13990 > Implement str_to_map SQL function > - > > Key: SPARK-16287 > URL: https://issues.apache.org/jira/browse/SPARK-16287 > Project: Spark > Issue Type: Sub-task > Components: SQL >Reporter: Reynold Xin > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16287) Implement str_to_map SQL function
[ https://issues.apache.org/jira/browse/SPARK-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16287: Assignee: (was: Apache Spark) > Implement str_to_map SQL function > - > > Key: SPARK-16287 > URL: https://issues.apache.org/jira/browse/SPARK-16287 > Project: Spark > Issue Type: Sub-task > Components: SQL >Reporter: Reynold Xin > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16318) xpath_int, xpath_short, xpath_long, xpath_float, xpath_double, xpath_string
[ https://issues.apache.org/jira/browse/SPARK-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16318: Assignee: (was: Apache Spark) > xpath_int, xpath_short, xpath_long, xpath_float, xpath_double, xpath_string > --- > > Key: SPARK-16318 > URL: https://issues.apache.org/jira/browse/SPARK-16318 > Project: Spark > Issue Type: Sub-task > Components: SQL >Reporter: Peter Lee > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16318) xpath_int, xpath_short, xpath_long, xpath_float, xpath_double, xpath_string
[ https://issues.apache.org/jira/browse/SPARK-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356629#comment-15356629 ] Apache Spark commented on SPARK-16318: -- User 'petermaxlee' has created a pull request for this issue: https://github.com/apache/spark/pull/13991 > xpath_int, xpath_short, xpath_long, xpath_float, xpath_double, xpath_string > --- > > Key: SPARK-16318 > URL: https://issues.apache.org/jira/browse/SPARK-16318 > Project: Spark > Issue Type: Sub-task > Components: SQL >Reporter: Peter Lee > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16318) xpath_int, xpath_short, xpath_long, xpath_float, xpath_double, xpath_string
[ https://issues.apache.org/jira/browse/SPARK-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16318: Assignee: Apache Spark > xpath_int, xpath_short, xpath_long, xpath_float, xpath_double, xpath_string > --- > > Key: SPARK-16318 > URL: https://issues.apache.org/jira/browse/SPARK-16318 > Project: Spark > Issue Type: Sub-task > Components: SQL >Reporter: Peter Lee >Assignee: Apache Spark > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12177) Update KafkaDStreams to new Kafka 0.10 Consumer API
[ https://issues.apache.org/jira/browse/SPARK-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356704#comment-15356704 ] Apache Spark commented on SPARK-12177: -- User 'tdas' has created a pull request for this issue: https://github.com/apache/spark/pull/13992 > Update KafkaDStreams to new Kafka 0.10 Consumer API > --- > > Key: SPARK-12177 > URL: https://issues.apache.org/jira/browse/SPARK-12177 > Project: Spark > Issue Type: Improvement > Components: Streaming >Affects Versions: 1.6.0 >Reporter: Nikita Tarasenko >Assignee: Cody Koeninger > Labels: consumer, kafka > Fix For: 2.0.0 > > > Kafka 0.9 already released and it introduce new consumer API that not > compatible with old one. So, I added new consumer api. I made separate > classes in package org.apache.spark.streaming.kafka.v09 with changed API. I > didn't remove old classes for more backward compatibility. User will not need > to change his old spark applications when he uprgade to new Spark version. > Please rewiew my changes -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16144) Add a separate Rd for ML generic methods: read.ml, write.ml, summary, predict
[ https://issues.apache.org/jira/browse/SPARK-16144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16144: Assignee: Apache Spark (was: Yanbo Liang) > Add a separate Rd for ML generic methods: read.ml, write.ml, summary, predict > - > > Key: SPARK-16144 > URL: https://issues.apache.org/jira/browse/SPARK-16144 > Project: Spark > Issue Type: Sub-task > Components: Documentation, MLlib, SparkR >Affects Versions: 2.0.0 >Reporter: Xiangrui Meng >Assignee: Apache Spark > > After we grouped generic methods by the algorithm, it would be nice to add a > separate Rd for each ML generic methods, in particular, write.ml, read.ml, > summary, and predict and link the implementations with seealso. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16144) Add a separate Rd for ML generic methods: read.ml, write.ml, summary, predict
[ https://issues.apache.org/jira/browse/SPARK-16144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16144: Assignee: Yanbo Liang (was: Apache Spark) > Add a separate Rd for ML generic methods: read.ml, write.ml, summary, predict > - > > Key: SPARK-16144 > URL: https://issues.apache.org/jira/browse/SPARK-16144 > Project: Spark > Issue Type: Sub-task > Components: Documentation, MLlib, SparkR >Affects Versions: 2.0.0 >Reporter: Xiangrui Meng >Assignee: Yanbo Liang > > After we grouped generic methods by the algorithm, it would be nice to add a > separate Rd for each ML generic methods, in particular, write.ml, read.ml, > summary, and predict and link the implementations with seealso. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16144) Add a separate Rd for ML generic methods: read.ml, write.ml, summary, predict
[ https://issues.apache.org/jira/browse/SPARK-16144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356742#comment-15356742 ] Apache Spark commented on SPARK-16144: -- User 'felixcheung' has created a pull request for this issue: https://github.com/apache/spark/pull/13993 > Add a separate Rd for ML generic methods: read.ml, write.ml, summary, predict > - > > Key: SPARK-16144 > URL: https://issues.apache.org/jira/browse/SPARK-16144 > Project: Spark > Issue Type: Sub-task > Components: Documentation, MLlib, SparkR >Affects Versions: 2.0.0 >Reporter: Xiangrui Meng >Assignee: Yanbo Liang > > After we grouped generic methods by the algorithm, it would be nice to add a > separate Rd for each ML generic methods, in particular, write.ml, read.ml, > summary, and predict and link the implementations with seealso. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12177) Update KafkaDStreams to new Kafka 0.10 Consumer API
[ https://issues.apache.org/jira/browse/SPARK-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356930#comment-15356930 ] Apache Spark commented on SPARK-12177: -- User 'tdas' has created a pull request for this issue: https://github.com/apache/spark/pull/13996 > Update KafkaDStreams to new Kafka 0.10 Consumer API > --- > > Key: SPARK-12177 > URL: https://issues.apache.org/jira/browse/SPARK-12177 > Project: Spark > Issue Type: Improvement > Components: Streaming >Affects Versions: 1.6.0 >Reporter: Nikita Tarasenko >Assignee: Cody Koeninger > Labels: consumer, kafka > Fix For: 2.0.0 > > > Kafka 0.9 already released and it introduce new consumer API that not > compatible with old one. So, I added new consumer api. I made separate > classes in package org.apache.spark.streaming.kafka.v09 with changed API. I > didn't remove old classes for more backward compatibility. User will not need > to change his old spark applications when he uprgade to new Spark version. > Please rewiew my changes -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16328) Implement conversion utility functions for single instances in Python
[ https://issues.apache.org/jira/browse/SPARK-16328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16328: Assignee: Apache Spark (was: Nick Pentreath) > Implement conversion utility functions for single instances in Python > - > > Key: SPARK-16328 > URL: https://issues.apache.org/jira/browse/SPARK-16328 > Project: Spark > Issue Type: Sub-task > Components: ML, MLlib, PySpark >Reporter: Nick Pentreath >Assignee: Apache Spark > > We have {{asML}}/{{fromML}} utility methods in Scala/Java to convert between > the old and new linalg types. These are missing in Python. > For dense vectors it's actually easy to do without, e.g. {{mlDenseVector = > Vectors.dense(mllibDenseVector)}}, but for sparse it doesn't work easily. So > it would be good to have utility methods available for users. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16328) Implement conversion utility functions for single instances in Python
[ https://issues.apache.org/jira/browse/SPARK-16328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16328: Assignee: Nick Pentreath (was: Apache Spark) > Implement conversion utility functions for single instances in Python > - > > Key: SPARK-16328 > URL: https://issues.apache.org/jira/browse/SPARK-16328 > Project: Spark > Issue Type: Sub-task > Components: ML, MLlib, PySpark >Reporter: Nick Pentreath >Assignee: Nick Pentreath > > We have {{asML}}/{{fromML}} utility methods in Scala/Java to convert between > the old and new linalg types. These are missing in Python. > For dense vectors it's actually easy to do without, e.g. {{mlDenseVector = > Vectors.dense(mllibDenseVector)}}, but for sparse it doesn't work easily. So > it would be good to have utility methods available for users. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16328) Implement conversion utility functions for single instances in Python
[ https://issues.apache.org/jira/browse/SPARK-16328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15357130#comment-15357130 ] Apache Spark commented on SPARK-16328: -- User 'MLnick' has created a pull request for this issue: https://github.com/apache/spark/pull/13997 > Implement conversion utility functions for single instances in Python > - > > Key: SPARK-16328 > URL: https://issues.apache.org/jira/browse/SPARK-16328 > Project: Spark > Issue Type: Sub-task > Components: ML, MLlib, PySpark >Reporter: Nick Pentreath >Assignee: Nick Pentreath > > We have {{asML}}/{{fromML}} utility methods in Scala/Java to convert between > the old and new linalg types. These are missing in Python. > For dense vectors it's actually easy to do without, e.g. {{mlDenseVector = > Vectors.dense(mllibDenseVector)}}, but for sparse it doesn't work easily. So > it would be good to have utility methods available for users. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12177) Update KafkaDStreams to new Kafka 0.10 Consumer API
[ https://issues.apache.org/jira/browse/SPARK-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15357480#comment-15357480 ] Apache Spark commented on SPARK-12177: -- User 'koeninger' has created a pull request for this issue: https://github.com/apache/spark/pull/13998 > Update KafkaDStreams to new Kafka 0.10 Consumer API > --- > > Key: SPARK-12177 > URL: https://issues.apache.org/jira/browse/SPARK-12177 > Project: Spark > Issue Type: Improvement > Components: Streaming >Affects Versions: 1.6.0 >Reporter: Nikita Tarasenko >Assignee: Cody Koeninger > Labels: consumer, kafka > Fix For: 2.0.0 > > > Kafka 0.9 already released and it introduce new consumer API that not > compatible with old one. So, I added new consumer api. I made separate > classes in package org.apache.spark.streaming.kafka.v09 with changed API. I > didn't remove old classes for more backward compatibility. User will not need > to change his old spark applications when he uprgade to new Spark version. > Please rewiew my changes -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16281) Implement parse_url SQL function
[ https://issues.apache.org/jira/browse/SPARK-16281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16281: Assignee: Apache Spark > Implement parse_url SQL function > > > Key: SPARK-16281 > URL: https://issues.apache.org/jira/browse/SPARK-16281 > Project: Spark > Issue Type: Sub-task > Components: SQL >Reporter: Reynold Xin >Assignee: Apache Spark > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16281) Implement parse_url SQL function
[ https://issues.apache.org/jira/browse/SPARK-16281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15357501#comment-15357501 ] Apache Spark commented on SPARK-16281: -- User 'janplus' has created a pull request for this issue: https://github.com/apache/spark/pull/13999 > Implement parse_url SQL function > > > Key: SPARK-16281 > URL: https://issues.apache.org/jira/browse/SPARK-16281 > Project: Spark > Issue Type: Sub-task > Components: SQL >Reporter: Reynold Xin > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16281) Implement parse_url SQL function
[ https://issues.apache.org/jira/browse/SPARK-16281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16281: Assignee: (was: Apache Spark) > Implement parse_url SQL function > > > Key: SPARK-16281 > URL: https://issues.apache.org/jira/browse/SPARK-16281 > Project: Spark > Issue Type: Sub-task > Components: SQL >Reporter: Reynold Xin > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16331) [SQL] Reduce code generation time
[ https://issues.apache.org/jira/browse/SPARK-16331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16331: Assignee: Apache Spark > [SQL] Reduce code generation time > -- > > Key: SPARK-16331 > URL: https://issues.apache.org/jira/browse/SPARK-16331 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.0.0, 2.1.0 >Reporter: Hiroshi Inoue >Assignee: Apache Spark > > During the code generation, a {{LocalRelation}} often has a huge {{Vector}} > object as {{data}}. In the simple example below, a {{LocalRelation}} has a > Vector with 100 elements of {{UnsafeRow}}. > {quote} > val numRows = 100 > val ds = (1 to numRows).toDS().persist() > benchmark.addCase("filter+reduce") { iter => > ds.filter(a => (a & 1) == 0).reduce(_ + _) > } > {quote} > At {{TreeNode.transformChildren}}, all elements of the vector is > unnecessarily iterated to check whether any children exist in the vector > since {{Vector}} is Traversable. This part significantly increases code > generation time. > This patch avoids this overhead by checking the number of children before > iterating all elements; {{LocalRelation}} does not have children since it > extends {{LeafNode}}. > The performance of the above example > {quote} > without this patch > Java HotSpot(TM) 64-Bit Server VM 1.8.0_91-b14 on Mac OS X 10.11.5 > Intel(R) Core(TM) i5-5257U CPU @ 2.70GHz > compilationTime: Best/Avg Time(ms)Rate(M/s) Per > Row(ns) Relative > > filter+reduce 4426 / 4533 0.2 > 4426.0 1.0X > with this patch > compilationTime: Best/Avg Time(ms)Rate(M/s) Per > Row(ns) Relative > > filter+reduce 3117 / 3391 0.3 > 3116.6 1.0X > {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16331) [SQL] Reduce code generation time
[ https://issues.apache.org/jira/browse/SPARK-16331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16331: Assignee: (was: Apache Spark) > [SQL] Reduce code generation time > -- > > Key: SPARK-16331 > URL: https://issues.apache.org/jira/browse/SPARK-16331 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.0.0, 2.1.0 >Reporter: Hiroshi Inoue > > During the code generation, a {{LocalRelation}} often has a huge {{Vector}} > object as {{data}}. In the simple example below, a {{LocalRelation}} has a > Vector with 100 elements of {{UnsafeRow}}. > {quote} > val numRows = 100 > val ds = (1 to numRows).toDS().persist() > benchmark.addCase("filter+reduce") { iter => > ds.filter(a => (a & 1) == 0).reduce(_ + _) > } > {quote} > At {{TreeNode.transformChildren}}, all elements of the vector is > unnecessarily iterated to check whether any children exist in the vector > since {{Vector}} is Traversable. This part significantly increases code > generation time. > This patch avoids this overhead by checking the number of children before > iterating all elements; {{LocalRelation}} does not have children since it > extends {{LeafNode}}. > The performance of the above example > {quote} > without this patch > Java HotSpot(TM) 64-Bit Server VM 1.8.0_91-b14 on Mac OS X 10.11.5 > Intel(R) Core(TM) i5-5257U CPU @ 2.70GHz > compilationTime: Best/Avg Time(ms)Rate(M/s) Per > Row(ns) Relative > > filter+reduce 4426 / 4533 0.2 > 4426.0 1.0X > with this patch > compilationTime: Best/Avg Time(ms)Rate(M/s) Per > Row(ns) Relative > > filter+reduce 3117 / 3391 0.3 > 3116.6 1.0X > {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16331) [SQL] Reduce code generation time
[ https://issues.apache.org/jira/browse/SPARK-16331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15357519#comment-15357519 ] Apache Spark commented on SPARK-16331: -- User 'inouehrs' has created a pull request for this issue: https://github.com/apache/spark/pull/14000 > [SQL] Reduce code generation time > -- > > Key: SPARK-16331 > URL: https://issues.apache.org/jira/browse/SPARK-16331 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.0.0, 2.1.0 >Reporter: Hiroshi Inoue > > During the code generation, a {{LocalRelation}} often has a huge {{Vector}} > object as {{data}}. In the simple example below, a {{LocalRelation}} has a > Vector with 100 elements of {{UnsafeRow}}. > {quote} > val numRows = 100 > val ds = (1 to numRows).toDS().persist() > benchmark.addCase("filter+reduce") { iter => > ds.filter(a => (a & 1) == 0).reduce(_ + _) > } > {quote} > At {{TreeNode.transformChildren}}, all elements of the vector is > unnecessarily iterated to check whether any children exist in the vector > since {{Vector}} is Traversable. This part significantly increases code > generation time. > This patch avoids this overhead by checking the number of children before > iterating all elements; {{LocalRelation}} does not have children since it > extends {{LeafNode}}. > The performance of the above example > {quote} > without this patch > Java HotSpot(TM) 64-Bit Server VM 1.8.0_91-b14 on Mac OS X 10.11.5 > Intel(R) Core(TM) i5-5257U CPU @ 2.70GHz > compilationTime: Best/Avg Time(ms)Rate(M/s) Per > Row(ns) Relative > > filter+reduce 4426 / 4533 0.2 > 4426.0 1.0X > with this patch > compilationTime: Best/Avg Time(ms)Rate(M/s) Per > Row(ns) Relative > > filter+reduce 3117 / 3391 0.3 > 3116.6 1.0X > {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16256) Add Structured Streaming Programming Guide
[ https://issues.apache.org/jira/browse/SPARK-16256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15357825#comment-15357825 ] Apache Spark commented on SPARK-16256: -- User 'tdas' has created a pull request for this issue: https://github.com/apache/spark/pull/14001 > Add Structured Streaming Programming Guide > -- > > Key: SPARK-16256 > URL: https://issues.apache.org/jira/browse/SPARK-16256 > Project: Spark > Issue Type: Sub-task > Components: SQL, Streaming >Reporter: Tathagata Das >Assignee: Tathagata Das > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16335) Streaming source should fail if file does not exist
[ https://issues.apache.org/jira/browse/SPARK-16335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16335: Assignee: Reynold Xin (was: Apache Spark) > Streaming source should fail if file does not exist > --- > > Key: SPARK-16335 > URL: https://issues.apache.org/jira/browse/SPARK-16335 > Project: Spark > Issue Type: Bug > Components: SQL, Streaming >Reporter: Reynold Xin >Assignee: Reynold Xin > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16335) Streaming source should fail if file does not exist
[ https://issues.apache.org/jira/browse/SPARK-16335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15357876#comment-15357876 ] Apache Spark commented on SPARK-16335: -- User 'rxin' has created a pull request for this issue: https://github.com/apache/spark/pull/14002 > Streaming source should fail if file does not exist > --- > > Key: SPARK-16335 > URL: https://issues.apache.org/jira/browse/SPARK-16335 > Project: Spark > Issue Type: Bug > Components: SQL, Streaming >Reporter: Reynold Xin >Assignee: Reynold Xin > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16335) Streaming source should fail if file does not exist
[ https://issues.apache.org/jira/browse/SPARK-16335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16335: Assignee: Apache Spark (was: Reynold Xin) > Streaming source should fail if file does not exist > --- > > Key: SPARK-16335 > URL: https://issues.apache.org/jira/browse/SPARK-16335 > Project: Spark > Issue Type: Bug > Components: SQL, Streaming >Reporter: Reynold Xin >Assignee: Apache Spark > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16336) Suggest doing table refresh when encountering FileNotFoundException at runtime
[ https://issues.apache.org/jira/browse/SPARK-16336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16336: Assignee: (was: Apache Spark) > Suggest doing table refresh when encountering FileNotFoundException at runtime > -- > > Key: SPARK-16336 > URL: https://issues.apache.org/jira/browse/SPARK-16336 > Project: Spark > Issue Type: Sub-task > Components: SQL >Reporter: Peter Lee > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16336) Suggest doing table refresh when encountering FileNotFoundException at runtime
[ https://issues.apache.org/jira/browse/SPARK-16336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15357926#comment-15357926 ] Apache Spark commented on SPARK-16336: -- User 'petermaxlee' has created a pull request for this issue: https://github.com/apache/spark/pull/14003 > Suggest doing table refresh when encountering FileNotFoundException at runtime > -- > > Key: SPARK-16336 > URL: https://issues.apache.org/jira/browse/SPARK-16336 > Project: Spark > Issue Type: Sub-task > Components: SQL >Reporter: Peter Lee > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16336) Suggest doing table refresh when encountering FileNotFoundException at runtime
[ https://issues.apache.org/jira/browse/SPARK-16336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16336: Assignee: Apache Spark > Suggest doing table refresh when encountering FileNotFoundException at runtime > -- > > Key: SPARK-16336 > URL: https://issues.apache.org/jira/browse/SPARK-16336 > Project: Spark > Issue Type: Sub-task > Components: SQL >Reporter: Peter Lee >Assignee: Apache Spark > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16285) Implement sentences SQL function
[ https://issues.apache.org/jira/browse/SPARK-16285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16285: Assignee: Apache Spark > Implement sentences SQL function > > > Key: SPARK-16285 > URL: https://issues.apache.org/jira/browse/SPARK-16285 > Project: Spark > Issue Type: Sub-task > Components: SQL >Reporter: Reynold Xin >Assignee: Apache Spark > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16285) Implement sentences SQL function
[ https://issues.apache.org/jira/browse/SPARK-16285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15357958#comment-15357958 ] Apache Spark commented on SPARK-16285: -- User 'dongjoon-hyun' has created a pull request for this issue: https://github.com/apache/spark/pull/14004 > Implement sentences SQL function > > > Key: SPARK-16285 > URL: https://issues.apache.org/jira/browse/SPARK-16285 > Project: Spark > Issue Type: Sub-task > Components: SQL >Reporter: Reynold Xin > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16285) Implement sentences SQL function
[ https://issues.apache.org/jira/browse/SPARK-16285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16285: Assignee: (was: Apache Spark) > Implement sentences SQL function > > > Key: SPARK-16285 > URL: https://issues.apache.org/jira/browse/SPARK-16285 > Project: Spark > Issue Type: Sub-task > Components: SQL >Reporter: Reynold Xin > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15954) TestHive has issues being used in PySpark
[ https://issues.apache.org/jira/browse/SPARK-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15358008#comment-15358008 ] Apache Spark commented on SPARK-15954: -- User 'rxin' has created a pull request for this issue: https://github.com/apache/spark/pull/14005 > TestHive has issues being used in PySpark > - > > Key: SPARK-15954 > URL: https://issues.apache.org/jira/browse/SPARK-15954 > Project: Spark > Issue Type: Bug > Components: PySpark, SQL >Reporter: holdenk > > SPARK-15745 made TestHive unreliable from PySpark test cases, to support it > we should allow both resource or system property based lookup for loading the > hive file. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13015) Replace example code in mllib-data-types.md using include_example
[ https://issues.apache.org/jira/browse/SPARK-13015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15358028#comment-15358028 ] Apache Spark commented on SPARK-13015: -- User 'yinxusen' has created a pull request for this issue: https://github.com/apache/spark/pull/14006 > Replace example code in mllib-data-types.md using include_example > - > > Key: SPARK-13015 > URL: https://issues.apache.org/jira/browse/SPARK-13015 > Project: Spark > Issue Type: Sub-task > Components: Documentation >Reporter: Xusen Yin >Assignee: Xin Ren >Priority: Minor > Labels: starter > > The example code in the user guide is embedded in the markdown and hence it > is not easy to test. It would be nice to automatically test them. This JIRA > is to discuss options to automate example code testing and see what we can do > in Spark 1.6. > Goal is to move actual example code to spark/examples and test compilation in > Jenkins builds. Then in the markdown, we can reference part of the code to > show in the user guide. This requires adding a Jekyll tag that is similar to > https://github.com/jekyll/jekyll/blob/master/lib/jekyll/tags/include.rb, > e.g., called include_example. > {code}{% include_example > scala/org/apache/spark/examples/mllib/LocalVectorExample.scala %}{code} > Jekyll will find > `examples/src/main/scala/org/apache/spark/examples/mllib/LocalVectorExample.scala` > and pick code blocks marked "example" and replace code block in > {code}{% highlight %}{code} > in the markdown. > See more sub-tasks in parent ticket: > https://issues.apache.org/jira/browse/SPARK-11337 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16329) select * from temp_table_no_cols fails
[ https://issues.apache.org/jira/browse/SPARK-16329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16329: Assignee: (was: Apache Spark) > select * from temp_table_no_cols fails > -- > > Key: SPARK-16329 > URL: https://issues.apache.org/jira/browse/SPARK-16329 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.6.0, 1.6.1, 1.6.2 >Reporter: Adrian Ionescu > > The following works with spark 1.5.1, but not anymore with spark 1.6.0: > {code} > import org.apache.spark.sql.{ DataFrame, Row } > import org.apache.spark.sql.types.StructType > val rddNoCols = sqlContext.sparkContext.parallelize(1 to 10).map(_ => > Row.empty) > val dfNoCols = sqlContext.createDataFrame(rddNoCols, StructType(Seq.empty)) > dfNoCols.registerTempTable("temp_table_no_cols") > sqlContext.sql("select * from temp_table_no_cols").show > {code} > spark 1.5.1 result: > {noformat} > ++ > || > ++ > || > || > || > || > || > || > || > || > || > || > ++ > {noformat} > spark 1.6.0 result: > {noformat} > java.lang.IllegalArgumentException: requirement failed > at scala.Predef$.require(Predef.scala:221) > at > org.apache.spark.sql.catalyst.analysis.UnresolvedStar.expand(unresolved.scala:199) > at > org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$10$$anonfun$applyOrElse$14.apply(Analyzer.scala:354) > at > org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$10$$anonfun$applyOrElse$14.apply(Analyzer.scala:353) > at > scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) > at > scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) > at > scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251) > at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105) > at > org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$10.applyOrElse(Analyzer.scala:353) > at > org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$10.applyOrElse(Analyzer.scala:347) > at > org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolveOperators$1.apply(LogicalPlan.scala:57) > at > org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolveOperators$1.apply(LogicalPlan.scala:57) > at > org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:53) > at > org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperators(LogicalPlan.scala:56) > at > org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$.apply(Analyzer.scala:347) > at > org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$.apply(Analyzer.scala:328) > at > org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:83) > at > org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:80) > at > scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:111) > at scala.collection.immutable.List.foldLeft(List.scala:84) > at > org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:80) > at > org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:72) > at scala.collection.immutable.List.foreach(List.scala:318) > at > org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:72) > at > org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:36) > at > org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:36) > at > org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:34) > at org.apache.spark.sql.DataFrame.(DataFrame.scala:133) > at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:52) > at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:817) > at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:28) > at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:33) > at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:35) > at $iwC$$iwC$$iwC$$iwC$$iwC.(:37) > at $iwC$$iwC$$iwC$$iwC.(:39) > at $iwC$$iwC$$iwC.(:41) > at $iwC$$iwC.(:43) > at $iwC.(:45) > at (:47) > at .(:51) > at .() > at .(:7) > at .() > at $print() >
[jira] [Commented] (SPARK-16329) select * from temp_table_no_cols fails
[ https://issues.apache.org/jira/browse/SPARK-16329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15358306#comment-15358306 ] Apache Spark commented on SPARK-16329: -- User 'gatorsmile' has created a pull request for this issue: https://github.com/apache/spark/pull/14007 > select * from temp_table_no_cols fails > -- > > Key: SPARK-16329 > URL: https://issues.apache.org/jira/browse/SPARK-16329 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.6.0, 1.6.1, 1.6.2 >Reporter: Adrian Ionescu > > The following works with spark 1.5.1, but not anymore with spark 1.6.0: > {code} > import org.apache.spark.sql.{ DataFrame, Row } > import org.apache.spark.sql.types.StructType > val rddNoCols = sqlContext.sparkContext.parallelize(1 to 10).map(_ => > Row.empty) > val dfNoCols = sqlContext.createDataFrame(rddNoCols, StructType(Seq.empty)) > dfNoCols.registerTempTable("temp_table_no_cols") > sqlContext.sql("select * from temp_table_no_cols").show > {code} > spark 1.5.1 result: > {noformat} > ++ > || > ++ > || > || > || > || > || > || > || > || > || > || > ++ > {noformat} > spark 1.6.0 result: > {noformat} > java.lang.IllegalArgumentException: requirement failed > at scala.Predef$.require(Predef.scala:221) > at > org.apache.spark.sql.catalyst.analysis.UnresolvedStar.expand(unresolved.scala:199) > at > org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$10$$anonfun$applyOrElse$14.apply(Analyzer.scala:354) > at > org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$10$$anonfun$applyOrElse$14.apply(Analyzer.scala:353) > at > scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) > at > scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) > at > scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251) > at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105) > at > org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$10.applyOrElse(Analyzer.scala:353) > at > org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$10.applyOrElse(Analyzer.scala:347) > at > org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolveOperators$1.apply(LogicalPlan.scala:57) > at > org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolveOperators$1.apply(LogicalPlan.scala:57) > at > org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:53) > at > org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperators(LogicalPlan.scala:56) > at > org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$.apply(Analyzer.scala:347) > at > org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$.apply(Analyzer.scala:328) > at > org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:83) > at > org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:80) > at > scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:111) > at scala.collection.immutable.List.foldLeft(List.scala:84) > at > org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:80) > at > org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:72) > at scala.collection.immutable.List.foreach(List.scala:318) > at > org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:72) > at > org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:36) > at > org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:36) > at > org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:34) > at org.apache.spark.sql.DataFrame.(DataFrame.scala:133) > at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:52) > at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:817) > at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:28) > at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:33) > at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:35) > at $iwC$$iwC$$iwC$$iwC$$iwC.(:37) > at $iwC$$iwC$$iwC$$iwC.(:39) > at $iwC$$iwC$$iwC.(:41) > at $iwC$$iwC.(:43) > at $iwC.(:45) >
[jira] [Assigned] (SPARK-16329) select * from temp_table_no_cols fails
[ https://issues.apache.org/jira/browse/SPARK-16329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16329: Assignee: Apache Spark > select * from temp_table_no_cols fails > -- > > Key: SPARK-16329 > URL: https://issues.apache.org/jira/browse/SPARK-16329 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.6.0, 1.6.1, 1.6.2 >Reporter: Adrian Ionescu >Assignee: Apache Spark > > The following works with spark 1.5.1, but not anymore with spark 1.6.0: > {code} > import org.apache.spark.sql.{ DataFrame, Row } > import org.apache.spark.sql.types.StructType > val rddNoCols = sqlContext.sparkContext.parallelize(1 to 10).map(_ => > Row.empty) > val dfNoCols = sqlContext.createDataFrame(rddNoCols, StructType(Seq.empty)) > dfNoCols.registerTempTable("temp_table_no_cols") > sqlContext.sql("select * from temp_table_no_cols").show > {code} > spark 1.5.1 result: > {noformat} > ++ > || > ++ > || > || > || > || > || > || > || > || > || > || > ++ > {noformat} > spark 1.6.0 result: > {noformat} > java.lang.IllegalArgumentException: requirement failed > at scala.Predef$.require(Predef.scala:221) > at > org.apache.spark.sql.catalyst.analysis.UnresolvedStar.expand(unresolved.scala:199) > at > org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$10$$anonfun$applyOrElse$14.apply(Analyzer.scala:354) > at > org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$10$$anonfun$applyOrElse$14.apply(Analyzer.scala:353) > at > scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) > at > scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) > at > scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251) > at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105) > at > org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$10.applyOrElse(Analyzer.scala:353) > at > org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$10.applyOrElse(Analyzer.scala:347) > at > org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolveOperators$1.apply(LogicalPlan.scala:57) > at > org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolveOperators$1.apply(LogicalPlan.scala:57) > at > org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:53) > at > org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperators(LogicalPlan.scala:56) > at > org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$.apply(Analyzer.scala:347) > at > org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$.apply(Analyzer.scala:328) > at > org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:83) > at > org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:80) > at > scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:111) > at scala.collection.immutable.List.foldLeft(List.scala:84) > at > org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:80) > at > org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:72) > at scala.collection.immutable.List.foreach(List.scala:318) > at > org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:72) > at > org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:36) > at > org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:36) > at > org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:34) > at org.apache.spark.sql.DataFrame.(DataFrame.scala:133) > at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:52) > at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:817) > at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:28) > at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:33) > at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:35) > at $iwC$$iwC$$iwC$$iwC$$iwC.(:37) > at $iwC$$iwC$$iwC$$iwC.(:39) > at $iwC$$iwC$$iwC.(:41) > at $iwC$$iwC.(:43) > at $iwC.(:45) > at (:47) > at .(:51) > at .() > at .(:7) > at .() >
[jira] [Commented] (SPARK-16281) Implement parse_url SQL function
[ https://issues.apache.org/jira/browse/SPARK-16281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15358423#comment-15358423 ] Apache Spark commented on SPARK-16281: -- User 'janplus' has created a pull request for this issue: https://github.com/apache/spark/pull/14008 > Implement parse_url SQL function > > > Key: SPARK-16281 > URL: https://issues.apache.org/jira/browse/SPARK-16281 > Project: Spark > Issue Type: Sub-task > Components: SQL >Reporter: Reynold Xin > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16311) Improve metadata refresh
[ https://issues.apache.org/jira/browse/SPARK-16311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15358456#comment-15358456 ] Apache Spark commented on SPARK-16311: -- User 'rxin' has created a pull request for this issue: https://github.com/apache/spark/pull/14009 > Improve metadata refresh > > > Key: SPARK-16311 > URL: https://issues.apache.org/jira/browse/SPARK-16311 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: Reynold Xin > > When the underlying file changes, it can be very confusing to users when they > see a FileNotFoundException. It would be great to do the following: > (1) Append a message to the FileNotFoundException that a workaround is to do > explicitly metadata refresh. > (2) Make metadata refresh work on temporary tables/views. > (3) Make metadata refresh work on Datasets/DataFrames, by introducing a > Dataset.refresh() method. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16339) ScriptTransform does not print stderr when outstream is lost
[ https://issues.apache.org/jira/browse/SPARK-16339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16339: Assignee: Apache Spark > ScriptTransform does not print stderr when outstream is lost > > > Key: SPARK-16339 > URL: https://issues.apache.org/jira/browse/SPARK-16339 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.6.2 >Reporter: Tejas Patil >Assignee: Apache Spark >Priority: Trivial > > Currently, if due to some failure, the `outstream` gets destroyed or closed > and later `outstream.close()` leads to `IOException` in such case : > https://github.com/apache/spark/blob/4f869f88ee96fa57be79f972f218111b6feac67f/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformation.scala#L325 > Due to this, the `stderrBuffer` does not get logged and there is no way for > users to see why the job failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16339) ScriptTransform does not print stderr when outstream is lost
[ https://issues.apache.org/jira/browse/SPARK-16339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15358473#comment-15358473 ] Apache Spark commented on SPARK-16339: -- User 'tejasapatil' has created a pull request for this issue: https://github.com/apache/spark/pull/13834 > ScriptTransform does not print stderr when outstream is lost > > > Key: SPARK-16339 > URL: https://issues.apache.org/jira/browse/SPARK-16339 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.6.2 >Reporter: Tejas Patil >Priority: Trivial > > Currently, if due to some failure, the `outstream` gets destroyed or closed > and later `outstream.close()` leads to `IOException` in such case : > https://github.com/apache/spark/blob/4f869f88ee96fa57be79f972f218111b6feac67f/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformation.scala#L325 > Due to this, the `stderrBuffer` does not get logged and there is no way for > users to see why the job failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16339) ScriptTransform does not print stderr when outstream is lost
[ https://issues.apache.org/jira/browse/SPARK-16339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16339: Assignee: (was: Apache Spark) > ScriptTransform does not print stderr when outstream is lost > > > Key: SPARK-16339 > URL: https://issues.apache.org/jira/browse/SPARK-16339 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.6.2 >Reporter: Tejas Patil >Priority: Trivial > > Currently, if due to some failure, the `outstream` gets destroyed or closed > and later `outstream.close()` leads to `IOException` in such case : > https://github.com/apache/spark/blob/4f869f88ee96fa57be79f972f218111b6feac67f/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformation.scala#L325 > Due to this, the `stderrBuffer` does not get logged and there is no way for > users to see why the job failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16343) Improve the PushDownPredicate rule to pushdown predicates currectly in non-deterministic condition
[ https://issues.apache.org/jira/browse/SPARK-16343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15358691#comment-15358691 ] Apache Spark commented on SPARK-16343: -- User 'jiangxb1987' has created a pull request for this issue: https://github.com/apache/spark/pull/14012 > Improve the PushDownPredicate rule to pushdown predicates currectly in > non-deterministic condition > -- > > Key: SPARK-16343 > URL: https://issues.apache.org/jira/browse/SPARK-16343 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 1.6.2 >Reporter: Jiang Xingbo >Priority: Critical > > Currently our Optimizer may reorder the predicates to run them more > efficient, but in non-deterministic condition, change the order between > deterministic parts and non-deterministic parts may change the number of > input rows. For example: > {code:sql} > SELECT a FROM t WHERE rand() < 0.1 AND a = 1 > {code} > And > {code:sql} > SELECT a FROM t WHERE a = 1 AND rand() < 0.1 > {code} > may call rand() for different times and therefore the output rows differ. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16343) Improve the PushDownPredicate rule to pushdown predicates currectly in non-deterministic condition
[ https://issues.apache.org/jira/browse/SPARK-16343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16343: Assignee: (was: Apache Spark) > Improve the PushDownPredicate rule to pushdown predicates currectly in > non-deterministic condition > -- > > Key: SPARK-16343 > URL: https://issues.apache.org/jira/browse/SPARK-16343 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 1.6.2 >Reporter: Jiang Xingbo >Priority: Critical > > Currently our Optimizer may reorder the predicates to run them more > efficient, but in non-deterministic condition, change the order between > deterministic parts and non-deterministic parts may change the number of > input rows. For example: > {code:sql} > SELECT a FROM t WHERE rand() < 0.1 AND a = 1 > {code} > And > {code:sql} > SELECT a FROM t WHERE a = 1 AND rand() < 0.1 > {code} > may call rand() for different times and therefore the output rows differ. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16343) Improve the PushDownPredicate rule to pushdown predicates currectly in non-deterministic condition
[ https://issues.apache.org/jira/browse/SPARK-16343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16343: Assignee: Apache Spark > Improve the PushDownPredicate rule to pushdown predicates currectly in > non-deterministic condition > -- > > Key: SPARK-16343 > URL: https://issues.apache.org/jira/browse/SPARK-16343 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 1.6.2 >Reporter: Jiang Xingbo >Assignee: Apache Spark >Priority: Critical > > Currently our Optimizer may reorder the predicates to run them more > efficient, but in non-deterministic condition, change the order between > deterministic parts and non-deterministic parts may change the number of > input rows. For example: > {code:sql} > SELECT a FROM t WHERE rand() < 0.1 AND a = 1 > {code} > And > {code:sql} > SELECT a FROM t WHERE a = 1 AND rand() < 0.1 > {code} > may call rand() for different times and therefore the output rows differ. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16344) Array of struct with a single field name "element" can't be decoded from Parquet files written by Spark 1.6+
[ https://issues.apache.org/jira/browse/SPARK-16344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15358799#comment-15358799 ] Apache Spark commented on SPARK-16344: -- User 'liancheng' has created a pull request for this issue: https://github.com/apache/spark/pull/14013 > Array of struct with a single field name "element" can't be decoded from > Parquet files written by Spark 1.6+ > > > Key: SPARK-16344 > URL: https://issues.apache.org/jira/browse/SPARK-16344 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.6.0, 1.6.1, 1.6.2, 2.0.0 >Reporter: Cheng Lian >Assignee: Cheng Lian > > Array of struct with a single field name "element" can't be decoded from > Parquet files written by Spark 1.6+ > The following Spark shell snippet for Spark 1.6 reproduces this bug: > {code} > case class A(element: Long) > case class B(f: Array[A]) > val path = "/tmp/silly.parquet" > Seq(B(Array(A(42.toDF("f0").write.mode("overwrite").parquet(path) > val df = sqlContext.read.parquet(path) > df.printSchema() > // root > // |-- f0: array (nullable = true) > // ||-- element: struct (containsNull = true) > // |||-- element: long (nullable = true) > df.show() > {code} > Exception thrown: > {noformat} > org.apache.parquet.io.ParquetDecodingException: Can not read value at 0 in > block -1 in file > file:/tmp/silly.parquet/part-r-7-e06db7b0-5181-4a14-9fee-5bb452e883a0.gz.parquet > at > org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:228) > at > org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:201) > at > org.apache.spark.rdd.SqlNewHadoopRDD$$anon$1.hasNext(SqlNewHadoopRDD.scala:194) > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) > at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:308) > at scala.collection.Iterator$class.foreach(Iterator.scala:727) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) > at > scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48) > at > scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103) > at > scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47) > at > scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273) > at scala.collection.AbstractIterator.to(Iterator.scala:1157) > at > scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265) > at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157) > at > scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252) > at scala.collection.AbstractIterator.toArray(Iterator.scala:1157) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:212) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:212) > at > org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858) > at > org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) > at org.apache.spark.scheduler.Task.run(Task.scala:89) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.ClassCastException: Expected instance of group converter > but got > "org.apache.spark.sql.execution.datasources.parquet.CatalystPrimitiveConverter" > at > org.apache.parquet.io.api.Converter.asGroupConverter(Converter.java:37) > at > org.apache.parquet.io.RecordReaderImplementation.(RecordReaderImplementation.java:266) > at > org.apache.parquet.io.MessageColumnIO$1.visit(MessageColumnIO.java:134) > at > org.apache.parquet.io.MessageColumnIO$1.visit(MessageColumnIO.java:99) > at > org.apache.parquet.filter2.compat.FilterCompat$NoOpFilter.accept(FilterCompat.java:154) > at > org.apache.parquet.io.MessageColumnIO.getRecordReader(MessageColumnIO.java:99) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:137) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:208) >
[jira] [Assigned] (SPARK-16344) Array of struct with a single field name "element" can't be decoded from Parquet files written by Spark 1.6+
[ https://issues.apache.org/jira/browse/SPARK-16344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16344: Assignee: Cheng Lian (was: Apache Spark) > Array of struct with a single field name "element" can't be decoded from > Parquet files written by Spark 1.6+ > > > Key: SPARK-16344 > URL: https://issues.apache.org/jira/browse/SPARK-16344 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.6.0, 1.6.1, 1.6.2, 2.0.0 >Reporter: Cheng Lian >Assignee: Cheng Lian > > Array of struct with a single field name "element" can't be decoded from > Parquet files written by Spark 1.6+ > The following Spark shell snippet for Spark 1.6 reproduces this bug: > {code} > case class A(element: Long) > case class B(f: Array[A]) > val path = "/tmp/silly.parquet" > Seq(B(Array(A(42.toDF("f0").write.mode("overwrite").parquet(path) > val df = sqlContext.read.parquet(path) > df.printSchema() > // root > // |-- f0: array (nullable = true) > // ||-- element: struct (containsNull = true) > // |||-- element: long (nullable = true) > df.show() > {code} > Exception thrown: > {noformat} > org.apache.parquet.io.ParquetDecodingException: Can not read value at 0 in > block -1 in file > file:/tmp/silly.parquet/part-r-7-e06db7b0-5181-4a14-9fee-5bb452e883a0.gz.parquet > at > org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:228) > at > org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:201) > at > org.apache.spark.rdd.SqlNewHadoopRDD$$anon$1.hasNext(SqlNewHadoopRDD.scala:194) > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) > at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:308) > at scala.collection.Iterator$class.foreach(Iterator.scala:727) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) > at > scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48) > at > scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103) > at > scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47) > at > scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273) > at scala.collection.AbstractIterator.to(Iterator.scala:1157) > at > scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265) > at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157) > at > scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252) > at scala.collection.AbstractIterator.toArray(Iterator.scala:1157) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:212) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:212) > at > org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858) > at > org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) > at org.apache.spark.scheduler.Task.run(Task.scala:89) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.ClassCastException: Expected instance of group converter > but got > "org.apache.spark.sql.execution.datasources.parquet.CatalystPrimitiveConverter" > at > org.apache.parquet.io.api.Converter.asGroupConverter(Converter.java:37) > at > org.apache.parquet.io.RecordReaderImplementation.(RecordReaderImplementation.java:266) > at > org.apache.parquet.io.MessageColumnIO$1.visit(MessageColumnIO.java:134) > at > org.apache.parquet.io.MessageColumnIO$1.visit(MessageColumnIO.java:99) > at > org.apache.parquet.filter2.compat.FilterCompat$NoOpFilter.accept(FilterCompat.java:154) > at > org.apache.parquet.io.MessageColumnIO.getRecordReader(MessageColumnIO.java:99) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:137) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:208) > ... 26 more > {noformat} > Spark 2.0.0-SNAPSHOT and Spark master also suffer this issue. To reproduce
[jira] [Assigned] (SPARK-16344) Array of struct with a single field name "element" can't be decoded from Parquet files written by Spark 1.6+
[ https://issues.apache.org/jira/browse/SPARK-16344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16344: Assignee: Apache Spark (was: Cheng Lian) > Array of struct with a single field name "element" can't be decoded from > Parquet files written by Spark 1.6+ > > > Key: SPARK-16344 > URL: https://issues.apache.org/jira/browse/SPARK-16344 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.6.0, 1.6.1, 1.6.2, 2.0.0 >Reporter: Cheng Lian >Assignee: Apache Spark > > Array of struct with a single field name "element" can't be decoded from > Parquet files written by Spark 1.6+ > The following Spark shell snippet for Spark 1.6 reproduces this bug: > {code} > case class A(element: Long) > case class B(f: Array[A]) > val path = "/tmp/silly.parquet" > Seq(B(Array(A(42.toDF("f0").write.mode("overwrite").parquet(path) > val df = sqlContext.read.parquet(path) > df.printSchema() > // root > // |-- f0: array (nullable = true) > // ||-- element: struct (containsNull = true) > // |||-- element: long (nullable = true) > df.show() > {code} > Exception thrown: > {noformat} > org.apache.parquet.io.ParquetDecodingException: Can not read value at 0 in > block -1 in file > file:/tmp/silly.parquet/part-r-7-e06db7b0-5181-4a14-9fee-5bb452e883a0.gz.parquet > at > org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:228) > at > org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:201) > at > org.apache.spark.rdd.SqlNewHadoopRDD$$anon$1.hasNext(SqlNewHadoopRDD.scala:194) > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) > at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:308) > at scala.collection.Iterator$class.foreach(Iterator.scala:727) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) > at > scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48) > at > scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103) > at > scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47) > at > scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273) > at scala.collection.AbstractIterator.to(Iterator.scala:1157) > at > scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265) > at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157) > at > scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252) > at scala.collection.AbstractIterator.toArray(Iterator.scala:1157) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:212) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:212) > at > org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858) > at > org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) > at org.apache.spark.scheduler.Task.run(Task.scala:89) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.ClassCastException: Expected instance of group converter > but got > "org.apache.spark.sql.execution.datasources.parquet.CatalystPrimitiveConverter" > at > org.apache.parquet.io.api.Converter.asGroupConverter(Converter.java:37) > at > org.apache.parquet.io.RecordReaderImplementation.(RecordReaderImplementation.java:266) > at > org.apache.parquet.io.MessageColumnIO$1.visit(MessageColumnIO.java:134) > at > org.apache.parquet.io.MessageColumnIO$1.visit(MessageColumnIO.java:99) > at > org.apache.parquet.filter2.compat.FilterCompat$NoOpFilter.accept(FilterCompat.java:154) > at > org.apache.parquet.io.MessageColumnIO.getRecordReader(MessageColumnIO.java:99) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:137) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:208) > ... 26 more > {noformat} > Spark 2.0.0-SNAPSHOT and Spark master also suffer this issue. To reproduc
[jira] [Commented] (SPARK-16344) Array of struct with a single field name "element" can't be decoded from Parquet files written by Spark 1.6+
[ https://issues.apache.org/jira/browse/SPARK-16344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15358837#comment-15358837 ] Apache Spark commented on SPARK-16344: -- User 'liancheng' has created a pull request for this issue: https://github.com/apache/spark/pull/14014 > Array of struct with a single field name "element" can't be decoded from > Parquet files written by Spark 1.6+ > > > Key: SPARK-16344 > URL: https://issues.apache.org/jira/browse/SPARK-16344 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.6.0, 1.6.1, 1.6.2, 2.0.0 >Reporter: Cheng Lian >Assignee: Cheng Lian > > Array of struct with a single field name "element" can't be decoded from > Parquet files written by Spark 1.6+ > The following Spark shell snippet for Spark 1.6 reproduces this bug: > {code} > case class A(element: Long) > case class B(f: Array[A]) > val path = "/tmp/silly.parquet" > Seq(B(Array(A(42.toDF("f0").write.mode("overwrite").parquet(path) > val df = sqlContext.read.parquet(path) > df.printSchema() > // root > // |-- f0: array (nullable = true) > // ||-- element: struct (containsNull = true) > // |||-- element: long (nullable = true) > df.show() > {code} > Exception thrown: > {noformat} > org.apache.parquet.io.ParquetDecodingException: Can not read value at 0 in > block -1 in file > file:/tmp/silly.parquet/part-r-7-e06db7b0-5181-4a14-9fee-5bb452e883a0.gz.parquet > at > org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:228) > at > org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:201) > at > org.apache.spark.rdd.SqlNewHadoopRDD$$anon$1.hasNext(SqlNewHadoopRDD.scala:194) > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) > at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:308) > at scala.collection.Iterator$class.foreach(Iterator.scala:727) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) > at > scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48) > at > scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103) > at > scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47) > at > scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273) > at scala.collection.AbstractIterator.to(Iterator.scala:1157) > at > scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265) > at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157) > at > scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252) > at scala.collection.AbstractIterator.toArray(Iterator.scala:1157) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:212) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:212) > at > org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858) > at > org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) > at org.apache.spark.scheduler.Task.run(Task.scala:89) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.ClassCastException: Expected instance of group converter > but got > "org.apache.spark.sql.execution.datasources.parquet.CatalystPrimitiveConverter" > at > org.apache.parquet.io.api.Converter.asGroupConverter(Converter.java:37) > at > org.apache.parquet.io.RecordReaderImplementation.(RecordReaderImplementation.java:266) > at > org.apache.parquet.io.MessageColumnIO$1.visit(MessageColumnIO.java:134) > at > org.apache.parquet.io.MessageColumnIO$1.visit(MessageColumnIO.java:99) > at > org.apache.parquet.filter2.compat.FilterCompat$NoOpFilter.accept(FilterCompat.java:154) > at > org.apache.parquet.io.MessageColumnIO.getRecordReader(MessageColumnIO.java:99) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:137) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:208) >
[jira] [Assigned] (SPARK-16345) Extract graphx programming guide example snippets from source files instead of hard code them
[ https://issues.apache.org/jira/browse/SPARK-16345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16345: Assignee: Apache Spark > Extract graphx programming guide example snippets from source files instead > of hard code them > - > > Key: SPARK-16345 > URL: https://issues.apache.org/jira/browse/SPARK-16345 > Project: Spark > Issue Type: Bug > Components: Documentation, Examples, GraphX >Affects Versions: 2.0.0, 2.0.1, 2.1.0 >Reporter: Weichen Xu >Assignee: Apache Spark > > Currently, all example snippets in the graphx programming guide are > hard-coded, which can be pretty hard to update and verify. On the contrary, > ML document pages are using the include_example Jekyll plugin to extract > snippets from actual source files under the examples sub-project. In this > way, we can guarantee that Java and Scala code are compilable, and it would > be much easier to verify these example snippets since they are part of > complete Spark applications. > The similar task is SPARK-7924. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16345) Extract graphx programming guide example snippets from source files instead of hard code them
[ https://issues.apache.org/jira/browse/SPARK-16345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16345: Assignee: (was: Apache Spark) > Extract graphx programming guide example snippets from source files instead > of hard code them > - > > Key: SPARK-16345 > URL: https://issues.apache.org/jira/browse/SPARK-16345 > Project: Spark > Issue Type: Bug > Components: Documentation, Examples, GraphX >Affects Versions: 2.0.0, 2.0.1, 2.1.0 >Reporter: Weichen Xu > > Currently, all example snippets in the graphx programming guide are > hard-coded, which can be pretty hard to update and verify. On the contrary, > ML document pages are using the include_example Jekyll plugin to extract > snippets from actual source files under the examples sub-project. In this > way, we can guarantee that Java and Scala code are compilable, and it would > be much easier to verify these example snippets since they are part of > complete Spark applications. > The similar task is SPARK-7924. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16345) Extract graphx programming guide example snippets from source files instead of hard code them
[ https://issues.apache.org/jira/browse/SPARK-16345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15358844#comment-15358844 ] Apache Spark commented on SPARK-16345: -- User 'WeichenXu123' has created a pull request for this issue: https://github.com/apache/spark/pull/14015 > Extract graphx programming guide example snippets from source files instead > of hard code them > - > > Key: SPARK-16345 > URL: https://issues.apache.org/jira/browse/SPARK-16345 > Project: Spark > Issue Type: Bug > Components: Documentation, Examples, GraphX >Affects Versions: 2.0.0, 2.0.1, 2.1.0 >Reporter: Weichen Xu > > Currently, all example snippets in the graphx programming guide are > hard-coded, which can be pretty hard to update and verify. On the contrary, > ML document pages are using the include_example Jekyll plugin to extract > snippets from actual source files under the examples sub-project. In this > way, we can guarantee that Java and Scala code are compilable, and it would > be much easier to verify these example snippets since they are part of > complete Spark applications. > The similar task is SPARK-7924. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15761) pyspark shell should load if PYSPARK_DRIVER_PYTHON is ipython an Python3
[ https://issues.apache.org/jira/browse/SPARK-15761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15359334#comment-15359334 ] Apache Spark commented on SPARK-15761: -- User 'MechCoder' has created a pull request for this issue: https://github.com/apache/spark/pull/14016 > pyspark shell should load if PYSPARK_DRIVER_PYTHON is ipython an Python3 > > > Key: SPARK-15761 > URL: https://issues.apache.org/jira/browse/SPARK-15761 > Project: Spark > Issue Type: Improvement > Components: PySpark >Reporter: Manoj Kumar >Assignee: Manoj Kumar >Priority: Minor > Fix For: 1.6.3, 2.0.1 > > > My default python is ipython3 and it is odd that it fails with "IPython > requires Python 2.7+; please install python2.7 or set PYSPARK_PYTHON" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16212) code cleanup of kafka-0-8 to match review feedback on 0-10
[ https://issues.apache.org/jira/browse/SPARK-16212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15359591#comment-15359591 ] Apache Spark commented on SPARK-16212: -- User 'koeninger' has created a pull request for this issue: https://github.com/apache/spark/pull/14018 > code cleanup of kafka-0-8 to match review feedback on 0-10 > -- > > Key: SPARK-16212 > URL: https://issues.apache.org/jira/browse/SPARK-16212 > Project: Spark > Issue Type: Sub-task > Components: Streaming >Reporter: Cody Koeninger >Assignee: Cody Koeninger > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-16223) Codegen failure with a Dataframe program using an array
[ https://issues.apache.org/jira/browse/SPARK-16223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16223: Assignee: Apache Spark > Codegen failure with a Dataframe program using an array > --- > > Key: SPARK-16223 > URL: https://issues.apache.org/jira/browse/SPARK-16223 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: Kazuaki Ishizaki >Assignee: Apache Spark > > When we compile a Dataframe program with an operation to large array, > compilation failure occurs. This is because a local variable > {{inputadapter_value}} cannot be referenced in {{apply()}} method that is > generated by {{CodegenContext.splitExpressions()}}. The local variable is > defined in {{processNext()}} method. > What is better approach to resolve this? Is it better to pass > {{inputadapter_value}} to {{apply()}} method? > Example program > {code} > val n = 500 > val statement = (0 to n - 1).map(i => s"value + 1.0d") > .mkString("Array(", ",", ")") > sparkContext.parallelize(Seq(0.0d, 1.0d), 1).toDF > .selectExpr(statement).showString(1, true) > {code} > Generated code and stack trace > {code:java} > 23:10:45.801 ERROR > org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator: failed to > compile: org.codehaus.commons.compiler.CompileException: File > 'generated.java', Line 30, Column 36: Expression "inputadapter_value" is not > an rvalue > /* 001 */ public Object generate(Object[] references) { > /* 002 */ return new GeneratedIterator(references); > /* 003 */ } > /* 004 */ > /* 005 */ final class GeneratedIterator extends > org.apache.spark.sql.execution.BufferedRowIterator { > /* 006 */ private Object[] references; > /* 007 */ private scala.collection.Iterator inputadapter_input; > /* 008 */ private Object[] project_values; > /* 009 */ private UnsafeRow project_result; > /* 010 */ private > org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder project_holder; > /* 011 */ private > org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter > project_rowWriter; > /* 012 */ private > org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter > project_arrayWriter; > /* 013 */ > /* 014 */ public GeneratedIterator(Object[] references) { > /* 015 */ this.references = references; > /* 016 */ } > /* 017 */ > /* 018 */ public void init(int index, scala.collection.Iterator inputs[]) { > /* 019 */ partitionIndex = index; > /* 020 */ inputadapter_input = inputs[0]; > /* 021 */ this.project_values = null; > /* 022 */ project_result = new UnsafeRow(1); > /* 023 */ this.project_holder = new > org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(project_result, > 32); > /* 024 */ this.project_rowWriter = new > org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(project_holder, > 1); > /* 025 */ this.project_arrayWriter = new > org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter(); > /* 026 */ } > /* 027 */ > /* 028 */ private void project_apply_0(InternalRow inputadapter_row) { > /* 029 */ double project_value1 = -1.0; > /* 030 */ project_value1 = inputadapter_value + 1.0D; > /* 031 */ if (false) { > /* 032 */ project_values[0] = null; > /* 033 */ } else { > /* 034 */ project_values[0] = project_value1; > /* 035 */ } > /* 036 */ > /* 037 */ double project_value4 = -1.0; > /* 038 */ project_value4 = inputadapter_value + 1.0D; > /* 039 */ if (false) { > /* 040 */ project_values[1] = null; > /* 041 */ } else { > /* 042 */ project_values[1] = project_value4; > /* 043 */ } > ... > /* 4032 */ } > /* 4033 */ > /* 4034 */ protected void processNext() throws java.io.IOException { > /* 4035 */ while (inputadapter_input.hasNext()) { > /* 4036 */ InternalRow inputadapter_row = (InternalRow) > inputadapter_input.next(); > /* 4037 */ System.out.println("row: " + inputadapter_row.getClass() + > ", " + inputadapter_row); > /* 4038 */ double inputadapter_value = inputadapter_row.getDouble(0); > /* 4039 */ > /* 4040 */ final boolean project_isNull = false; > /* 4041 */ this.project_values = new > Object[500];project_apply_0(inputadapter_row); > /* 4042 */ project_apply_1(inputadapter_row); > /* 4043 */ /* final ArrayData project_value = > org.apache.spark.sql.catalyst.util.GenericArrayData.allocate(project_values); > */ > /* 4044 */ final ArrayData project_value = new > org.apache.spark.sql.catalyst.util.GenericArrayData(project_values); > /* 4045 */ this.project_values = null; > /* 4046 */ project_holder.reset(); > /* 4047 */ > /* 4048 */ project_rowWriter.zeroOutNullBytes(); > /* 4049 */ > /* 4050 */ if (project
[jira] [Assigned] (SPARK-16223) Codegen failure with a Dataframe program using an array
[ https://issues.apache.org/jira/browse/SPARK-16223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16223: Assignee: (was: Apache Spark) > Codegen failure with a Dataframe program using an array > --- > > Key: SPARK-16223 > URL: https://issues.apache.org/jira/browse/SPARK-16223 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: Kazuaki Ishizaki > > When we compile a Dataframe program with an operation to large array, > compilation failure occurs. This is because a local variable > {{inputadapter_value}} cannot be referenced in {{apply()}} method that is > generated by {{CodegenContext.splitExpressions()}}. The local variable is > defined in {{processNext()}} method. > What is better approach to resolve this? Is it better to pass > {{inputadapter_value}} to {{apply()}} method? > Example program > {code} > val n = 500 > val statement = (0 to n - 1).map(i => s"value + 1.0d") > .mkString("Array(", ",", ")") > sparkContext.parallelize(Seq(0.0d, 1.0d), 1).toDF > .selectExpr(statement).showString(1, true) > {code} > Generated code and stack trace > {code:java} > 23:10:45.801 ERROR > org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator: failed to > compile: org.codehaus.commons.compiler.CompileException: File > 'generated.java', Line 30, Column 36: Expression "inputadapter_value" is not > an rvalue > /* 001 */ public Object generate(Object[] references) { > /* 002 */ return new GeneratedIterator(references); > /* 003 */ } > /* 004 */ > /* 005 */ final class GeneratedIterator extends > org.apache.spark.sql.execution.BufferedRowIterator { > /* 006 */ private Object[] references; > /* 007 */ private scala.collection.Iterator inputadapter_input; > /* 008 */ private Object[] project_values; > /* 009 */ private UnsafeRow project_result; > /* 010 */ private > org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder project_holder; > /* 011 */ private > org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter > project_rowWriter; > /* 012 */ private > org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter > project_arrayWriter; > /* 013 */ > /* 014 */ public GeneratedIterator(Object[] references) { > /* 015 */ this.references = references; > /* 016 */ } > /* 017 */ > /* 018 */ public void init(int index, scala.collection.Iterator inputs[]) { > /* 019 */ partitionIndex = index; > /* 020 */ inputadapter_input = inputs[0]; > /* 021 */ this.project_values = null; > /* 022 */ project_result = new UnsafeRow(1); > /* 023 */ this.project_holder = new > org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(project_result, > 32); > /* 024 */ this.project_rowWriter = new > org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(project_holder, > 1); > /* 025 */ this.project_arrayWriter = new > org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter(); > /* 026 */ } > /* 027 */ > /* 028 */ private void project_apply_0(InternalRow inputadapter_row) { > /* 029 */ double project_value1 = -1.0; > /* 030 */ project_value1 = inputadapter_value + 1.0D; > /* 031 */ if (false) { > /* 032 */ project_values[0] = null; > /* 033 */ } else { > /* 034 */ project_values[0] = project_value1; > /* 035 */ } > /* 036 */ > /* 037 */ double project_value4 = -1.0; > /* 038 */ project_value4 = inputadapter_value + 1.0D; > /* 039 */ if (false) { > /* 040 */ project_values[1] = null; > /* 041 */ } else { > /* 042 */ project_values[1] = project_value4; > /* 043 */ } > ... > /* 4032 */ } > /* 4033 */ > /* 4034 */ protected void processNext() throws java.io.IOException { > /* 4035 */ while (inputadapter_input.hasNext()) { > /* 4036 */ InternalRow inputadapter_row = (InternalRow) > inputadapter_input.next(); > /* 4037 */ System.out.println("row: " + inputadapter_row.getClass() + > ", " + inputadapter_row); > /* 4038 */ double inputadapter_value = inputadapter_row.getDouble(0); > /* 4039 */ > /* 4040 */ final boolean project_isNull = false; > /* 4041 */ this.project_values = new > Object[500];project_apply_0(inputadapter_row); > /* 4042 */ project_apply_1(inputadapter_row); > /* 4043 */ /* final ArrayData project_value = > org.apache.spark.sql.catalyst.util.GenericArrayData.allocate(project_values); > */ > /* 4044 */ final ArrayData project_value = new > org.apache.spark.sql.catalyst.util.GenericArrayData(project_values); > /* 4045 */ this.project_values = null; > /* 4046 */ project_holder.reset(); > /* 4047 */ > /* 4048 */ project_rowWriter.zeroOutNullBytes(); > /* 4049 */ > /* 4050 */ if (project_isNull) { > /* 4051 */
[jira] [Commented] (SPARK-16223) Codegen failure with a Dataframe program using an array
[ https://issues.apache.org/jira/browse/SPARK-16223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15359716#comment-15359716 ] Apache Spark commented on SPARK-16223: -- User 'dongjoon-hyun' has created a pull request for this issue: https://github.com/apache/spark/pull/14019 > Codegen failure with a Dataframe program using an array > --- > > Key: SPARK-16223 > URL: https://issues.apache.org/jira/browse/SPARK-16223 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: Kazuaki Ishizaki > > When we compile a Dataframe program with an operation to large array, > compilation failure occurs. This is because a local variable > {{inputadapter_value}} cannot be referenced in {{apply()}} method that is > generated by {{CodegenContext.splitExpressions()}}. The local variable is > defined in {{processNext()}} method. > What is better approach to resolve this? Is it better to pass > {{inputadapter_value}} to {{apply()}} method? > Example program > {code} > val n = 500 > val statement = (0 to n - 1).map(i => s"value + 1.0d") > .mkString("Array(", ",", ")") > sparkContext.parallelize(Seq(0.0d, 1.0d), 1).toDF > .selectExpr(statement).showString(1, true) > {code} > Generated code and stack trace > {code:java} > 23:10:45.801 ERROR > org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator: failed to > compile: org.codehaus.commons.compiler.CompileException: File > 'generated.java', Line 30, Column 36: Expression "inputadapter_value" is not > an rvalue > /* 001 */ public Object generate(Object[] references) { > /* 002 */ return new GeneratedIterator(references); > /* 003 */ } > /* 004 */ > /* 005 */ final class GeneratedIterator extends > org.apache.spark.sql.execution.BufferedRowIterator { > /* 006 */ private Object[] references; > /* 007 */ private scala.collection.Iterator inputadapter_input; > /* 008 */ private Object[] project_values; > /* 009 */ private UnsafeRow project_result; > /* 010 */ private > org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder project_holder; > /* 011 */ private > org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter > project_rowWriter; > /* 012 */ private > org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter > project_arrayWriter; > /* 013 */ > /* 014 */ public GeneratedIterator(Object[] references) { > /* 015 */ this.references = references; > /* 016 */ } > /* 017 */ > /* 018 */ public void init(int index, scala.collection.Iterator inputs[]) { > /* 019 */ partitionIndex = index; > /* 020 */ inputadapter_input = inputs[0]; > /* 021 */ this.project_values = null; > /* 022 */ project_result = new UnsafeRow(1); > /* 023 */ this.project_holder = new > org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(project_result, > 32); > /* 024 */ this.project_rowWriter = new > org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(project_holder, > 1); > /* 025 */ this.project_arrayWriter = new > org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter(); > /* 026 */ } > /* 027 */ > /* 028 */ private void project_apply_0(InternalRow inputadapter_row) { > /* 029 */ double project_value1 = -1.0; > /* 030 */ project_value1 = inputadapter_value + 1.0D; > /* 031 */ if (false) { > /* 032 */ project_values[0] = null; > /* 033 */ } else { > /* 034 */ project_values[0] = project_value1; > /* 035 */ } > /* 036 */ > /* 037 */ double project_value4 = -1.0; > /* 038 */ project_value4 = inputadapter_value + 1.0D; > /* 039 */ if (false) { > /* 040 */ project_values[1] = null; > /* 041 */ } else { > /* 042 */ project_values[1] = project_value4; > /* 043 */ } > ... > /* 4032 */ } > /* 4033 */ > /* 4034 */ protected void processNext() throws java.io.IOException { > /* 4035 */ while (inputadapter_input.hasNext()) { > /* 4036 */ InternalRow inputadapter_row = (InternalRow) > inputadapter_input.next(); > /* 4037 */ System.out.println("row: " + inputadapter_row.getClass() + > ", " + inputadapter_row); > /* 4038 */ double inputadapter_value = inputadapter_row.getDouble(0); > /* 4039 */ > /* 4040 */ final boolean project_isNull = false; > /* 4041 */ this.project_values = new > Object[500];project_apply_0(inputadapter_row); > /* 4042 */ project_apply_1(inputadapter_row); > /* 4043 */ /* final ArrayData project_value = > org.apache.spark.sql.catalyst.util.GenericArrayData.allocate(project_values); > */ > /* 4044 */ final ArrayData project_value = new > org.apache.spark.sql.catalyst.util.GenericArrayData(project_values); > /* 4045 */ this.project_values = null; > /* 4046 */ project_holder.reset(); > /* 4047 */ > /*
[jira] [Assigned] (SPARK-16233) test_sparkSQL.R is failing
[ https://issues.apache.org/jira/browse/SPARK-16233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16233: Assignee: Apache Spark > test_sparkSQL.R is failing > -- > > Key: SPARK-16233 > URL: https://issues.apache.org/jira/browse/SPARK-16233 > Project: Spark > Issue Type: Bug > Components: SparkR, Tests >Affects Versions: 2.0.0 >Reporter: Xin Ren >Assignee: Apache Spark >Priority: Minor > > By running > {code} > ./R/run-tests.sh > {code} > Getting error: > {code} > xin:spark xr$ ./R/run-tests.sh > Warning: Ignoring non-spark config property: SPARK_SCALA_VERSION=2.11 > Loading required package: methods > Attaching package: ‘SparkR’ > The following object is masked from ‘package:testthat’: > describe > The following objects are masked from ‘package:stats’: > cov, filter, lag, na.omit, predict, sd, var, window > The following objects are masked from ‘package:base’: > as.data.frame, colnames, colnames<-, drop, endsWith, intersect, > rank, rbind, sample, startsWith, subset, summary, transform, union > binary functions: ... > functions on binary files: > broadcast variables: .. > functions in client.R: . > test functions in sparkR.R: .Re-using existing Spark Context. Call > sparkR.session.stop() or restart R to create a new Spark Context > Re-using existing Spark Context. Call sparkR.session.stop() or restart R > to create a new Spark Context > ... > include an external JAR in SparkContext: Warning: Ignoring non-spark config > property: SPARK_SCALA_VERSION=2.11 > .. > include R packages: > MLlib functions: .SLF4J: Failed to load class > "org.slf4j.impl.StaticLoggerBinder". > SLF4J: Defaulting to no-operation (NOP) logger implementation > SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further > details. > .27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig: > Compression: SNAPPY > 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: > Parquet block size to 134217728 > 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: > Parquet page size to 1048576 > 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: > Parquet dictionary page size to 1048576 > 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: > Dictionary is on > 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: > Validation is off > 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: > Writer version is: PARQUET_1_0 > 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: > Maximum row group padding size is 0 bytes > 27-Jun-2016 1:51:25 PM INFO: > org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing mem > columnStore to file. allocated memory: 65,622 > 27-Jun-2016 1:51:25 PM INFO: > org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 70B for [label] > BINARY: 1 values, 21B raw, 23B comp, 1 pages, encodings: [PLAIN, RLE, > BIT_PACKED] > 27-Jun-2016 1:51:25 PM INFO: > org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 87B for [terms, > list, element, list, element] BINARY: 2 values, 42B raw, 43B comp, 1 pages, > encodings: [PLAIN, RLE] > 27-Jun-2016 1:51:25 PM INFO: > org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 30B for > [hasIntercept] BOOLEAN: 1 values, 1B raw, 3B comp, 1 pages, encodings: > [PLAIN, BIT_PACKED] > 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig: > Compression: SNAPPY > 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: > Parquet block size to 134217728 > 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: > Parquet page size to 1048576 > 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: > Parquet dictionary page size to 1048576 > 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: > Dictionary is on > 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: > Validation is off > 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: > Writer version is: PARQUET_1_0 > 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: > Maximum row group padding size is 0 bytes > 27-Jun-2016 1:51:26 PM INFO: > org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing mem > columnStore to file. allocated memory: 49 > 27-Jun-2016 1:51:26 PM INFO: > org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 90B for [labels, > list, element] BINARY: 3 values, 50B raw, 50B comp, 1 pages, encodings: > [PLAIN, RLE] > 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig: >