[jira] [Assigned] (SPARK-18886) Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout

2016-12-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18886: Assignee: (was: Apache Spark) > Delay scheduling should not delay some executors

[jira] [Commented] (SPARK-18859) Catalyst codegen does not mark column as nullable when it should. Causes NPE

2016-12-20 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15764789#comment-15764789 ] Kazuaki Ishizaki commented on SPARK-18859: -- I think that this is an issue in join operation.

[jira] [Commented] (SPARK-18940) Percentile and approximate percentile support for frequency distribution table

2016-12-20 Thread gagan taneja (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15764702#comment-15764702 ] gagan taneja commented on SPARK-18940: -- i am working on the fix. Should be able to make the changes

[jira] [Updated] (SPARK-18940) Percentile and approximate percentile support for frequency distribution table

2016-12-20 Thread gagan taneja (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] gagan taneja updated SPARK-18940: - Shepherd: Herman van Hovell > Percentile and approximate percentile support for frequency

[jira] [Commented] (SPARK-18455) General support for correlated subquery processing

2016-12-20 Thread Nattavut Sutyanyong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15764604#comment-15764604 ] Nattavut Sutyanyong commented on SPARK-18455: - I have placed a link to the detailed design

[jira] [Assigned] (SPARK-18948) Add Mean Percentile Rank metric for ranking algorithms

2016-12-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18948: Assignee: Apache Spark > Add Mean Percentile Rank metric for ranking algorithms >

[jira] [Assigned] (SPARK-18948) Add Mean Percentile Rank metric for ranking algorithms

2016-12-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18948: Assignee: (was: Apache Spark) > Add Mean Percentile Rank metric for ranking

[jira] [Commented] (SPARK-18948) Add Mean Percentile Rank metric for ranking algorithms

2016-12-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15764601#comment-15764601 ] Apache Spark commented on SPARK-18948: -- User 'daniloascione' has created a pull request for this

[jira] [Commented] (SPARK-18874) First phase: Deferring the correlated predicate pull up to Optimizer phase

2016-12-20 Thread Nattavut Sutyanyong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15764597#comment-15764597 ] Nattavut Sutyanyong commented on SPARK-18874: - Here is an initial version of the detailed

[jira] [Created] (SPARK-18948) Add Mean Percentile Rank metric for ranking algorithms

2016-12-20 Thread Danilo Ascione (JIRA)
Danilo Ascione created SPARK-18948: -- Summary: Add Mean Percentile Rank metric for ranking algorithms Key: SPARK-18948 URL: https://issues.apache.org/jira/browse/SPARK-18948 Project: Spark

[jira] [Resolved] (SPARK-18910) Can't use UDF that jar file in hdfs

2016-12-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-18910. --- Resolution: Duplicate > Can't use UDF that jar file in hdfs > --- >

[jira] [Commented] (SPARK-18910) Can't use UDF that jar file in hdfs

2016-12-20 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15764350#comment-15764350 ] Yuming Wang commented on SPARK-18910: - This should be a duplicate of SPARK-12868. > Can't use UDF

[jira] [Assigned] (SPARK-18947) SQLContext.tableNames should not call Catalog.listTables

2016-12-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18947: Assignee: Apache Spark (was: Wenchen Fan) > SQLContext.tableNames should not call

[jira] [Commented] (SPARK-18947) SQLContext.tableNames should not call Catalog.listTables

2016-12-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15764345#comment-15764345 ] Apache Spark commented on SPARK-18947: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-18947) SQLContext.tableNames should not call Catalog.listTables

2016-12-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18947: Assignee: Wenchen Fan (was: Apache Spark) > SQLContext.tableNames should not call

[jira] [Created] (SPARK-18947) SQLContext.tableNames should not call Catalog.listTables

2016-12-20 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-18947: --- Summary: SQLContext.tableNames should not call Catalog.listTables Key: SPARK-18947 URL: https://issues.apache.org/jira/browse/SPARK-18947 Project: Spark Issue

[jira] [Commented] (SPARK-18359) Let user specify locale in CSV parsing

2016-12-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15764307#comment-15764307 ] Sean Owen commented on SPARK-18359: --- They're similar, but this is about locale vs time zone. > Let

[jira] [Commented] (SPARK-18359) Let user specify locale in CSV parsing

2016-12-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15764298#comment-15764298 ] Hyukjin Kwon commented on SPARK-18359: -- Could this be resolved as a duplicate of SPARK-18937? > Let

[jira] [Updated] (SPARK-17838) Strict type checking for arguments with a better messages across APIs.

2016-12-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-17838: -- Assignee: Hyukjin Kwon > Strict type checking for arguments with a better messages across APIs. >

[jira] [Resolved] (SPARK-17838) Strict type checking for arguments with a better messages across APIs.

2016-12-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-17838. --- Resolution: Fixed > Strict type checking for arguments with a better messages across APIs. >

[jira] [Updated] (SPARK-18738) Some Spark SQL queries has poor performance on HDFS Erasure Coding feature when enabling dynamic allocation.

2016-12-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-18738: -- Don't set Fix version. Naively, in the EC case, there are fewer replicas of the data, right? is it

[jira] [Resolved] (SPARK-18938) Addition of peak memory usage metric for an executor

2016-12-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-18938. --- Resolution: Invalid Fix Version/s: (was: 1.5.2) Looks like maybe this was opened by

[jira] [Updated] (SPARK-18859) Catalyst codegen does not mark column as nullable when it should. Causes NPE

2016-12-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-18859: -- Target Version/s: (was: 2.0.2) > Catalyst codegen does not mark column as nullable when it should.

[jira] [Updated] (SPARK-18669) Update Apache docs regard watermarking in Structured Streaming

2016-12-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-18669: -- Target Version/s: 2.1.1 (was: 2.1.0) > Update Apache docs regard watermarking in Structured Streaming

[jira] [Commented] (SPARK-18946) treeAggregate will be low effficiency when aggregate high dimension vectors in ML algorithm

2016-12-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15764191#comment-15764191 ] Sean Owen commented on SPARK-18946: --- I'm not sure what you're proposing as a fix though -- a big object

[jira] [Updated] (SPARK-18946) treeAggregate will be low effficiency when aggregate high dimension vectors in ML algorithm

2016-12-20 Thread zunwen you (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zunwen you updated SPARK-18946: --- Summary: treeAggregate will be low effficiency when aggregate high dimension vectors in ML algorithm

[jira] [Created] (SPARK-18946) treeAggregate will be low effficiency when aggregate high dimension vector in ML algorithm

2016-12-20 Thread zunwen you (JIRA)
zunwen you created SPARK-18946: -- Summary: treeAggregate will be low effficiency when aggregate high dimension vector in ML algorithm Key: SPARK-18946 URL: https://issues.apache.org/jira/browse/SPARK-18946

[jira] [Commented] (SPARK-18926) run-example SparkPi terminates with error message

2016-12-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15764176#comment-15764176 ] Sean Owen commented on SPARK-18926: --- I can't reproduce this on my laptop. I think this might be

[jira] [Resolved] (SPARK-18944) Understanding BroadcastNestedLoopJoin and number of partitions

2016-12-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-18944. --- Resolution: Invalid Questions belong on u...@spark.apache.org > Understanding

[jira] [Comment Edited] (SPARK-18945) java.lang.ClassCastException when Tuple2 field is an array

2016-12-20 Thread Nira Amit (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15764163#comment-15764163 ] Nira Amit edited comment on SPARK-18945 at 12/20/16 1:03 PM: - There's a bug

[jira] [Closed] (SPARK-18945) java.lang.ClassCastException when Tuple2 field is an array

2016-12-20 Thread Nira Amit (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nira Amit closed SPARK-18945. - Resolution: Not A Problem There's a bug in my code > java.lang.ClassCastException when Tuple2 field is

[jira] [Created] (SPARK-18945) java.lang.ClassCastException when Tuple2 field is an array

2016-12-20 Thread Nira Amit (JIRA)
Nira Amit created SPARK-18945: - Summary: java.lang.ClassCastException when Tuple2 field is an array Key: SPARK-18945 URL: https://issues.apache.org/jira/browse/SPARK-18945 Project: Spark Issue

[jira] [Created] (SPARK-18944) Understanding BroadcastNestedLoopJoin and number of partitions

2016-12-20 Thread David Hodeffi (JIRA)
David Hodeffi created SPARK-18944: - Summary: Understanding BroadcastNestedLoopJoin and number of partitions Key: SPARK-18944 URL: https://issues.apache.org/jira/browse/SPARK-18944 Project: Spark

[jira] [Assigned] (SPARK-18943) Avoid per-record type dispatch in CSV when reading

2016-12-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18943: Assignee: Apache Spark > Avoid per-record type dispatch in CSV when reading >

[jira] [Assigned] (SPARK-18943) Avoid per-record type dispatch in CSV when reading

2016-12-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18943: Assignee: (was: Apache Spark) > Avoid per-record type dispatch in CSV when reading >

[jira] [Commented] (SPARK-18943) Avoid per-record type dispatch in CSV when reading

2016-12-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15764103#comment-15764103 ] Apache Spark commented on SPARK-18943: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Created] (SPARK-18943) Avoid per-record type dispatch in CSV when reading

2016-12-20 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-18943: Summary: Avoid per-record type dispatch in CSV when reading Key: SPARK-18943 URL: https://issues.apache.org/jira/browse/SPARK-18943 Project: Spark Issue

[jira] [Issue Comment Deleted] (SPARK-16845) org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering" grows beyond 64 KB

2016-12-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-16845: -- Comment: was deleted (was: I am currently out of office, and will be back on Monday, 5th of December,

[jira] [Created] (SPARK-18942) Support output operations for kinesis

2016-12-20 Thread Takeshi Yamamuro (JIRA)
Takeshi Yamamuro created SPARK-18942: Summary: Support output operations for kinesis Key: SPARK-18942 URL: https://issues.apache.org/jira/browse/SPARK-18942 Project: Spark Issue Type:

[jira] [Updated] (SPARK-18940) Percentile and approximate percentile support for frequency distribution table

2016-12-20 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell updated SPARK-18940: -- Description: I have a frequency distribution table with following entries {noformat}

[jira] [Commented] (SPARK-18940) Percentile and approximate percentile support for frequency distribution table

2016-12-20 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15763987#comment-15763987 ] Herman van Hovell commented on SPARK-18940: --- I like this idea. We can maintain Hive

[jira] [Comment Edited] (SPARK-18940) Percentile and approximate percentile support for frequency distribution table

2016-12-20 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15763987#comment-15763987 ] Herman van Hovell edited comment on SPARK-18940 at 12/20/16 11:31 AM:

[jira] [Commented] (SPARK-18700) getCached in HiveMetastoreCatalog not thread safe cause driver OOM

2016-12-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15763927#comment-15763927 ] Apache Spark commented on SPARK-18700: -- User 'xuanyuanking' has created a pull request for this

[jira] [Commented] (SPARK-18878) Fix/investigate the more identified test failures in Java/Scala on Windows

2016-12-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15763835#comment-15763835 ] Hyukjin Kwon commented on SPARK-18878: -- Yup, and I think I figured this out -

[jira] [Updated] (SPARK-18941) Spark thrift server, Spark 2.0.2, The "drop table" command doesn't delete the directory associated with the table (not EXTERNAL table) from the file system

2016-12-20 Thread luat (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luat updated SPARK-18941: - Description: Spark thrift server, Spark 2.0.2, The "drop table" command doesn't delete the directory associated

[jira] [Updated] (SPARK-18941) Spark2-HiveThriftServer2, Spark 2.0, The "drop table" command doesn't delete the directory associated with the table (not EXTERNAL table) from the file system

2016-12-20 Thread luat (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luat updated SPARK-18941: - Summary: Spark2-HiveThriftServer2, Spark 2.0, The "drop table" command doesn't delete the directory associated

[jira] [Created] (SPARK-18941) Spark2-HiveThriftServer2, Spark 2.0, The "drop table" command doesn't delete the directory associated with the table from the file system

2016-12-20 Thread luat (JIRA)
luat created SPARK-18941: Summary: Spark2-HiveThriftServer2, Spark 2.0, The "drop table" command doesn't delete the directory associated with the table from the file system Key: SPARK-18941 URL:

[jira] [Resolved] (SPARK-18881) Spark never finishes jobs and stages, JobProgressListener fails

2016-12-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-18881. --- Resolution: Duplicate > Spark never finishes jobs and stages, JobProgressListener fails >

[jira] [Resolved] (SPARK-18933) Different log output between Terminal screen and stderr file

2016-12-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-18933. --- Resolution: Invalid This sounds like a question, which should go to user@. It sounds like you have a

[jira] [Updated] (SPARK-18881) Spark never finishes jobs and stages, JobProgressListener fails

2016-12-20 Thread Mathieu D (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mathieu D updated SPARK-18881: -- Description: We have a Spark application that process continuously a lot of incoming jobs. Several

[jira] [Commented] (SPARK-18883) FileNotFoundException on _temporary directory

2016-12-20 Thread Mathieu D (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15763747#comment-15763747 ] Mathieu D commented on SPARK-18883: --- The problem does not appear with

[jira] [Commented] (SPARK-18926) run-example SparkPi terminates with error message

2016-12-20 Thread Alex DeCastro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15763736#comment-15763736 ] Alex DeCastro commented on SPARK-18926: --- Hi [~dongjoon], thanks for the swift follow-up. Not all

[jira] [Updated] (SPARK-18800) Correct the assert in UnsafeKVExternalSorter which ensures array size

2016-12-20 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-18800: Issue Type: Improvement (was: Bug) > Correct the assert in UnsafeKVExternalSorter which

[jira] [Updated] (SPARK-18800) Correct the assert in UnsafeKVExternalSorter which ensures array size

2016-12-20 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-18800: Description: UnsafeKVExternalSorter uses UnsafeInMemorySorter to sort the records of

[jira] [Updated] (SPARK-18800) Correct the assert in UnsafeKVExternalSorter which ensures array size

2016-12-20 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-18800: Summary: Correct the assert in UnsafeKVExternalSorter which ensures array size (was:

[jira] [Commented] (SPARK-18710) Add offset to GeneralizedLinearRegression models

2016-12-20 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15763622#comment-15763622 ] Yanbo Liang commented on SPARK-18710: - We can add a new class for GLR instance(named as

[jira] [Created] (SPARK-18940) Percentile and approximate percentile support for frequency distribution table

2016-12-20 Thread gagan taneja (JIRA)
gagan taneja created SPARK-18940: Summary: Percentile and approximate percentile support for frequency distribution table Key: SPARK-18940 URL: https://issues.apache.org/jira/browse/SPARK-18940

<    1   2