[jira] [Commented] (SPARK-17248) Add native Scala enum support to Dataset Encoders

2017-03-09 Thread Lee Dongjin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903245#comment-15903245 ] Lee Dongjin commented on SPARK-17248: - [~pdxleif] // Although it may be an expired question, let me

[jira] [Commented] (SPARK-19884) Add the ability to get all registered functions from a SparkSession

2017-03-09 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903257#comment-15903257 ] Herman van Hovell commented on SPARK-19884: --- You can use the catalog for that by calling

[jira] [Resolved] (SPARK-19884) Add the ability to get all registered functions from a SparkSession

2017-03-09 Thread Yael Aharon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yael Aharon resolved SPARK-19884. - Resolution: Not A Problem Thank you so much for your reply > Add the ability to get all

[jira] [Commented] (SPARK-15790) Audit @Since annotations in ML

2017-03-09 Thread Ehsun Behravesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904652#comment-15904652 ] Ehsun Behravesh commented on SPARK-15790: - Does this JIRA still need someone to work on? >

[jira] [Resolved] (SPARK-17557) SQL query on parquet table java.lang.UnsupportedOperationException: org.apache.parquet.column.values.dictionary.PlainValuesDictionary

2017-03-09 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-17557. -- Resolution: Cannot Reproduce ^ I can't reproduce too. Let me resolve this. Please reopen this

[jira] [Commented] (SPARK-6634) Allow replacing columns in Transformers

2017-03-09 Thread Tree Field (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904445#comment-15904445 ] Tree Field commented on SPARK-6634: --- I want this feature too. because I often overwrite UnaryTransformer

[jira] [Commented] (SPARK-14083) Analyze JVM bytecode and turn closures into Catalyst expressions

2017-03-09 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904460#comment-15904460 ] Kazuaki Ishizaki commented on SPARK-14083: -- I rebased this with master:

[jira] [Assigned] (SPARK-19892) Implement findAnalogies method for Word2VecModel

2017-03-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19892: Assignee: Apache Spark > Implement findAnalogies method for Word2VecModel >

[jira] [Commented] (SPARK-19892) Implement findAnalogies method for Word2VecModel

2017-03-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904471#comment-15904471 ] Apache Spark commented on SPARK-19892: -- User 'benradford' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19892) Implement findAnalogies method for Word2VecModel

2017-03-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19892: Assignee: (was: Apache Spark) > Implement findAnalogies method for Word2VecModel >

[jira] [Commented] (SPARK-19320) Allow guaranteed amount of GPU to be used when launching jobs

2017-03-09 Thread Ji Yan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904479#comment-15904479 ] Ji Yan commented on SPARK-19320: i'm proposing to add a configuration parameter to guarantee a hard limit

[jira] [Assigned] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

2017-03-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18112: Assignee: Apache Spark (was: Xiao Li) > Spark2.x does not support read data from Hive

[jira] [Assigned] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

2017-03-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18112: Assignee: Xiao Li (was: Apache Spark) > Spark2.x does not support read data from Hive

[jira] [Commented] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

2017-03-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904429#comment-15904429 ] Apache Spark commented on SPARK-18112: -- User 'gatorsmile' has created a pull request for this issue:

[jira] [Created] (SPARK-19892) Implement findAnalogies method for Word2VecModel

2017-03-09 Thread Benjamin Radford (JIRA)
Benjamin Radford created SPARK-19892: Summary: Implement findAnalogies method for Word2VecModel Key: SPARK-19892 URL: https://issues.apache.org/jira/browse/SPARK-19892 Project: Spark

[jira] [Commented] (SPARK-11569) StringIndexer transform fails when column contains nulls

2017-03-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904448#comment-15904448 ] Apache Spark commented on SPARK-11569: -- User 'crackcell' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19320) Allow guaranteed amount of GPU to be used when launching jobs

2017-03-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19320: Assignee: Apache Spark > Allow guaranteed amount of GPU to be used when launching jobs >

[jira] [Assigned] (SPARK-19320) Allow guaranteed amount of GPU to be used when launching jobs

2017-03-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19320: Assignee: (was: Apache Spark) > Allow guaranteed amount of GPU to be used when

[jira] [Commented] (SPARK-19320) Allow guaranteed amount of GPU to be used when launching jobs

2017-03-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904484#comment-15904484 ] Apache Spark commented on SPARK-19320: -- User 'yanji84' has created a pull request for this issue:

[jira] [Resolved] (SPARK-19757) Executor with task scheduled could be killed due to idleness

2017-03-09 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-19757. Resolution: Fixed Assignee: Jimmy Xiang Fix Version/s: 2.2.0 > Executor

[jira] [Resolved] (SPARK-19793) Use clock.getTimeMillis when mark task as finished in TaskSetManager.

2017-03-09 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-19793. Resolution: Fixed Assignee: jin xing Fix Version/s: 2.2.0 > Use

[jira] [Commented] (SPARK-19882) Pivot with null as the pivot value throws NPE

2017-03-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903643#comment-15903643 ] Apache Spark commented on SPARK-19882: -- User 'aray' has created a pull request for this issue:

[jira] [Commented] (SPARK-19887) __HIVE_DEFAULT_PARTITION__ not interpreted as NULL partition value in partitioned persisted tables

2017-03-09 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903749#comment-15903749 ] Cheng Lian commented on SPARK-19887: cc [~cloud_fan] > __HIVE_DEFAULT_PARTITION__ not interpreted as

[jira] [Created] (SPARK-19885) The config ignoreCorruptFiles doesn't work for CSV

2017-03-09 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-19885: Summary: The config ignoreCorruptFiles doesn't work for CSV Key: SPARK-19885 URL: https://issues.apache.org/jira/browse/SPARK-19885 Project: Spark Issue

[jira] [Resolved] (SPARK-19715) Option to Strip Paths in FileSource

2017-03-09 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19715. -- Resolution: Fixed Assignee: Liwei Lin Fix Version/s: 2.2.0 > Option to Strip

[jira] [Commented] (SPARK-19507) pyspark.sql.types._verify_type() exceptions too broad to debug collections or nested data

2017-03-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903680#comment-15903680 ] Apache Spark commented on SPARK-19507: -- User 'dgingrich' has created a pull request for this issue:

[jira] [Resolved] (SPARK-19861) watermark should not be a negative time.

2017-03-09 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19861. -- Resolution: Fixed Assignee: Genmao Yu Fix Version/s: 2.2.0

[jira] [Assigned] (SPARK-19886) reportDataLoss cause != null check is wrong for Structured Streaming KafkaSource

2017-03-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19886: Assignee: Apache Spark (was: Burak Yavuz) > reportDataLoss cause != null check is wrong

[jira] [Created] (SPARK-19887) __HIVE_DEFAULT_PARTITION__ not interpreted as NULL partition value in partitioned persisted tables

2017-03-09 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-19887: -- Summary: __HIVE_DEFAULT_PARTITION__ not interpreted as NULL partition value in partitioned persisted tables Key: SPARK-19887 URL: https://issues.apache.org/jira/browse/SPARK-19887

[jira] [Assigned] (SPARK-19886) reportDataLoss cause != null check is wrong for Structured Streaming KafkaSource

2017-03-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19886: Assignee: Burak Yavuz (was: Apache Spark) > reportDataLoss cause != null check is wrong

[jira] [Commented] (SPARK-19886) reportDataLoss cause != null check is wrong for Structured Streaming KafkaSource

2017-03-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903733#comment-15903733 ] Apache Spark commented on SPARK-19886: -- User 'brkyvz' has created a pull request for this issue:

[jira] [Created] (SPARK-19886) reportDataLoss cause != null check is wrong for Structured Streaming KafkaSource

2017-03-09 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-19886: --- Summary: reportDataLoss cause != null check is wrong for Structured Streaming KafkaSource Key: SPARK-19886 URL: https://issues.apache.org/jira/browse/SPARK-19886

[jira] [Comment Edited] (SPARK-19067) mapGroupsWithState - arbitrary stateful operations with Structured Streaming (similar to DStream.mapWithState)

2017-03-09 Thread Amit Sela (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903532#comment-15903532 ] Amit Sela edited comment on SPARK-19067 at 3/9/17 6:15 PM: --- [~tdas] I just read

[jira] [Updated] (SPARK-19757) Executor with task scheduled could be killed due to idleness

2017-03-09 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-19757: --- Reporter: jin xing (was: Jimmy Xiang) > Executor with task scheduled could be killed due to

[jira] [Updated] (SPARK-19887) __HIVE_DEFAULT_PARTITION__ is not interpreted as NULL partition value in partitioned persisted tables

2017-03-09 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-19887: --- Summary: __HIVE_DEFAULT_PARTITION__ is not interpreted as NULL partition value in partitioned

[jira] [Commented] (SPARK-19067) mapGroupsWithState - arbitrary stateful operations with Structured Streaming (similar to DStream.mapWithState)

2017-03-09 Thread Amit Sela (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903532#comment-15903532 ] Amit Sela commented on SPARK-19067: --- [~tdas] I just read the PR, and I'm very excited for Spark to

[jira] [Commented] (SPARK-19875) Map->filter on many columns gets stuck in constraint inference optimization code

2017-03-09 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903584#comment-15903584 ] Kazuaki Ishizaki commented on SPARK-19875: -- I got the following stack trace. This stuck seems to

[jira] [Resolved] (SPARK-12334) Support read from multiple input paths for orc file in DataFrameReader.orc

2017-03-09 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-12334. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 10307

[jira] [Assigned] (SPARK-12334) Support read from multiple input paths for orc file in DataFrameReader.orc

2017-03-09 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk reassigned SPARK-12334: --- Assignee: Jeff Zhang > Support read from multiple input paths for orc file in DataFrameReader.orc >

[jira] [Resolved] (SPARK-19611) Spark 2.1.0 breaks some Hive tables backed by case-sensitive data files

2017-03-09 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-19611. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 16944

[jira] [Comment Edited] (SPARK-19364) Stream Blocks in Storage Persists Forever when Kinesis Checkpoints are enabled and an exception is thrown

2017-03-09 Thread Andrew Milkowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903778#comment-15903778 ] Andrew Milkowski edited comment on SPARK-19364 at 3/9/17 8:17 PM: --

[jira] [Commented] (SPARK-19364) Stream Blocks in Storage Persists Forever when Kinesis Checkpoints are enabled and an exception is thrown

2017-03-09 Thread Andrew Milkowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903778#comment-15903778 ] Andrew Milkowski commented on SPARK-19364: -- thanks @Takeshi Yamamuro , will try to see if I can

[jira] [Assigned] (SPARK-19611) Spark 2.1.0 breaks some Hive tables backed by case-sensitive data files

2017-03-09 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-19611: --- Assignee: Adam Budde > Spark 2.1.0 breaks some Hive tables backed by case-sensitive data

[jira] [Created] (SPARK-19890) Make MetastoreRelation statistics estimation more accurately

2017-03-09 Thread Zhan Zhang (JIRA)
Zhan Zhang created SPARK-19890: -- Summary: Make MetastoreRelation statistics estimation more accurately Key: SPARK-19890 URL: https://issues.apache.org/jira/browse/SPARK-19890 Project: Spark

[jira] [Updated] (SPARK-19890) Make MetastoreRelation statistics estimation more accurately

2017-03-09 Thread Zhan Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhan Zhang updated SPARK-19890: --- Description: Currently the MetastoreRelation statistics is retrieved on the analyze phase, and the

[jira] [Commented] (SPARK-19611) Spark 2.1.0 breaks some Hive tables backed by case-sensitive data files

2017-03-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904033#comment-15904033 ] Apache Spark commented on SPARK-19611: -- User 'budde' has created a pull request for this issue:

[jira] [Created] (SPARK-19889) Make TaskContext synchronized

2017-03-09 Thread Herman van Hovell (JIRA)
Herman van Hovell created SPARK-19889: - Summary: Make TaskContext synchronized Key: SPARK-19889 URL: https://issues.apache.org/jira/browse/SPARK-19889 Project: Spark Issue Type:

[jira] [Created] (SPARK-19888) Seeing offsets not resetting even when reset policy is configured explicitly

2017-03-09 Thread Justin Miller (JIRA)
Justin Miller created SPARK-19888: - Summary: Seeing offsets not resetting even when reset policy is configured explicitly Key: SPARK-19888 URL: https://issues.apache.org/jira/browse/SPARK-19888

[jira] [Commented] (SPARK-19353) Support binary I/O in PipedRDD

2017-03-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904058#comment-15904058 ] Apache Spark commented on SPARK-19353: -- User 'superbobry' has created a pull request for this issue:

[jira] [Created] (SPARK-19891) Await Batch Lock not signaled on stream execution exit

2017-03-09 Thread Tyson Condie (JIRA)
Tyson Condie created SPARK-19891: Summary: Await Batch Lock not signaled on stream execution exit Key: SPARK-19891 URL: https://issues.apache.org/jira/browse/SPARK-19891 Project: Spark Issue

[jira] [Assigned] (SPARK-19891) Await Batch Lock not signaled on stream execution exit

2017-03-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19891: Assignee: Apache Spark > Await Batch Lock not signaled on stream execution exit >

[jira] [Assigned] (SPARK-19891) Await Batch Lock not signaled on stream execution exit

2017-03-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19891: Assignee: (was: Apache Spark) > Await Batch Lock not signaled on stream execution

[jira] [Commented] (SPARK-19891) Await Batch Lock not signaled on stream execution exit

2017-03-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904116#comment-15904116 ] Apache Spark commented on SPARK-19891: -- User 'tcondie' has created a pull request for this issue:

[jira] [Reopened] (SPARK-19507) pyspark.sql.types._verify_type() exceptions too broad to debug collections or nested data

2017-03-09 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-19507: -- I am reopening this per https://github.com/apache/spark/pull/17213#issuecomment-285530248 and

[jira] [Resolved] (SPARK-19871) Improve error message in verify_type to indicate which field the error is for

2017-03-09 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-19871. -- Resolution: Duplicate > Improve error message in verify_type to indicate which field the error

[jira] [Assigned] (SPARK-19507) pyspark.sql.types._verify_type() exceptions too broad to debug collections or nested data

2017-03-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19507: Assignee: (was: Apache Spark) > pyspark.sql.types._verify_type() exceptions too broad

[jira] [Assigned] (SPARK-19507) pyspark.sql.types._verify_type() exceptions too broad to debug collections or nested data

2017-03-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19507: Assignee: Apache Spark > pyspark.sql.types._verify_type() exceptions too broad to debug

[jira] [Resolved] (SPARK-19886) reportDataLoss cause != null check is wrong for Structured Streaming KafkaSource

2017-03-09 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19886. -- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > reportDataLoss cause

[jira] [Assigned] (SPARK-19893) Cannot run intersect/except with map type

2017-03-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19893: Assignee: Apache Spark (was: Wenchen Fan) > Cannot run intersect/except with map type >

[jira] [Assigned] (SPARK-19893) Cannot run intersect/except with map type

2017-03-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19893: Assignee: Wenchen Fan (was: Apache Spark) > Cannot run intersect/except with map type >

[jira] [Created] (SPARK-19894) Tasks entirely assigned to one executor on Yarn-cluster mode for default-rack

2017-03-09 Thread Yuechen Chen (JIRA)
Yuechen Chen created SPARK-19894: Summary: Tasks entirely assigned to one executor on Yarn-cluster mode for default-rack Key: SPARK-19894 URL: https://issues.apache.org/jira/browse/SPARK-19894

[jira] [Resolved] (SPARK-19891) Await Batch Lock not signaled on stream execution exit

2017-03-09 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19891. -- Resolution: Fixed Assignee: Tyson Condie Fix Version/s: 2.2.0

[jira] [Assigned] (SPARK-19852) StringIndexer.setHandleInvalid should have another option 'new': Python API and docs

2017-03-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19852: Assignee: Apache Spark > StringIndexer.setHandleInvalid should have another option 'new':

[jira] [Commented] (SPARK-19852) StringIndexer.setHandleInvalid should have another option 'new': Python API and docs

2017-03-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904590#comment-15904590 ] Apache Spark commented on SPARK-19852: -- User 'VinceShieh' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19852) StringIndexer.setHandleInvalid should have another option 'new': Python API and docs

2017-03-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19852: Assignee: (was: Apache Spark) > StringIndexer.setHandleInvalid should have another

[jira] [Commented] (SPARK-19885) The config ignoreCorruptFiles doesn't work for CSV

2017-03-09 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904606#comment-15904606 ] Wenchen Fan commented on SPARK-19885: - This is because we support different charset for CSV files,

[jira] [Comment Edited] (SPARK-19885) The config ignoreCorruptFiles doesn't work for CSV

2017-03-09 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904606#comment-15904606 ] Wenchen Fan edited comment on SPARK-19885 at 3/10/17 7:22 AM: -- This is

[jira] [Commented] (SPARK-19894) Tasks entirely assigned to one executor on Yarn-cluster mode for default-rack

2017-03-09 Thread Yuechen Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904621#comment-15904621 ] Yuechen Chen commented on SPARK-19894: -- https://github.com/apache/spark/pull/17238 > Tasks entirely

[jira] [Resolved] (SPARK-16255) Spark2.0 doesn't support the following SQL statement:"insert into directory "/u_qa_user/hive_testdata/test1/t1" select * from d_test_tpc_2g_txt.auction" while Hive supp

2017-03-09 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-16255. -- Resolution: Duplicate I think this is a duplicate of SPARK-4131. Please reopen this if I was

[jira] [Updated] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

2017-03-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-18112: Priority: Critical (was: Major) > Spark2.x does not support read data from Hive 2.x metastore >

[jira] [Commented] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

2017-03-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904362#comment-15904362 ] Xiao Li commented on SPARK-18112: - Let me resolve it for supporting Hive 2.1.0 metastore. > Spark2.x

[jira] [Updated] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

2017-03-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-18112: Component/s: (was: Spark Submit) > Spark2.x does not support read data from Hive 2.x metastore >

[jira] [Assigned] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

2017-03-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-18112: --- Assignee: Xiao Li > Spark2.x does not support read data from Hive 2.x metastore >

[jira] [Commented] (SPARK-16754) NPE when defining case class and searching Encoder in the same line

2017-03-09 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904272#comment-15904272 ] Hyukjin Kwon commented on SPARK-16754: -- Today, I just tested this for my curiosity. It seems prints

[jira] [Commented] (SPARK-17322) 'ANY n' clause for SQL queries to increase the ease of use of WHERE clause predicates

2017-03-09 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904285#comment-15904285 ] Hyukjin Kwon commented on SPARK-17322: -- Let me leave a link that might be helpful -

[jira] [Commented] (SPARK-19711) Bug in gapply function

2017-03-09 Thread Yeonseop Kim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904346#comment-15904346 ] Yeonseop Kim commented on SPARK-19711: -- It seems to be more nice Using "stringsAsFactor = FALSE",

[jira] [Reopened] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

2017-03-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reopened SPARK-18112: - > Spark2.x does not support read data from Hive 2.x metastore >

[jira] [Comment Edited] (SPARK-16283) Implement percentile_approx SQL function

2017-03-09 Thread Zhenhua Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904397#comment-15904397 ] Zhenhua Wang edited comment on SPARK-16283 at 3/10/17 4:09 AM: --- [~erlu] I

[jira] [Commented] (SPARK-16283) Implement percentile_approx SQL function

2017-03-09 Thread Zhenhua Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904397#comment-15904397 ] Zhenhua Wang commented on SPARK-16283: -- [~erlu] I think it's been made clear from above discussions,

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

2017-03-09 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904497#comment-15904497 ] Saisai Shao commented on SPARK-19143: - Hi all, I wrote a rough design doc based on the comments

[jira] [Commented] (SPARK-19893) Cannot run intersect/except with map type

2017-03-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904587#comment-15904587 ] Apache Spark commented on SPARK-19893: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19008) Avoid boxing/unboxing overhead of calling a lambda with primitive type from Dataset program

2017-03-09 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-19008: --- Assignee: Kazuaki Ishizaki > Avoid boxing/unboxing overhead of calling a lambda with

[jira] [Created] (SPARK-19893) Cannot run intersect/except with map type

2017-03-09 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-19893: --- Summary: Cannot run intersect/except with map type Key: SPARK-19893 URL: https://issues.apache.org/jira/browse/SPARK-19893 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-19008) Avoid boxing/unboxing overhead of calling a lambda with primitive type from Dataset program

2017-03-09 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-19008. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17172

[jira] [Commented] (SPARK-14409) Investigate adding a RankingEvaluator to ML

2017-03-09 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902649#comment-15902649 ] Nick Pentreath commented on SPARK-14409: [~josephkb] in reference to your [PR

[jira] [Commented] (SPARK-19875) Map->filter on many columns gets stuck in constraint inference optimization code

2017-03-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902668#comment-15902668 ] Sean Owen commented on SPARK-19875: --- It's easier to inline the code in a comment: {code:scala} package

[jira] [Comment Edited] (SPARK-19875) Map->filter on many columns gets stuck in constraint inference optimization code

2017-03-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902668#comment-15902668 ] Sean Owen edited comment on SPARK-19875 at 3/9/17 8:32 AM: --- It's easier to

[jira] [Resolved] (SPARK-19880) About spark2.0.2 and spark1.4.1 beeline to show the database, use the default operation such as dealing with different

2017-03-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19880. --- Resolution: Invalid I'm not clear what this means. It may be better to clarify and re-ask on the

[jira] [Issue Comment Deleted] (SPARK-19862) In SparkEnv.scala,shortShuffleMgrNames tungsten-sort can be deleted.

2017-03-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-19862: -- Comment: was deleted (was: It's true, though I'm not sure this alone is worth changing, let alone a

[jira] [Resolved] (SPARK-19862) In SparkEnv.scala,shortShuffleMgrNames tungsten-sort can be deleted.

2017-03-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19862. --- Resolution: Won't Fix > In SparkEnv.scala,shortShuffleMgrNames tungsten-sort can be deleted. >

[jira] [Commented] (SPARK-12694) The detailed rest API documentation for each field is missing

2017-03-09 Thread Ehsun Behravesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902689#comment-15902689 ] Ehsun Behravesh commented on SPARK-12694: - So this JIRA ticket should be close? > The detailed

[jira] [Assigned] (SPARK-19763) qualified external datasource table location stored in catalog

2017-03-09 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-19763: --- Assignee: Song Jun > qualified external datasource table location stored in catalog >

[jira] [Comment Edited] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

2017-03-09 Thread Swaranga Sarma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902634#comment-15902634 ] Swaranga Sarma edited comment on SPARK-12009 at 3/9/17 7:59 AM: The JIRA

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

2017-03-09 Thread Swaranga Sarma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902634#comment-15902634 ] Swaranga Sarma commented on SPARK-12009: The JIRA says that the issue is fixed but I still see

[jira] [Resolved] (SPARK-12694) The detailed rest API documentation for each field is missing

2017-03-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-12694. --- Resolution: Duplicate Yes probably superseded by other changes > The detailed rest API

[jira] [Commented] (SPARK-19880) About spark2.0.2 and spark1.4.1 beeline to show the database, use the default operation such as dealing with different

2017-03-09 Thread guoxiaolong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902679#comment-15902679 ] guoxiaolong commented on SPARK-19880: - But When a job dispose time is very long,such as query

[jira] [Commented] (SPARK-12694) The detailed rest API documentation for each field is missing

2017-03-09 Thread Ehsun Behravesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902692#comment-15902692 ] Ehsun Behravesh commented on SPARK-12694: - Thanks > The detailed rest API documentation for each

[jira] [Commented] (SPARK-12694) The detailed rest API documentation for each field is missing

2017-03-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902690#comment-15902690 ] Sean Owen commented on SPARK-12694: --- Already done, yes. > The detailed rest API documentation for each

[jira] [Issue Comment Deleted] (SPARK-12694) The detailed rest API documentation for each field is missing

2017-03-09 Thread Ehsun Behravesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ehsun Behravesh updated SPARK-12694: Comment: was deleted (was: So this JIRA ticket should be close?) > The detailed rest API

[jira] [Resolved] (SPARK-19763) qualified external datasource table location stored in catalog

2017-03-09 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-19763. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17095

  1   2   >