[jira] [Commented] (SPARK-15790) Audit @Since annotations in ML

2017-03-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904698#comment-15904698 ] Sean Owen commented on SPARK-15790: --- [~ehsun7b] you're welcome to look for any public methods without

[jira] [Commented] (SPARK-19885) The config ignoreCorruptFiles doesn't work for CSV

2017-03-10 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904747#comment-15904747 ] Hyukjin Kwon commented on SPARK-19885: -- Thank you for cc'ing me. Up to my knowledge, {{LineReader}}

[jira] [Updated] (SPARK-19893) Cannot run intersect/except/distinct with map type

2017-03-10 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-19893: Summary: Cannot run intersect/except/distinct with map type (was: Cannot run intersect/except

[jira] [Commented] (SPARK-19885) The config ignoreCorruptFiles doesn't work for CSV

2017-03-10 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904810#comment-15904810 ] Hyukjin Kwon commented on SPARK-19885: -- Also, we recently introduced reading a CSV from text

[jira] [Commented] (SPARK-17080) join reorder

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904813#comment-15904813 ] Apache Spark commented on SPARK-17080: -- User 'wzhfy' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19901) Clean up the clunky method signature of acquireMemory

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19901: Assignee: Apache Spark > Clean up the clunky method signature of acquireMemory >

[jira] [Assigned] (SPARK-19901) Clean up the clunky method signature of acquireMemory

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19901: Assignee: (was: Apache Spark) > Clean up the clunky method signature of acquireMemory

[jira] [Commented] (SPARK-19901) Clean up the clunky method signature of acquireMemory

2017-03-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905114#comment-15905114 ] Sean Owen commented on SPARK-19901: --- I don't see a problem statement here or why that's an improvement.

[jira] [Created] (SPARK-19902) Support more expression canonicalization: Add, Subtract, Multiply and Divide

2017-03-10 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-19902: --- Summary: Support more expression canonicalization: Add, Subtract, Multiply and Divide Key: SPARK-19902 URL: https://issues.apache.org/jira/browse/SPARK-19902

[jira] [Commented] (SPARK-19902) Support more expression canonicalization: Add, Subtract, Multiply and Divide

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905104#comment-15905104 ] Apache Spark commented on SPARK-19902: -- User 'viirya' has created a pull request for this issue:

[jira] [Updated] (SPARK-19900) [Standalone] Master registers application again when driver relaunched

2017-03-10 Thread Sergey (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey updated SPARK-19900: --- Description: I've found some problems when node, where driver is running, has unstable network. A situation

[jira] [Commented] (SPARK-19901) Clean up the clunky method signature of acquireMemory

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905098#comment-15905098 ] Apache Spark commented on SPARK-19901: -- User 'ConeyLiu' has created a pull request for this issue:

[jira] [Updated] (SPARK-19899) FPGrowth input column naming

2017-03-10 Thread Maciej Szymkiewicz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-19899: --- Description: Current implementation extends {{HasFeaturesCol}}. Personally I find it

[jira] [Assigned] (SPARK-19889) Make TaskContext callbacks synchronized

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19889: Assignee: Apache Spark > Make TaskContext callbacks synchronized >

[jira] [Commented] (SPARK-19889) Make TaskContext callbacks synchronized

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905207#comment-15905207 ] Apache Spark commented on SPARK-19889: -- User 'hvanhovell' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19889) Make TaskContext callbacks synchronized

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19889: Assignee: (was: Apache Spark) > Make TaskContext callbacks synchronized >

[jira] [Updated] (SPARK-19889) Make TaskContext callbacks synchronized

2017-03-10 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell updated SPARK-19889: -- Summary: Make TaskContext callbacks synchronized (was: Make TaskContext synchronized)

[jira] [Assigned] (SPARK-19902) Support more expression canonicalization: Add, Subtract, Multiply and Divide

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19902: Assignee: (was: Apache Spark) > Support more expression canonicalization: Add,

[jira] [Assigned] (SPARK-19902) Support more expression canonicalization: Add, Subtract, Multiply and Divide

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19902: Assignee: Apache Spark > Support more expression canonicalization: Add, Subtract,

[jira] [Commented] (SPARK-19802) Remote History Server

2017-03-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905115#comment-15905115 ] Sean Owen commented on SPARK-19802: --- Yeah, I am not sure that the refactoring and change that would be

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

2017-03-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905247#comment-15905247 ] Thomas Graves commented on SPARK-19143: --- Made some comments in the design doc. My original idea

[jira] [Updated] (SPARK-19889) Make TaskContext callbacks synchronized

2017-03-10 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell updated SPARK-19889: -- Description: In some cases you want to fork of some part of a task to a different

[jira] [Created] (SPARK-19901) Clean up the clunky method signature of acquireMemory

2017-03-10 Thread coneyliu (JIRA)
coneyliu created SPARK-19901: Summary: Clean up the clunky method signature of acquireMemory Key: SPARK-19901 URL: https://issues.apache.org/jira/browse/SPARK-19901 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-19821) Throw out the Read-only disk information when create file for Shuffle

2017-03-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19821. --- Resolution: Not A Problem > Throw out the Read-only disk information when create file for Shuffle >

[jira] [Commented] (SPARK-19901) Clean up the clunky method signature of acquireMemory

2017-03-10 Thread coneyliu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905159#comment-15905159 ] coneyliu commented on SPARK-19901: -- Hi [~srowen], this patch is used to streamline the method signature,

[jira] [Comment Edited] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

2017-03-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905247#comment-15905247 ] Thomas Graves edited comment on SPARK-19143 at 3/10/17 3:20 PM: Made some

[jira] [Resolved] (SPARK-19786) Facilitate loop optimizations in a JIT compiler regarding range()

2017-03-10 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-19786. --- Resolution: Fixed Assignee: Kazuaki Ishizaki Fix Version/s: 2.2.0 >

[jira] [Commented] (SPARK-19899) FPGrowth input column naming

2017-03-10 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905484#comment-15905484 ] yuhao yang commented on SPARK-19899: Thanks for the reply. We can wait for some time to see if people

[jira] [Updated] (SPARK-19887) __HIVE_DEFAULT_PARTITION__ is not interpreted as NULL partition value in partitioned persisted tables

2017-03-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-19887: --- Description: The following Spark shell snippet under Spark 2.1 reproduces this issue: {code} val

[jira] [Resolved] (SPARK-19620) Incorrect exchange coordinator Id in physical plan

2017-03-10 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai resolved SPARK-19620. -- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 16952

[jira] [Closed] (SPARK-19907) Spark Submit Does not pick up the HBase Jars

2017-03-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen closed SPARK-19907. - > Spark Submit Does not pick up the HBase Jars > > >

[jira] [Updated] (SPARK-19893) should not run DataFrame set oprations with map type

2017-03-10 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-19893: Summary: should not run DataFrame set oprations with map type (was: Cannot run

[jira] [Commented] (SPARK-14453) Remove SPARK_JAVA_OPTS environment variable

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905518#comment-15905518 ] Apache Spark commented on SPARK-14453: -- User 'yongtang' has created a pull request for this issue:

[jira] [Created] (SPARK-19906) Add Documentation for Kafka Write paths

2017-03-10 Thread Tyson Condie (JIRA)
Tyson Condie created SPARK-19906: Summary: Add Documentation for Kafka Write paths Key: SPARK-19906 URL: https://issues.apache.org/jira/browse/SPARK-19906 Project: Spark Issue Type:

[jira] [Updated] (SPARK-19887) __HIVE_DEFAULT_PARTITION__ is not interpreted as NULL partition value in partitioned persisted tables

2017-03-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-19887: --- Description: The following Spark shell snippet under Spark 2.1 reproduces this issue: {code} val

[jira] [Commented] (SPARK-19885) The config ignoreCorruptFiles doesn't work for CSV

2017-03-10 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905564#comment-15905564 ] Wenchen Fan commented on SPARK-19885: - Oh, so this issue is already fixed by SPARK-18362 in Spark 2.2

[jira] [Created] (SPARK-19905) Dataset.inputFiles is broken for Hive SerDe tables

2017-03-10 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-19905: -- Summary: Dataset.inputFiles is broken for Hive SerDe tables Key: SPARK-19905 URL: https://issues.apache.org/jira/browse/SPARK-19905 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-19885) The config ignoreCorruptFiles doesn't work for CSV

2017-03-10 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-19885. - Resolution: Fixed Fix Version/s: 2.2.0 > The config ignoreCorruptFiles doesn't work for

[jira] [Assigned] (SPARK-19620) Incorrect exchange coordinator Id in physical plan

2017-03-10 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai reassigned SPARK-19620: Assignee: Carson Wang > Incorrect exchange coordinator Id in physical plan >

[jira] [Resolved] (SPARK-19907) Spark Submit Does not pick up the HBase Jars

2017-03-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19907. --- Resolution: Invalid Target Version/s: (was: 2.0.0) A huge dump of your config and logs

[jira] [Commented] (SPARK-19899) FPGrowth input column naming

2017-03-10 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905451#comment-15905451 ] yuhao yang commented on SPARK-19899: {quote} if we mix-in HasFeaturesCol the featuresCol should be

[jira] [Commented] (SPARK-19899) FPGrowth input column naming

2017-03-10 Thread Maciej Szymkiewicz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905480#comment-15905480 ] Maciej Szymkiewicz commented on SPARK-19899: This is just an idea, but I would start with: -

[jira] [Commented] (SPARK-19899) FPGrowth input column naming

2017-03-10 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905492#comment-15905492 ] yuhao yang commented on SPARK-19899: also cc [~podongfeng] since I recalled he mentioned to use

[jira] [Commented] (SPARK-19906) Add Documentation for Kafka Write paths

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905580#comment-15905580 ] Apache Spark commented on SPARK-19906: -- User 'tcondie' has created a pull request for this issue:

[jira] [Created] (SPARK-19907) Spark Submit Does not pick up the HBase Jars

2017-03-10 Thread Ramchandhar Rapolu (JIRA)
Ramchandhar Rapolu created SPARK-19907: -- Summary: Spark Submit Does not pick up the HBase Jars Key: SPARK-19907 URL: https://issues.apache.org/jira/browse/SPARK-19907 Project: Spark

[jira] [Assigned] (SPARK-19906) Add Documentation for Kafka Write paths

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19906: Assignee: (was: Apache Spark) > Add Documentation for Kafka Write paths >

[jira] [Assigned] (SPARK-19906) Add Documentation for Kafka Write paths

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19906: Assignee: Apache Spark > Add Documentation for Kafka Write paths >

[jira] [Updated] (SPARK-19900) [Standalone] Master registers application again when driver relaunched

2017-03-10 Thread Sergey (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey updated SPARK-19900: --- Description: I've found some problems when node, where driver is running, has unstable network. A situation

[jira] [Created] (SPARK-19903) PySpark Kafka streaming query ouput append mode not possible

2017-03-10 Thread Piotr Nestorow (JIRA)
Piotr Nestorow created SPARK-19903: -- Summary: PySpark Kafka streaming query ouput append mode not possible Key: SPARK-19903 URL: https://issues.apache.org/jira/browse/SPARK-19903 Project: Spark

[jira] [Commented] (SPARK-19850) Support aliased expressions in function parameters

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905404#comment-15905404 ] Apache Spark commented on SPARK-19850: -- User 'hvanhovell' has created a pull request for this issue:

[jira] [Created] (SPARK-19904) SPIP Add Spark Project Improvement Proposal doc to website

2017-03-10 Thread Cody Koeninger (JIRA)
Cody Koeninger created SPARK-19904: -- Summary: SPIP Add Spark Project Improvement Proposal doc to website Key: SPARK-19904 URL: https://issues.apache.org/jira/browse/SPARK-19904 Project: Spark

[jira] [Commented] (SPARK-19899) FPGrowth input column naming

2017-03-10 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905367#comment-15905367 ] yuhao yang commented on SPARK-19899: Thanks for the suggestion. I'm neutral on this. Not sure if we

[jira] [Commented] (SPARK-19899) FPGrowth input column naming

2017-03-10 Thread Maciej Szymkiewicz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905405#comment-15905405 ] Maciej Szymkiewicz commented on SPARK-19899: In my opinion a trait for each input category

[jira] [Assigned] (SPARK-19850) Support aliased expressions in function parameters

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19850: Assignee: Herman van Hovell (was: Apache Spark) > Support aliased expressions in

[jira] [Assigned] (SPARK-19850) Support aliased expressions in function parameters

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19850: Assignee: Apache Spark (was: Herman van Hovell) > Support aliased expressions in

[jira] [Resolved] (SPARK-18270) Users schema with non-nullable properties is overidden with true

2017-03-10 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-18270. -- Resolution: Duplicate I am resolving this as a duplicate of SPARK-16472. Please reopen this if

[jira] [Comment Edited] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0

2017-03-10 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905716#comment-15905716 ] Shixiong Zhu edited comment on SPARK-18057 at 3/10/17 9:21 PM: --- I did some

[jira] [Created] (SPARK-19911) Add builder interface for Kinesis DStreams

2017-03-10 Thread Adam Budde (JIRA)
Adam Budde created SPARK-19911: -- Summary: Add builder interface for Kinesis DStreams Key: SPARK-19911 URL: https://issues.apache.org/jira/browse/SPARK-19911 Project: Spark Issue Type: New

[jira] [Resolved] (SPARK-17979) Remove deprecated support for config SPARK_YARN_USER_ENV

2017-03-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-17979. Resolution: Fixed Assignee: Yong Tang Fix Version/s: 2.2.0 > Remove

[jira] [Resolved] (SPARK-14453) Remove SPARK_JAVA_OPTS environment variable

2017-03-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-14453. Resolution: Fixed Assignee: Yong Tang Fix Version/s: 2.2.0 > Remove

[jira] [Comment Edited] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0

2017-03-10 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905716#comment-15905716 ] Shixiong Zhu edited comment on SPARK-18057 at 3/10/17 9:21 PM: --- I did some

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0

2017-03-10 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905716#comment-15905716 ] Shixiong Zhu commented on SPARK-18057: -- I did some investigation yesterday, and found one issue in

[jira] [Commented] (SPARK-19888) Seeing offsets not resetting even when reset policy is configured explicitly

2017-03-10 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905730#comment-15905730 ] Cody Koeninger commented on SPARK-19888: That stacktrace also shows a concurrent modification

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0

2017-03-10 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905747#comment-15905747 ] Cody Koeninger commented on SPARK-18057: I think the bigger question is once there's a kafka

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0

2017-03-10 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905774#comment-15905774 ] Cody Koeninger commented on SPARK-18057: Based on previous kafka client upgrades I wouldn't

[jira] [Updated] (SPARK-19887) __HIVE_DEFAULT_PARTITION__ is not interpreted as NULL partition value in partitioned persisted tables

2017-03-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-19887: --- Affects Version/s: 2.2.0 > __HIVE_DEFAULT_PARTITION__ is not interpreted as NULL partition value in

[jira] [Commented] (SPARK-19611) Spark 2.1.0 breaks some Hive tables backed by case-sensitive data files

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905705#comment-15905705 ] Apache Spark commented on SPARK-19611: -- User 'budde' has created a pull request for this issue:

[jira] [Commented] (SPARK-19911) Add builder interface for Kinesis DStreams

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905741#comment-15905741 ] Apache Spark commented on SPARK-19911: -- User 'budde' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19911) Add builder interface for Kinesis DStreams

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19911: Assignee: (was: Apache Spark) > Add builder interface for Kinesis DStreams >

[jira] [Assigned] (SPARK-19911) Add builder interface for Kinesis DStreams

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19911: Assignee: Apache Spark > Add builder interface for Kinesis DStreams >

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0

2017-03-10 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905755#comment-15905755 ] Michael Armbrust commented on SPARK-18057: -- It seems like we can upgrade the existing Kafka10

[jira] [Commented] (SPARK-19910) `stack` should not reject NULL values due to type mismatch

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905766#comment-15905766 ] Apache Spark commented on SPARK-19910: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-19910) `stack` should not reject NULL values due to type mismatch

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19910: Assignee: Apache Spark > `stack` should not reject NULL values due to type mismatch >

[jira] [Assigned] (SPARK-19910) `stack` should not reject NULL values due to type mismatch

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19910: Assignee: (was: Apache Spark) > `stack` should not reject NULL values due to type

[jira] [Commented] (SPARK-19899) FPGrowth input column naming

2017-03-10 Thread Maciej Szymkiewicz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905834#comment-15905834 ] Maciej Szymkiewicz commented on SPARK-19899: Thanks [~yuhaoyan]. > FPGrowth input column

[jira] [Commented] (SPARK-19863) Whether or not use CachedKafkaConsumer need to be configured, when you use DirectKafkaInputDStream to connect the kafka in a Spark Streaming application

2017-03-10 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905603#comment-15905603 ] Cody Koeninger commented on SPARK-19863: Isn't this basically a duplicate of SPARK-19185 with the

[jira] [Commented] (SPARK-19905) Dataset.inputFiles is broken for Hive SerDe tables

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905608#comment-15905608 ] Apache Spark commented on SPARK-19905: -- User 'liancheng' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19905) Dataset.inputFiles is broken for Hive SerDe tables

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19905: Assignee: Apache Spark (was: Cheng Lian) > Dataset.inputFiles is broken for Hive SerDe

[jira] [Created] (SPARK-19909) Batches will fail in case that temporary checkpoint dir is on local file system while metadata dir is on HDFS

2017-03-10 Thread Kousuke Saruta (JIRA)
Kousuke Saruta created SPARK-19909: -- Summary: Batches will fail in case that temporary checkpoint dir is on local file system while metadata dir is on HDFS Key: SPARK-19909 URL:

[jira] [Commented] (SPARK-19909) Batches will fail in case that temporary checkpoint dir is on local file system while metadata dir is on HDFS

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905641#comment-15905641 ] Apache Spark commented on SPARK-19909: -- User 'sarutak' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19909) Batches will fail in case that temporary checkpoint dir is on local file system while metadata dir is on HDFS

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19909: Assignee: (was: Apache Spark) > Batches will fail in case that temporary checkpoint

[jira] [Updated] (SPARK-19904) SPIP Add Spark Project Improvement Proposal doc to website

2017-03-10 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cody Koeninger updated SPARK-19904: --- Description: see

[jira] [Assigned] (SPARK-19909) Batches will fail in case that temporary checkpoint dir is on local file system while metadata dir is on HDFS

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19909: Assignee: Apache Spark > Batches will fail in case that temporary checkpoint dir is on

[jira] [Created] (SPARK-19908) Direct buffer memory OOM should not cause stage retries.

2017-03-10 Thread Zhan Zhang (JIRA)
Zhan Zhang created SPARK-19908: -- Summary: Direct buffer memory OOM should not cause stage retries. Key: SPARK-19908 URL: https://issues.apache.org/jira/browse/SPARK-19908 Project: Spark Issue

[jira] [Assigned] (SPARK-19905) Dataset.inputFiles is broken for Hive SerDe tables

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19905: Assignee: Cheng Lian (was: Apache Spark) > Dataset.inputFiles is broken for Hive SerDe

[jira] [Updated] (SPARK-19888) Seeing offsets not resetting even when reset policy is configured explicitly

2017-03-10 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-19888: - Component/s: (was: Spark Core) DStreams > Seeing offsets not

[jira] [Created] (SPARK-19910) `stack` should not reject NULL values due to type mismatch

2017-03-10 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-19910: - Summary: `stack` should not reject NULL values due to type mismatch Key: SPARK-19910 URL: https://issues.apache.org/jira/browse/SPARK-19910 Project: Spark

[jira] [Resolved] (SPARK-19905) Dataset.inputFiles is broken for Hive SerDe tables

2017-03-10 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-19905. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17247

[jira] [Assigned] (SPARK-19913) Log warning rather than throw AnalysisException when output is partitioned although format is memory, console or foreach

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19913: Assignee: (was: Apache Spark) > Log warning rather than throw AnalysisException when

[jira] [Resolved] (SPARK-19893) should not run DataFrame set oprations with map type

2017-03-10 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-19893. - Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 2.0.3

[jira] [Updated] (SPARK-19611) Spark 2.1.0 breaks some Hive tables backed by case-sensitive data files

2017-03-10 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-19611: Fix Version/s: 2.1.1 > Spark 2.1.0 breaks some Hive tables backed by case-sensitive data files >

[jira] [Updated] (SPARK-19912) String literals are not escaped while performing partition pruning at Hive metastore level

2017-03-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-19912: --- Description: {{Shim_v0_13.convertFilters()}} doesn't escape string literals while generating Hive

[jira] [Commented] (SPARK-19913) Log warning rather than throw AnalysisException when output is partitioned although format is memory, console or foreach

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905904#comment-15905904 ] Apache Spark commented on SPARK-19913: -- User 'sarutak' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19913) Log warning rather than throw AnalysisException when output is partitioned although format is memory, console or foreach

2017-03-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19913: Assignee: Apache Spark > Log warning rather than throw AnalysisException when output is

[jira] [Updated] (SPARK-19887) __HIVE_DEFAULT_PARTITION__ is not interpreted as NULL partition value in partitioned persisted tables

2017-03-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-19887: --- Labels: correctness (was: ) > __HIVE_DEFAULT_PARTITION__ is not interpreted as NULL partition value

[jira] [Created] (SPARK-19912) String literals are not escaped while performing partition pruning at Hive metastore level

2017-03-10 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-19912: -- Summary: String literals are not escaped while performing partition pruning at Hive metastore level Key: SPARK-19912 URL: https://issues.apache.org/jira/browse/SPARK-19912

[jira] [Updated] (SPARK-19912) String literals are not escaped while performing Hive metastore level partition pruning

2017-03-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-19912: --- Summary: String literals are not escaped while performing Hive metastore level partition pruning

[jira] [Created] (SPARK-19913) Log warning rather than throw AnalysisException when output is partitioned although format is memory, console or foreach

2017-03-10 Thread Kousuke Saruta (JIRA)
Kousuke Saruta created SPARK-19913: -- Summary: Log warning rather than throw AnalysisException when output is partitioned although format is memory, console or foreach Key: SPARK-19913 URL:

[jira] [Created] (SPARK-19914) Spark Scala - Calling persist after reading a parquet file makes certain spark.sql queries return empty results

2017-03-10 Thread Yifeng Li (JIRA)
Yifeng Li created SPARK-19914: - Summary: Spark Scala - Calling persist after reading a parquet file makes certain spark.sql queries return empty results Key: SPARK-19914 URL:

[jira] [Commented] (SPARK-14503) spark.ml Scala API for FPGrowth

2017-03-10 Thread Maciej Szymkiewicz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905840#comment-15905840 ] Maciej Szymkiewicz commented on SPARK-14503: I think we should keep only unique predictions

  1   2   >