[jira] [Resolved] (SPARK-21238) allow nested SQL execution

2017-06-28 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21238. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18450 [https://githu

[jira] [Assigned] (SPARK-3577) Add task metric to report spill time

2017-06-28 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-3577: -- Assignee: Sital Kedia > Add task metric to report spill time >

[jira] [Resolved] (SPARK-3577) Add task metric to report spill time

2017-06-28 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-3577. Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 17471 [https://github.c

[jira] [Created] (SPARK-21249) Is it possible to use File Sink with mapGroupsWithState in Structured Streaming?

2017-06-28 Thread Amit Baghel (JIRA)
Amit Baghel created SPARK-21249: --- Summary: Is it possible to use File Sink with mapGroupsWithState in Structured Streaming? Key: SPARK-21249 URL: https://issues.apache.org/jira/browse/SPARK-21249 Projec

[jira] [Commented] (SPARK-21093) Multiple gapply execution occasionally failed in SparkR

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067679#comment-16067679 ] Apache Spark commented on SPARK-21093: -- User 'HyukjinKwon' has created a pull reques

[jira] [Updated] (SPARK-21223) Thread-safety issue in FsHistoryProvider

2017-06-28 Thread zenglinxi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zenglinxi updated SPARK-21223: -- Attachment: historyserver_jstack.txt BTW, this cause an infinite loop problem when we restart historyse

[jira] [Resolved] (SPARK-21237) Invalidate stats once table data is changed

2017-06-28 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21237. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18449 [https://githu

[jira] [Assigned] (SPARK-21237) Invalidate stats once table data is changed

2017-06-28 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-21237: --- Assignee: Zhenhua Wang > Invalidate stats once table data is changed > -

[jira] [Resolved] (SPARK-21229) remove QueryPlan.preCanonicalized

2017-06-28 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21229. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18440 [https://githu

[jira] [Issue Comment Deleted] (SPARK-21208) Ability to "setLocalProperty" from sc, in sparkR

2017-06-28 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-21208: - Comment: was deleted (was: User 'HyukjinKwon' has created a pull request for this issue: https://

[jira] [Assigned] (SPARK-21093) Multiple gapply execution occasionally failed in SparkR

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21093: Assignee: Apache Spark (was: Hyukjin Kwon) > Multiple gapply execution occasionally faile

[jira] [Assigned] (SPARK-21093) Multiple gapply execution occasionally failed in SparkR

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21093: Assignee: Hyukjin Kwon (was: Apache Spark) > Multiple gapply execution occasionally faile

[jira] [Comment Edited] (SPARK-21093) Multiple gapply execution occasionally failed in SparkR

2017-06-28 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067637#comment-16067637 ] Felix Cheung edited comment on SPARK-21093 at 6/29/17 3:12 AM:

[jira] [Reopened] (SPARK-21093) Multiple gapply execution occasionally failed in SparkR

2017-06-28 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung reopened SPARK-21093: -- this was reverted. > Multiple gapply execution occasionally failed in SparkR > -

[jira] [Commented] (SPARK-21246) Unexpected Data Type conversion from LONG to BIGINT

2017-06-28 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067620#comment-16067620 ] Yuming Wang commented on SPARK-21246: - {{Seq(3)}} should be {{Seq(3L)}}, This works f

[jira] [Assigned] (SPARK-21224) Support a DDL-formatted string as schema in reading for R

2017-06-28 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung reassigned SPARK-21224: Assignee: Hyukjin Kwon > Support a DDL-formatted string as schema in reading for R > -

[jira] [Commented] (SPARK-21224) Support a DDL-formatted string as schema in reading for R

2017-06-28 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067608#comment-16067608 ] Felix Cheung commented on SPARK-21224: -- let's add this to from_json, gapply, dapply

[jira] [Updated] (SPARK-21225) decrease the Mem using for variable 'tasks' in function resourceOffers

2017-06-28 Thread Jiang Xingbo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiang Xingbo updated SPARK-21225: - Issue Type: Bug (was: Improvement) > decrease the Mem using for variable 'tasks' in function res

[jira] [Resolved] (SPARK-14657) RFormula output wrong features when formula w/o intercept

2017-06-28 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang resolved SPARK-14657. - Resolution: Fixed Fix Version/s: 2.3.0 > RFormula output wrong features when formula w/o i

[jira] [Resolved] (SPARK-21222) Move elimination of Distinct clause from analyzer to optimizer

2017-06-28 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21222. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18429 [https://githu

[jira] [Assigned] (SPARK-21222) Move elimination of Distinct clause from analyzer to optimizer

2017-06-28 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-21222: --- Assignee: Gengliang Wang > Move elimination of Distinct clause from analyzer to optimizer >

[jira] [Commented] (SPARK-18441) Add Smote in spark mlib and ml

2017-06-28 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067494#comment-16067494 ] yuhao yang commented on SPARK-18441: Move the Smote code to https://gist.github.com/

[jira] [Commented] (SPARK-21248) Flaky test: o.a.s.sql.kafka010.KafkaSourceSuite.assign from specific offsets (failOnDataLoss: true)

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067425#comment-16067425 ] Apache Spark commented on SPARK-21248: -- User 'zsxwing' has created a pull request fo

[jira] [Assigned] (SPARK-21248) Flaky test: o.a.s.sql.kafka010.KafkaSourceSuite.assign from specific offsets (failOnDataLoss: true)

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21248: Assignee: Apache Spark > Flaky test: o.a.s.sql.kafka010.KafkaSourceSuite.assign from speci

[jira] [Assigned] (SPARK-21248) Flaky test: o.a.s.sql.kafka010.KafkaSourceSuite.assign from specific offsets (failOnDataLoss: true)

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21248: Assignee: (was: Apache Spark) > Flaky test: o.a.s.sql.kafka010.KafkaSourceSuite.assign

[jira] [Created] (SPARK-21248) Flaky test: o.a.s.sql.kafka010.KafkaSourceSuite.assign from specific offsets (failOnDataLoss: true)

2017-06-28 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-21248: Summary: Flaky test: o.a.s.sql.kafka010.KafkaSourceSuite.assign from specific offsets (failOnDataLoss: true) Key: SPARK-21248 URL: https://issues.apache.org/jira/browse/SPARK-2124

[jira] [Assigned] (SPARK-21247) Allow case-insensitive type equality in Set operation

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21247: Assignee: (was: Apache Spark) > Allow case-insensitive type equality in Set operation

[jira] [Commented] (SPARK-21247) Allow case-insensitive type equality in Set operation

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067379#comment-16067379 ] Apache Spark commented on SPARK-21247: -- User 'dongjoon-hyun' has created a pull requ

[jira] [Assigned] (SPARK-21247) Allow case-insensitive type equality in Set operation

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21247: Assignee: Apache Spark > Allow case-insensitive type equality in Set operation > -

[jira] [Updated] (SPARK-21247) Allow case-insensitive type equality in Set operation

2017-06-28 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21247: -- Summary: Allow case-insensitive type equality in Set operation (was: Allow case-insensitive ty

[jira] [Created] (SPARK-21247) Allow case-insensitive type comparisions in Set operation

2017-06-28 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-21247: - Summary: Allow case-insensitive type comparisions in Set operation Key: SPARK-21247 URL: https://issues.apache.org/jira/browse/SPARK-21247 Project: Spark I

[jira] [Commented] (SPARK-21242) Allow spark executors to function in mesos w/ container networking enabled

2017-06-28 Thread John Leach (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067257#comment-16067257 ] John Leach commented on SPARK-21242: [~mgummelt] We are using this for our service an

[jira] [Commented] (SPARK-21184) QuantileSummaries implementation is wrong and QuantileSummariesSuite fails with larger n

2017-06-28 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067167#comment-16067167 ] Andrew Ray commented on SPARK-21184: Also the lookup queries are just wrong {code} s

[jira] [Created] (SPARK-21246) Unexpected Data Type conversion from LONG to BIGINT

2017-06-28 Thread Monica Raj (JIRA)
Monica Raj created SPARK-21246: -- Summary: Unexpected Data Type conversion from LONG to BIGINT Key: SPARK-21246 URL: https://issues.apache.org/jira/browse/SPARK-21246 Project: Spark Issue Type: B

[jira] [Created] (SPARK-21245) Resolve code duplication for classification/regression summarizers

2017-06-28 Thread Seth Hendrickson (JIRA)
Seth Hendrickson created SPARK-21245: Summary: Resolve code duplication for classification/regression summarizers Key: SPARK-21245 URL: https://issues.apache.org/jira/browse/SPARK-21245 Project: S

[jira] [Commented] (SPARK-13534) Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066938#comment-16066938 ] Apache Spark commented on SPARK-13534: -- User 'BryanCutler' has created a pull reques

[jira] [Resolved] (SPARK-21216) Streaming DataFrames fail to join with Hive tables

2017-06-28 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-21216. -- Resolution: Fixed Fix Version/s: 2.3.0 > Streaming DataFrames fail to join with Hive tab

[jira] [Commented] (SPARK-20889) SparkR grouped documentation for Column methods

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066910#comment-16066910 ] Apache Spark commented on SPARK-20889: -- User 'actuaryzhang' has created a pull reque

[jira] [Assigned] (SPARK-21241) Add intercept to StreamingLinearRegressionWithSGD

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21241: Assignee: Apache Spark > Add intercept to StreamingLinearRegressionWithSGD > -

[jira] [Commented] (SPARK-21241) Add intercept to StreamingLinearRegressionWithSGD

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066893#comment-16066893 ] Apache Spark commented on SPARK-21241: -- User 'SoulGuedria' has created a pull reques

[jira] [Assigned] (SPARK-21241) Add intercept to StreamingLinearRegressionWithSGD

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21241: Assignee: (was: Apache Spark) > Add intercept to StreamingLinearRegressionWithSGD > --

[jira] [Commented] (SPARK-21244) KMeans applied to processed text day clumps almost all documents into one cluster

2017-06-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066833#comment-16066833 ] Sean Owen commented on SPARK-21244: --- There's no detail here that suggests a Spark bug.

[jira] [Commented] (SPARK-20082) Incremental update of LDA model, by adding initialModel as start point

2017-06-28 Thread Mathieu DESPRIEE (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066816#comment-16066816 ] Mathieu DESPRIEE commented on SPARK-20082: -- I updated the PR. Basically, here i

[jira] [Updated] (SPARK-21244) KMeans applied to processed text day clumps almost all documents into one cluster

2017-06-28 Thread Nassir (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nassir updated SPARK-21244: --- Description: I have observed this problem for quite a while now regarding the implementation of pyspark KMea

[jira] [Created] (SPARK-21244) KMeans applied to processed text day clumps almost all documents into one cluster

2017-06-28 Thread Nassir (JIRA)
Nassir created SPARK-21244: -- Summary: KMeans applied to processed text day clumps almost all documents into one cluster Key: SPARK-21244 URL: https://issues.apache.org/jira/browse/SPARK-21244 Project: Spark

[jira] [Commented] (SPARK-20696) tf-idf document clustering with K-means in Apache Spark putting points into one cluster

2017-06-28 Thread Nassir (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066652#comment-16066652 ] Nassir commented on SPARK-20696: Unfortunately, I have not found a place to make this kno

[jira] [Updated] (SPARK-21241) Add intercept to StreamingLinearRegressionWithSGD

2017-06-28 Thread Soulaimane GUEDRIA (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Soulaimane GUEDRIA updated SPARK-21241: --- Description: StreamingLinearRegressionWithSGD class in PySpark is missing the setInte

[jira] [Updated] (SPARK-21241) Add intercept to StreamingLinearRegressionWithSGD

2017-06-28 Thread Soulaimane GUEDRIA (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Soulaimane GUEDRIA updated SPARK-21241: --- Summary: Add intercept to StreamingLinearRegressionWithSGD (was: Can't add intercept

[jira] [Updated] (SPARK-21242) Allow spark executors to function in mesos w/ container networking enabled

2017-06-28 Thread Tara Gildersleeve (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tara Gildersleeve updated SPARK-21242: -- Priority: Major (was: Minor) > Allow spark executors to function in mesos w/ container

[jira] [Commented] (SPARK-21233) Support pluggable offset storage

2017-06-28 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066517#comment-16066517 ] Cody Koeninger commented on SPARK-21233: You already have the choice of where you

[jira] [Created] (SPARK-21243) Limit the number of maps in a single shuffle fetch

2017-06-28 Thread Dhruve Ashar (JIRA)
Dhruve Ashar created SPARK-21243: Summary: Limit the number of maps in a single shuffle fetch Key: SPARK-21243 URL: https://issues.apache.org/jira/browse/SPARK-21243 Project: Spark Issue Type

[jira] [Updated] (SPARK-21242) Allow spark executors to function in mesos w/ container networking enabled

2017-06-28 Thread Tara Gildersleeve (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tara Gildersleeve updated SPARK-21242: -- Attachment: patch_1.patch > Allow spark executors to function in mesos w/ container net

[jira] [Created] (SPARK-21242) Allow spark executors to function in mesos w/ container networking enabled

2017-06-28 Thread Tara Gildersleeve (JIRA)
Tara Gildersleeve created SPARK-21242: - Summary: Allow spark executors to function in mesos w/ container networking enabled Key: SPARK-21242 URL: https://issues.apache.org/jira/browse/SPARK-21242

[jira] [Created] (SPARK-21241) Can't add intercept to StreamingLinearRegressionWithSGD

2017-06-28 Thread Soulaimane GUEDRIA (JIRA)
Soulaimane GUEDRIA created SPARK-21241: -- Summary: Can't add intercept to StreamingLinearRegressionWithSGD Key: SPARK-21241 URL: https://issues.apache.org/jira/browse/SPARK-21241 Project: Spark

[jira] [Commented] (SPARK-21227) Unicode in Json field causes AnalysisException when selecting from Dataframe

2017-06-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066471#comment-16066471 ] Sean Owen commented on SPARK-21227: --- Yes, I think this is ultimately related to two dif

[jira] [Assigned] (SPARK-21228) InSet incorrect handling of structs

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21228: Assignee: Apache Spark > InSet incorrect handling of structs > ---

[jira] [Assigned] (SPARK-21228) InSet incorrect handling of structs

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21228: Assignee: (was: Apache Spark) > InSet incorrect handling of structs >

[jira] [Commented] (SPARK-21228) InSet incorrect handling of structs

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066457#comment-16066457 ] Apache Spark commented on SPARK-21228: -- User 'bogdanrdc' has created a pull request

[jira] [Commented] (SPARK-21227) Unicode in Json field causes AnalysisException when selecting from Dataframe

2017-06-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066423#comment-16066423 ] Hyukjin Kwon commented on SPARK-21227: -- I took a quick look and the cause seems to b

[jira] [Commented] (SPARK-17091) Convert IN predicate to equivalent Parquet filter

2017-06-28 Thread Michael Styles (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066363#comment-16066363 ] Michael Styles commented on SPARK-17091: n Parquet 1.7, there as a bug involving

[jira] [Assigned] (SPARK-21240) Fix code style for constructing and stopping a SparkContext in UT

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21240: Assignee: Apache Spark > Fix code style for constructing and stopping a SparkContext in UT

[jira] [Commented] (SPARK-21240) Fix code style for constructing and stopping a SparkContext in UT

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066360#comment-16066360 ] Apache Spark commented on SPARK-21240: -- User 'jinxing64' has created a pull request

[jira] [Assigned] (SPARK-21240) Fix code style for constructing and stopping a SparkContext in UT

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21240: Assignee: (was: Apache Spark) > Fix code style for constructing and stopping a SparkCo

[jira] [Created] (SPARK-21240) Fix code style for constructing and stopping a SparkContext in UT

2017-06-28 Thread jin xing (JIRA)
jin xing created SPARK-21240: Summary: Fix code style for constructing and stopping a SparkContext in UT Key: SPARK-21240 URL: https://issues.apache.org/jira/browse/SPARK-21240 Project: Spark Is

[jira] [Comment Edited] (SPARK-21223) Thread-safety issue in FsHistoryProvider

2017-06-28 Thread zenglinxi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066319#comment-16066319 ] zenglinxi edited comment on SPARK-21223 at 6/28/17 10:42 AM: -

[jira] [Commented] (SPARK-21223) Thread-safety issue in FsHistoryProvider

2017-06-28 Thread zenglinxi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066319#comment-16066319 ] zenglinxi commented on SPARK-21223: --- ok, i will check SPARK-21078 first. > Thread-safe

[jira] [Assigned] (SPARK-19852) StringIndexer.setHandleInvalid should have another option 'new': Python API and docs

2017-06-28 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang reassigned SPARK-19852: --- Assignee: Vincent > StringIndexer.setHandleInvalid should have another option 'new': Python

[jira] [Commented] (SPARK-19852) StringIndexer.setHandleInvalid should have another option 'new': Python API and docs

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066289#comment-16066289 ] Apache Spark commented on SPARK-19852: -- User 'yanboliang' has created a pull request

[jira] [Assigned] (SPARK-21239) Support WAL recover in windows

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21239: Assignee: Apache Spark > Support WAL recover in windows > -- >

[jira] [Commented] (SPARK-21239) Support WAL recover in windows

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066282#comment-16066282 ] Apache Spark commented on SPARK-21239: -- User 'Myasuka' has created a pull request fo

[jira] [Assigned] (SPARK-21239) Support WAL recover in windows

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21239: Assignee: (was: Apache Spark) > Support WAL recover in windows > -

[jira] [Updated] (SPARK-21239) Support WAL recover in windows

2017-06-28 Thread Yun Tang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Tang updated SPARK-21239: - Description: When driver failed over, it will read WAL from HDFS by calling WriteAheadLogBackedBlockRDD.

[jira] [Created] (SPARK-21239) Support WAL recover in windows

2017-06-28 Thread Yun Tang (JIRA)
Yun Tang created SPARK-21239: Summary: Support WAL recover in windows Key: SPARK-21239 URL: https://issues.apache.org/jira/browse/SPARK-21239 Project: Spark Issue Type: Bug Components:

[jira] [Assigned] (SPARK-17091) Convert IN predicate to equivalent Parquet filter

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17091: Assignee: (was: Apache Spark) > Convert IN predicate to equivalent Parquet filter > --

[jira] [Commented] (SPARK-17091) Convert IN predicate to equivalent Parquet filter

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066222#comment-16066222 ] Apache Spark commented on SPARK-17091: -- User 'ptkool' has created a pull request for

[jira] [Assigned] (SPARK-17091) Convert IN predicate to equivalent Parquet filter

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17091: Assignee: Apache Spark > Convert IN predicate to equivalent Parquet filter > -

[jira] [Comment Edited] (SPARK-21233) Support pluggable offset storage

2017-06-28 Thread darion yaphet (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066179#comment-16066179 ] darion yaphet edited comment on SPARK-21233 at 6/28/17 9:14 AM: ---

[jira] [Comment Edited] (SPARK-21233) Support pluggable offset storage

2017-06-28 Thread darion yaphet (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066179#comment-16066179 ] darion yaphet edited comment on SPARK-21233 at 6/28/17 9:13 AM: ---

[jira] [Comment Edited] (SPARK-21233) Support pluggable offset storage

2017-06-28 Thread darion yaphet (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066179#comment-16066179 ] darion yaphet edited comment on SPARK-21233 at 6/28/17 9:13 AM: ---

[jira] [Updated] (SPARK-21137) Spark reads many small files slowly off local filesystem

2017-06-28 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-21137: --- Summary: Spark reads many small files slowly off local filesystem (was: Spark reads many sma

[jira] [Commented] (SPARK-21233) Support pluggable offset storage

2017-06-28 Thread darion yaphet (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066179#comment-16066179 ] darion yaphet commented on SPARK-21233: --- [Sean Owen|sro...@gmail.com] In Kafka-0.8

[jira] [Commented] (SPARK-18004) DataFrame filter Predicate push-down fails for Oracle Timestamp type columns

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066162#comment-16066162 ] Apache Spark commented on SPARK-18004: -- User 'SharpRay' has created a pull request f

[jira] [Commented] (SPARK-21227) Unicode in Json field causes AnalysisException when selecting from Dataframe

2017-06-28 Thread Seydou Dia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066151#comment-16066151 ] Seydou Dia commented on SPARK-21227: Hi [~hyukjin.kwon], thanks for confirming this.

[jira] [Assigned] (SPARK-21238) allow nested SQL execution

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21238: Assignee: Wenchen Fan (was: Apache Spark) > allow nested SQL execution >

[jira] [Commented] (SPARK-21238) allow nested SQL execution

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066130#comment-16066130 ] Apache Spark commented on SPARK-21238: -- User 'cloud-fan' has created a pull request

[jira] [Assigned] (SPARK-21238) allow nested SQL execution

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21238: Assignee: Apache Spark (was: Wenchen Fan) > allow nested SQL execution >

[jira] [Created] (SPARK-21238) allow nested SQL execution

2017-06-28 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-21238: --- Summary: allow nested SQL execution Key: SPARK-21238 URL: https://issues.apache.org/jira/browse/SPARK-21238 Project: Spark Issue Type: Improvement Co

[jira] [Assigned] (SPARK-21237) Invalidate stats once table data is changed

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21237: Assignee: Apache Spark > Invalidate stats once table data is changed > ---

[jira] [Commented] (SPARK-21237) Invalidate stats once table data is changed

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066097#comment-16066097 ] Apache Spark commented on SPARK-21237: -- User 'wzhfy' has created a pull request for

[jira] [Assigned] (SPARK-21237) Invalidate stats once table data is changed

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21237: Assignee: (was: Apache Spark) > Invalidate stats once table data is changed >

[jira] [Created] (SPARK-21237) Invalidate stats once table data is changed

2017-06-28 Thread Zhenhua Wang (JIRA)
Zhenhua Wang created SPARK-21237: Summary: Invalidate stats once table data is changed Key: SPARK-21237 URL: https://issues.apache.org/jira/browse/SPARK-21237 Project: Spark Issue Type: Sub-t

[jira] [Commented] (SPARK-21233) Support pluggable offset storage

2017-06-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066088#comment-16066088 ] Sean Owen commented on SPARK-21233: --- Where would you put it instead? Kafka already prov

[jira] [Resolved] (SPARK-21234) When the function returns Option[Iterator[_]] is None,then get on None will cause java.util.NoSuchElementException: None.get

2017-06-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21234. --- Resolution: Invalid Not if it's known the value exists. I don't see you've established any actual pr

[jira] [Commented] (SPARK-20889) SparkR grouped documentation for Column methods

2017-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066033#comment-16066033 ] Apache Spark commented on SPARK-20889: -- User 'actuaryzhang' has created a pull reque