[jira] [Resolved] (SPARK-22273) Fix key/value schema field names in HashMapGenerators.

2017-10-13 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22273. - Resolution: Fixed Assignee: Takuya Ueshin Fix Version/s: 2.3.0 2.2.1

[jira] [Commented] (SPARK-21549) Spark fails to complete job correctly in case of OutputFormat which do not write into hdfs

2017-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16204504#comment-16204504 ] Apache Spark commented on SPARK-21549: -- User 'mridulm' has created a pull request fo

[jira] [Resolved] (SPARK-21762) FileFormatWriter/BasicWriteTaskStatsTracker metrics collection fails if a new file isn't yet visible

2017-10-13 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-21762. - Resolution: Fixed Assignee: Steve Loughran Fix Version/s: 2.3.0 > FileFormatWriter/BasicW

[jira] [Assigned] (SPARK-22271) Describe results in "null" for the value of "mean" of a numeric variable

2017-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22271: Assignee: Apache Spark > Describe results in "null" for the value of "mean" of a numeric v

[jira] [Commented] (SPARK-22271) Describe results in "null" for the value of "mean" of a numeric variable

2017-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16204487#comment-16204487 ] Apache Spark commented on SPARK-22271: -- User 'huaxingao' has created a pull request

[jira] [Assigned] (SPARK-22271) Describe results in "null" for the value of "mean" of a numeric variable

2017-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22271: Assignee: (was: Apache Spark) > Describe results in "null" for the value of "mean" of

[jira] [Commented] (SPARK-20883) Improve StateStore APIs for efficiency

2017-10-13 Thread Jayesh lalwani (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16204414#comment-16204414 ] Jayesh lalwani commented on SPARK-20883: Is this fix planned to go into 2.2.1? >

[jira] [Commented] (SPARK-22278) Expose current event time watermark and current processing time in GroupState

2017-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16204384#comment-16204384 ] Apache Spark commented on SPARK-22278: -- User 'tdas' has created a pull request for t

[jira] [Assigned] (SPARK-22278) Expose current event time watermark and current processing time in GroupState

2017-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22278: Assignee: Apache Spark (was: Tathagata Das) > Expose current event time watermark and cur

[jira] [Assigned] (SPARK-22278) Expose current event time watermark and current processing time in GroupState

2017-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22278: Assignee: Tathagata Das (was: Apache Spark) > Expose current event time watermark and cur

[jira] [Resolved] (SPARK-11034) Launcher: add support for monitoring Mesos apps

2017-10-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-11034. Resolution: Fixed Assignee: Devaraj K Fix Version/s: 2.3.0 > Launcher: add

[jira] [Created] (SPARK-22278) Expose current event time watermark and current processing time in GroupState

2017-10-13 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-22278: - Summary: Expose current event time watermark and current processing time in GroupState Key: SPARK-22278 URL: https://issues.apache.org/jira/browse/SPARK-22278 Proje

[jira] [Assigned] (SPARK-22187) Update unsaferow format for saved state such that we can set timeouts when state is null

2017-10-13 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-22187: - Assignee: Tathagata Das > Update unsaferow format for saved state such that we can set t

[jira] [Resolved] (SPARK-22262) Failed to recover Spark Structured Streaming job from checkpoint location

2017-10-13 Thread rohit verma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] rohit verma resolved SPARK-22262. - Resolution: Invalid Invalid.. > Failed to recover Spark Structured Streaming job from checkpoint

[jira] [Updated] (SPARK-22232) Row objects in pyspark created using the `Row(**kwars)` syntax do not get serialized/deserialized properly

2017-10-13 Thread Bago Amirbekian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bago Amirbekian updated SPARK-22232: Component/s: SQL > Row objects in pyspark created using the `Row(**kwars)` syntax do not ge

[jira] [Comment Edited] (SPARK-22271) Describe results in "null" for the value of "mean" of a numeric variable

2017-10-13 Thread Huaxin Gao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16204281#comment-16204281 ] Huaxin Gao edited comment on SPARK-22271 at 10/13/17 10:34 PM:

[jira] [Commented] (SPARK-22271) Describe results in "null" for the value of "mean" of a numeric variable

2017-10-13 Thread Huaxin Gao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16204281#comment-16204281 ] Huaxin Gao commented on SPARK-22271: I looked the code, in Average.scala, it has ```

[jira] [Created] (SPARK-22277) Chi Square selector garbling Vector content.

2017-10-13 Thread Cheburakshu (JIRA)
Cheburakshu created SPARK-22277: --- Summary: Chi Square selector garbling Vector content. Key: SPARK-22277 URL: https://issues.apache.org/jira/browse/SPARK-22277 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-22136) Implement stream-stream outer joins in append mode

2017-10-13 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-22136: - Assignee: Jose Torres > Implement stream-stream outer joins in append mode > ---

[jira] [Created] (SPARK-22276) Unnecessary repartitioning

2017-10-13 Thread Fernando Pereira (JIRA)
Fernando Pereira created SPARK-22276: Summary: Unnecessary repartitioning Key: SPARK-22276 URL: https://issues.apache.org/jira/browse/SPARK-22276 Project: Spark Issue Type: Bug

[jira] [Comment Edited] (SPARK-22250) Be less restrictive on type checking

2017-10-13 Thread Fernando Pereira (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16204059#comment-16204059 ] Fernando Pereira edited comment on SPARK-22250 at 10/13/17 7:28 PM: ---

[jira] [Commented] (SPARK-22250) Be less restrictive on type checking

2017-10-13 Thread Fernando Pereira (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16204059#comment-16204059 ] Fernando Pereira commented on SPARK-22250: -- I have to admit I was not aware of t

[jira] [Commented] (SPARK-22229) SPIP: RDMA Accelerated Shuffle Engine

2017-10-13 Thread Yuval Degani (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16204041#comment-16204041 ] Yuval Degani commented on SPARK-9: -- [~jerryshao], thanks for clearing your point

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-10-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16204019#comment-16204019 ] Hyukjin Kwon commented on SPARK-22240: -- Here, I could reproduce the issue in the JIR

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-10-13 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203966#comment-16203966 ] Steve Loughran commented on SPARK-22240: Point me at a simple test suite for the

[jira] [Commented] (SPARK-22269) Java style checks should be run in Jenkins

2017-10-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203902#comment-16203902 ] Sean Owen commented on SPARK-22269: --- Sure, maybe it's possible to run through SBT. That

[jira] [Updated] (SPARK-22275) SparkContext doesn't clean up after itself when "fatal" errors occur

2017-10-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-22275: --- Summary: SparkContext doesn't clean up after itself when "fatal" errors occur (was: ReplayLi

[jira] [Updated] (SPARK-22275) SparkContext doesn't clean up after itself when "fatal" errors occur

2017-10-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-22275: --- Component/s: Spark Core > SparkContext doesn't clean up after itself when "fatal" errors occu

[jira] [Commented] (SPARK-22275) ReplayListenerSuite doesn't clean up properly after itself

2017-10-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203887#comment-16203887 ] Marcelo Vanzin commented on SPARK-22275: Actually this is caused by the exception

[jira] [Commented] (SPARK-20144) spark.read.parquet no long maintains ordering of the data

2017-10-13 Thread sam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203882#comment-16203882 ] sam commented on SPARK-20144: - I think this is a regression. We used to be able to easily co

[jira] [Commented] (SPARK-17998) Reading Parquet files coalesces parts into too few in-memory partitions

2017-10-13 Thread sam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203877#comment-16203877 ] sam commented on SPARK-17998: - [~lwlin] I think this is a regression. We used to be able to

[jira] [Commented] (SPARK-22269) Java style checks should be run in Jenkins

2017-10-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203873#comment-16203873 ] Marcelo Vanzin commented on SPARK-22269: I'm saying that we should at least inves

[jira] [Commented] (SPARK-22269) Java style checks should be run in Jenkins

2017-10-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203865#comment-16203865 ] Sean Owen commented on SPARK-22269: --- You're saying we could have a separate Jenkins job

[jira] [Created] (SPARK-22275) ReplayListenerSuite doesn't clean up properly after itself

2017-10-13 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-22275: -- Summary: ReplayListenerSuite doesn't clean up properly after itself Key: SPARK-22275 URL: https://issues.apache.org/jira/browse/SPARK-22275 Project: Spark

[jira] [Assigned] (SPARK-21247) Type comparision should respect case-sensitive SQL conf

2017-10-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-21247: --- Assignee: Dongjoon Hyun > Type comparision should respect case-sensitive SQL conf >

[jira] [Resolved] (SPARK-21247) Type comparision should respect case-sensitive SQL conf

2017-10-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21247. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18460 [https://githu

[jira] [Reopened] (SPARK-22269) Java style checks should be run in Jenkins

2017-10-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reopened SPARK-22269: This is not a duplicate of that bug. If there's a bug about enabling lint-java in the PR build

[jira] [Updated] (SPARK-22181) ReplaceExceptWithFilter if one or both of the datasets are fully derived out of Filters from a same parent

2017-10-13 Thread Sathiya Kumar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sathiya Kumar updated SPARK-22181: -- Summary: ReplaceExceptWithFilter if one or both of the datasets are fully derived out of Filter

[jira] [Updated] (SPARK-22181) ReplaceExceptWithNotFilter if one or both of the datasets are fully derived out of Filters from a same parent

2017-10-13 Thread Sathiya Kumar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sathiya Kumar updated SPARK-22181: -- Description: While applying Except operator between two datasets, if one or both of the datase

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-10-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203771#comment-16203771 ] Hyukjin Kwon commented on SPARK-22240: -- Yea, this problem exists when {{multiLine}}.

[jira] [Assigned] (SPARK-22249) UnsupportedOperationException: empty.reduceLeft when caching a dataframe

2017-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22249: Assignee: Apache Spark > UnsupportedOperationException: empty.reduceLeft when caching a da

[jira] [Commented] (SPARK-22249) UnsupportedOperationException: empty.reduceLeft when caching a dataframe

2017-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203751#comment-16203751 ] Apache Spark commented on SPARK-22249: -- User 'mgaido91' has created a pull request f

[jira] [Assigned] (SPARK-22249) UnsupportedOperationException: empty.reduceLeft when caching a dataframe

2017-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22249: Assignee: (was: Apache Spark) > UnsupportedOperationException: empty.reduceLeft when c

[jira] [Commented] (SPARK-22247) Hive partition filter very slow

2017-10-13 Thread Patrick Duin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203716#comment-16203716 ] Patrick Duin commented on SPARK-22247: -- I think this is a related issue SPARK-20331,

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-10-13 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203709#comment-16203709 ] Steve Loughran commented on SPARK-22240: [~hyukjin.kwon]: we now see that on s3a,

[jira] [Assigned] (SPARK-18355) Spark SQL fails to read data from a ORC hive table that has a new column added to it

2017-10-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-18355: --- Assignee: Dongjoon Hyun > Spark SQL fails to read data from a ORC hive table that has a new

[jira] [Resolved] (SPARK-16628) OrcConversions should not convert an ORC table represented by MetastoreRelation to HadoopFsRelation if metastore schema does not match schema stored in ORC files

2017-10-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-16628. - Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 Issue resolved by pull req

[jira] [Assigned] (SPARK-16628) OrcConversions should not convert an ORC table represented by MetastoreRelation to HadoopFsRelation if metastore schema does not match schema stored in ORC files

2017-10-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-16628: --- Assignee: Dongjoon Hyun > OrcConversions should not convert an ORC table represented by > M

[jira] [Resolved] (SPARK-18355) Spark SQL fails to read data from a ORC hive table that has a new column added to it

2017-10-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-18355. - Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 Issue resolved by pull req

[jira] [Assigned] (SPARK-14387) Enable Hive-1.x ORC compatibility with spark.sql.hive.convertMetastoreOrc

2017-10-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-14387: --- Assignee: Dongjoon Hyun > Enable Hive-1.x ORC compatibility with spark.sql.hive.convertMetas

[jira] [Resolved] (SPARK-14387) Enable Hive-1.x ORC compatibility with spark.sql.hive.convertMetastoreOrc

2017-10-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-14387. - Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 Issue resolved by pull req

[jira] [Commented] (SPARK-22252) FileFormatWriter should respect the input query schema

2017-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203683#comment-16203683 ] Apache Spark commented on SPARK-22252: -- User 'cloud-fan' has created a pull request

[jira] [Updated] (SPARK-22274) Used-defined aggregation functions with pandas udf

2017-10-13 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Jin updated SPARK-22274: --- Description: This function doesn't implement partial aggregation and shuffles all data. A uadf that supports

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-10-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203600#comment-16203600 ] Hyukjin Kwon commented on SPARK-22240: -- I am sorry for late response. {quote} If i

[jira] [Created] (SPARK-22274) Used-defined aggregation functions with pandas udf

2017-10-13 Thread Li Jin (JIRA)
Li Jin created SPARK-22274: -- Summary: Used-defined aggregation functions with pandas udf Key: SPARK-22274 URL: https://issues.apache.org/jira/browse/SPARK-22274 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-22228) Add support for Array so from_json can parse

2017-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203506#comment-16203506 ] Apache Spark commented on SPARK-8: -- User 'mgaido91' has created a pull request f

[jira] [Assigned] (SPARK-22228) Add support for Array so from_json can parse

2017-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-8: Assignee: (was: Apache Spark) > Add support for Array so from_json can parse > ---

[jira] [Assigned] (SPARK-22228) Add support for Array so from_json can parse

2017-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-8: Assignee: Apache Spark > Add support for Array so from_json can parse > --

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-10-13 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203495#comment-16203495 ] Steve Loughran commented on SPARK-22240: We've got a test in HADOOP-14943 which l

[jira] [Commented] (SPARK-22242) streaming job failed to restart from checkpoint

2017-10-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203419#comment-16203419 ] Sean Owen commented on SPARK-22242: --- [~jerryshao] were you suggesting this is already r

[jira] [Created] (SPARK-22273) Fix key/value schema field names in HashMapGenerators.

2017-10-13 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-22273: - Summary: Fix key/value schema field names in HashMapGenerators. Key: SPARK-22273 URL: https://issues.apache.org/jira/browse/SPARK-22273 Project: Spark Issu

[jira] [Commented] (SPARK-22273) Fix key/value schema field names in HashMapGenerators.

2017-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203277#comment-16203277 ] Apache Spark commented on SPARK-22273: -- User 'ueshin' has created a pull request for

[jira] [Assigned] (SPARK-22273) Fix key/value schema field names in HashMapGenerators.

2017-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22273: Assignee: Apache Spark > Fix key/value schema field names in HashMapGenerators. >

[jira] [Assigned] (SPARK-22273) Fix key/value schema field names in HashMapGenerators.

2017-10-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22273: Assignee: (was: Apache Spark) > Fix key/value schema field names in HashMapGenerators.

[jira] [Commented] (SPARK-22272) killing task may cause the executor progress hang because of the JVM bug

2017-10-13 Thread roncenzhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203264#comment-16203264 ] roncenzhao commented on SPARK-22272: The jdk bug is not resolved yet. If spark does n

[jira] [Commented] (SPARK-22272) killing task may cause the executor progress hang because of the JVM bug

2017-10-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203199#comment-16203199 ] Sean Owen commented on SPARK-22272: --- This is long since fixed in the JDK though. Does s

[jira] [Updated] (SPARK-22272) killing task may cause the executor progress hang because of the JVM bug

2017-10-13 Thread roncenzhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao updated SPARK-22272: --- Description: JVM bug: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8132693 We kill the task u

[jira] [Updated] (SPARK-22272) killing task may cause the executor progress hang because of the JVM bug

2017-10-13 Thread roncenzhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao updated SPARK-22272: --- Attachment: 26883.jstack > killing task may cause the executor progress hang because of the JVM bug >

[jira] [Updated] (SPARK-22272) killing task may cause the executor progress hang because of the JVM bug

2017-10-13 Thread roncenzhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao updated SPARK-22272: --- Attachment: screenshot-1.png > killing task may cause the executor progress hang because of the JVM b

[jira] [Updated] (SPARK-22272) killing task may cause the executor progress hang because of the JVM bug

2017-10-13 Thread roncenzhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao updated SPARK-22272: --- Attachment: screenshot-2.png > killing task may cause the executor progress hang because of the JVM b

[jira] [Created] (SPARK-22272) killing task may cause the executor progress hang because of the JVM bug

2017-10-13 Thread roncenzhao (JIRA)
roncenzhao created SPARK-22272: -- Summary: killing task may cause the executor progress hang because of the JVM bug Key: SPARK-22272 URL: https://issues.apache.org/jira/browse/SPARK-22272 Project: Spark

[jira] [Commented] (SPARK-22269) Java style checks should be run in Jenkins

2017-10-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203161#comment-16203161 ] Sean Owen commented on SPARK-22269: --- https://issues.apache.org/jira/browse/SPARK-22066