[jira] [Updated] (SPARK-15214) Implement code generation for Generate

[ https://issues.apache.org/jira/browse/SPARK-15214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-15214: Summary: Implement code generation for Generate (was: Enable code generation for Generate) > Impl

[jira] [Resolved] (SPARK-15214) Enable code generation for Generate

[ https://issues.apache.org/jira/browse/SPARK-15214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-15214. - Resolution: Fixed Fix Version/s: 2.2.0 Target Version/s: (was: 2.1.0) > Enable

[jira] [Resolved] (SPARK-18508) Fix documentation for DateDiff

[ https://issues.apache.org/jira/browse/SPARK-18508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18508. - Resolution: Fixed Fix Version/s: 2.1.0 > Fix documentation for DateDiff >

[jira] [Resolved] (SPARK-18458) core dumped running Spark SQL on large data volume (100TB)

[ https://issues.apache.org/jira/browse/SPARK-18458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18458. - Resolution: Fixed Assignee: Kazuaki Ishizaki Fix Version/s: 2.1.0 > core dumped r

[jira] [Commented] (SPARK-18510) Partition schema inference corrupts data

[ https://issues.apache.org/jira/browse/SPARK-18510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15680487#comment-15680487 ] Reynold Xin commented on SPARK-18510: - Does your pull request for SPARK-18407 fix thi

[jira] [Commented] (SPARK-18511) Add an api for join operation with just the column name and join type.

[ https://issues.apache.org/jira/browse/SPARK-18511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15680374#comment-15680374 ] Apache Spark commented on SPARK-18511: -- User 'shiv4nsh' has created a pull request f

[jira] [Assigned] (SPARK-18511) Add an api for join operation with just the column name and join type.

[ https://issues.apache.org/jira/browse/SPARK-18511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18511: Assignee: (was: Apache Spark) > Add an api for join operation with just the column nam

[jira] [Assigned] (SPARK-18511) Add an api for join operation with just the column name and join type.

[ https://issues.apache.org/jira/browse/SPARK-18511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18511: Assignee: Apache Spark > Add an api for join operation with just the column name and join

[jira] [Created] (SPARK-18511) Add an api for join operation with just the column name and join type.

Shivansh created SPARK-18511: Summary: Add an api for join operation with just the column name and join type. Key: SPARK-18511 URL: https://issues.apache.org/jira/browse/SPARK-18511 Project: Spark

[jira] [Assigned] (SPARK-18407) Inferred partition columns cause assertion error

[ https://issues.apache.org/jira/browse/SPARK-18407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18407: Assignee: Apache Spark > Inferred partition columns cause assertion error > --

[jira] [Assigned] (SPARK-18407) Inferred partition columns cause assertion error

[ https://issues.apache.org/jira/browse/SPARK-18407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18407: Assignee: (was: Apache Spark) > Inferred partition columns cause assertion error > ---

[jira] [Commented] (SPARK-18407) Inferred partition columns cause assertion error

[ https://issues.apache.org/jira/browse/SPARK-18407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15680352#comment-15680352 ] Apache Spark commented on SPARK-18407: -- User 'brkyvz' has created a pull request for

[jira] [Commented] (SPARK-18510) Partition schema inference corrupts data

[ https://issues.apache.org/jira/browse/SPARK-18510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15680328#comment-15680328 ] Burak Yavuz commented on SPARK-18510: - cc [~r...@databricks.com] I marked this as a b

[jira] [Updated] (SPARK-18510) Partition schema inference corrupts data

[ https://issues.apache.org/jira/browse/SPARK-18510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-18510: Description: Not sure if this is a regression from 2.0 to 2.1. I was investigating this for Struct

[jira] [Created] (SPARK-18510) Partition schema inference corrupts data

Burak Yavuz created SPARK-18510: --- Summary: Partition schema inference corrupts data Key: SPARK-18510 URL: https://issues.apache.org/jira/browse/SPARK-18510 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-17380) Spark streaming with a multi shard Kinesis freezes after several days (memory/resource leak?)

[ https://issues.apache.org/jira/browse/SPARK-17380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15680276#comment-15680276 ] Udit Mehrotra commented on SPARK-17380: --- The above leak was seen with Spark 2.0 run

[jira] [Commented] (SPARK-17380) Spark streaming with a multi shard Kinesis freezes after several days (memory/resource leak?)

[ https://issues.apache.org/jira/browse/SPARK-17380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15680270#comment-15680270 ] Udit Mehrotra commented on SPARK-17380: --- We came across this Memory Leak in the exe

[jira] [Resolved] (SPARK-17062) Add --conf to mesos dispatcher process

[ https://issues.apache.org/jira/browse/SPARK-17062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-17062. Resolution: Fixed Assignee: Stavros Kontopoulos Fix Version/s: 2.2.0 > Add

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15680080#comment-15680080 ] Saikat Kanjilal commented on SPARK-9487: Ok guess I spoke too soon :), onto the ne

[jira] [Comment Edited] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15680080#comment-15680080 ] Saikat Kanjilal edited comment on SPARK-9487 at 11/19/16 11:59 PM: -

[jira] [Commented] (SPARK-18447) Fix `Note:`/`NOTE:`/`Note that` across Python API documentation

[ https://issues.apache.org/jira/browse/SPARK-18447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15679902#comment-15679902 ] Apache Spark commented on SPARK-18447: -- User 'aditya1702' has created a pull request

[jira] [Assigned] (SPARK-18447) Fix `Note:`/`NOTE:`/`Note that` across Python API documentation

[ https://issues.apache.org/jira/browse/SPARK-18447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18447: Assignee: (was: Apache Spark) > Fix `Note:`/`NOTE:`/`Note that` across Python API docu

[jira] [Assigned] (SPARK-18447) Fix `Note:`/`NOTE:`/`Note that` across Python API documentation

[ https://issues.apache.org/jira/browse/SPARK-18447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18447: Assignee: Apache Spark > Fix `Note:`/`NOTE:`/`Note that` across Python API documentation >

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15679880#comment-15679880 ] Saikat Kanjilal commented on SPARK-9487: Ok fixed the unit test, didnt have to res

[jira] [Commented] (SPARK-18447) Fix `Note:`/`NOTE:`/`Note that` across Python API documentation

[ https://issues.apache.org/jira/browse/SPARK-18447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15679826#comment-15679826 ] Aditya commented on SPARK-18447: Yes, I will work on this. I had some college lab exams h

[jira] [Commented] (SPARK-18506) kafka 0.10 with Spark 2.02 auto.offset.reset=earliest will only read from a single partition on a multi partition topic

[ https://issues.apache.org/jira/browse/SPARK-18506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15679667#comment-15679667 ] Heji Kim commented on SPARK-18506: -- We have spent two weeks trying different configurati

[jira] [Updated] (SPARK-18506) kafka 0.10 with Spark 2.02 auto.offset.reset=earliest will only read from a single partition on a multi partition topic

[ https://issues.apache.org/jira/browse/SPARK-18506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heji Kim updated SPARK-18506: - Description: Our team is trying to upgrade to Spark 2.0.2/Kafka 0.10.1.0/spark-streaming-kafka-0-10_2.11

[jira] [Updated] (SPARK-18506) kafka 0.10 with Spark 2.02 auto.offset.reset=earliest will only read from a single partition on a multi partition topic

[ https://issues.apache.org/jira/browse/SPARK-18506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heji Kim updated SPARK-18506: - Description: Our team is trying to upgrade to Spark 2.0.2/Kafka 0.10.1.0/spark-streaming-kafka-0-10_2.11

[jira] [Commented] (SPARK-3359) `sbt/sbt unidoc` doesn't work with Java 8

[ https://issues.apache.org/jira/browse/SPARK-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15679611#comment-15679611 ] Apache Spark commented on SPARK-3359: - User 'HyukjinKwon' has created a pull request f

[jira] [Updated] (SPARK-18073) Migrate wiki to spark.apache.org web site

[ https://issues.apache.org/jira/browse/SPARK-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-18073: -- Description: Per http://apache-spark-developers-list.1001551.n3.nabble.com/Mini-Proposal-Make-it-easie

[jira] [Updated] (SPARK-18073) Migrate wiki to spark.apache.org web site

[ https://issues.apache.org/jira/browse/SPARK-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-18073: -- Assignee: Sean Owen Priority: Critical (was: Major) WIP: https://github.com/apache/spark-website/p

[jira] [Comment Edited] (SPARK-18475) Be able to provide higher parallelization for StructuredStreaming Kafka Source

[ https://issues.apache.org/jira/browse/SPARK-18475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15679472#comment-15679472 ] Cody Koeninger edited comment on SPARK-18475 at 11/19/16 4:02 PM: -

[jira] [Commented] (SPARK-18475) Be able to provide higher parallelization for StructuredStreaming Kafka Source

[ https://issues.apache.org/jira/browse/SPARK-18475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15679472#comment-15679472 ] Cody Koeninger commented on SPARK-18475: Yes, an RDD does have an ordering guaran

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15679303#comment-15679303 ] Sean Owen commented on SPARK-17436: --- Opening a pull request is available to everyone. M

[jira] [Resolved] (SPARK-18509) spark-ec2 init.sh requests .tgz files not available at http://s3.amazonaws.com/spark-related-packages

[ https://issues.apache.org/jira/browse/SPARK-18509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-18509. --- Resolution: Not A Problem Yes, best to post the question to amplab/spark-ec2 > spark-ec2 init.sh req

[jira] [Reopened] (SPARK-18509) spark-ec2 init.sh requests .tgz files not available at http://s3.amazonaws.com/spark-related-packages

[ https://issues.apache.org/jira/browse/SPARK-18509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reopened SPARK-18509: --- > spark-ec2 init.sh requests .tgz files not available at > http://s3.amazonaws.com/spark-related-package

[jira] [Closed] (SPARK-18509) spark-ec2 init.sh requests .tgz files not available at http://s3.amazonaws.com/spark-related-packages

[ https://issues.apache.org/jira/browse/SPARK-18509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter B. Pearman closed SPARK-18509. Resolution: Fixed Not an Apache project bucket > spark-ec2 init.sh requests .tgz files not

[jira] [Commented] (SPARK-18509) spark-ec2 init.sh requests .tgz files not available at http://s3.amazonaws.com/spark-related-packages

[ https://issues.apache.org/jira/browse/SPARK-18509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15679219#comment-15679219 ] Peter B. Pearman commented on SPARK-18509: -- So, I guess I should post an issue t

[jira] [Updated] (SPARK-18509) spark-ec2 init.sh requests .tgz files not available at http://s3.amazonaws.com/spark-related-packages

[ https://issues.apache.org/jira/browse/SPARK-18509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter B. Pearman updated SPARK-18509: - Environment: AWS EC2, AWS Linux, OS X 10.12.x (local) (was: AWS EC2, Amazon Linux, OS X

[jira] [Comment Edited] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15679204#comment-15679204 ] Ran Haim edited comment on SPARK-17436 at 11/19/16 12:44 PM: -

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15679204#comment-15679204 ] Ran Haim commented on SPARK-17436: -- Hi, When you want to write your data to orc files or

[jira] [Commented] (SPARK-18509) spark-ec2 init.sh requests .tgz files not available at http://s3.amazonaws.com/spark-related-packages

[ https://issues.apache.org/jira/browse/SPARK-18509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15679205#comment-15679205 ] Sean Owen commented on SPARK-18509: --- This script is not part of Spark, and the project

[jira] [Commented] (SPARK-2984) FileNotFoundException on _temporary directory

[ https://issues.apache.org/jira/browse/SPARK-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15679196#comment-15679196 ] Steve Loughran commented on SPARK-2984: --- That sounds like a separate issue...could y

[jira] [Created] (SPARK-18509) spark-ec2 init.sh requests .tgz files not available at http://s3.amazonaws.com/spark-related-packages

Peter B. Pearman created SPARK-18509: Summary: spark-ec2 init.sh requests .tgz files not available at http://s3.amazonaws.com/spark-related-packages Key: SPARK-18509 URL: https://issues.apache.org/jira/browse/

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15679106#comment-15679106 ] Sean Owen commented on SPARK-17436: --- [~ran.h...@optimalplus.com] I'm not sure how to pr

[jira] [Updated] (SPARK-18353) spark.rpc.askTimeout defalut value is not 120s

[ https://issues.apache.org/jira/browse/SPARK-18353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-18353: -- Priority: Minor (was: Critical) > spark.rpc.askTimeout defalut value is not 120s > ---

[jira] [Resolved] (SPARK-18353) spark.rpc.askTimeout defalut value is not 120s

[ https://issues.apache.org/jira/browse/SPARK-18353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-18353. --- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 15833 [https://github.co

[jira] [Assigned] (SPARK-18353) spark.rpc.askTimeout defalut value is not 120s

[ https://issues.apache.org/jira/browse/SPARK-18353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-18353: - Assignee: Sean Owen > spark.rpc.askTimeout defalut value is not 120s > -

[jira] [Commented] (SPARK-18447) Fix `Note:`/`NOTE:`/`Note that` across Python API documentation

[ https://issues.apache.org/jira/browse/SPARK-18447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15679092#comment-15679092 ] Sean Owen commented on SPARK-18447: --- [~aditya1702] if you're 'claiming' this -- are you

[jira] [Resolved] (SPARK-18445) Fix `Note:`/`NOTE:`/`Note that` across Scala/Java API documentation

[ https://issues.apache.org/jira/browse/SPARK-18445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-18445. --- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 15889 [https://github.co

[jira] [Updated] (SPARK-18445) Fix `Note:`/`NOTE:`/`Note that` across Scala/Java API documentation

[ https://issues.apache.org/jira/browse/SPARK-18445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-18445: -- Assignee: Hyukjin Kwon > Fix `Note:`/`NOTE:`/`Note that` across Scala/Java API documentation >

[jira] [Commented] (SPARK-18448) SparkSession should implement java.lang.AutoCloseable like JavaSparkContext

[ https://issues.apache.org/jira/browse/SPARK-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15679084#comment-15679084 ] Apache Spark commented on SPARK-18448: -- User 'srowen' has created a pull request for

[jira] [Commented] (SPARK-18503) Pre 2.0 spark driver/executor memory default unit is bytes, post 2.0 default unit is MB

[ https://issues.apache.org/jira/browse/SPARK-18503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15679064#comment-15679064 ] Sean Owen commented on SPARK-18503: --- Yes, I don't think that's a problem per se. The va

[jira] [Updated] (SPARK-18485) Underlying integer overflow when create ChunkedByteBufferOutputStream in MemoryStore

[ https://issues.apache.org/jira/browse/SPARK-18485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-18485: -- Fix Version/s: (was: 1.6.4) (was: 2.0.3) (was: 2.1.0)

[jira] [Updated] (SPARK-18474) Add StreamingQuery.status in python

[ https://issues.apache.org/jira/browse/SPARK-18474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-18474: -- Fix Version/s: (was: 2.1.0) > Add StreamingQuery.status in python > ---

[jira] [Updated] (SPARK-7712) Window Function Improvements

[ https://issues.apache.org/jira/browse/SPARK-7712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-7712: - Assignee: Herman van Hovell > Window Function Improvements > > >

[jira] [Reopened] (SPARK-14171) UDAF aggregates argument object inspector not parsed correctly

[ https://issues.apache.org/jira/browse/SPARK-14171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reopened SPARK-14171: --- > UDAF aggregates argument object inspector not parsed correctly > --

[jira] [Resolved] (SPARK-14171) UDAF aggregates argument object inspector not parsed correctly

[ https://issues.apache.org/jira/browse/SPARK-14171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-14171. --- Resolution: Duplicate Fix Version/s: (was: 2.1.0) > UDAF aggregates argument object inspec

[jira] [Updated] (SPARK-18172) AnalysisException in first/last during aggregation

[ https://issues.apache.org/jira/browse/SPARK-18172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-18172: -- Assignee: Song Jun > AnalysisException in first/last during aggregation > -

[jira] [Assigned] (SPARK-18448) SparkSession should implement java.lang.AutoCloseable like JavaSparkContext

[ https://issues.apache.org/jira/browse/SPARK-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-18448: - Assignee: Sean Owen > SparkSession should implement java.lang.AutoCloseable like JavaSparkContex

[jira] [Updated] (SPARK-18464) Spark SQL fails to load tables created without providing a schema

[ https://issues.apache.org/jira/browse/SPARK-18464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-18464: -- Assignee: Wenchen Fan > Spark SQL fails to load tables created without providing a schema > ---

[jira] [Updated] (SPARK-18469) Cannot make MLlib model predictions in Spark streaming with checkpointing

[ https://issues.apache.org/jira/browse/SPARK-18469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-18469: -- Target Version/s: (was: 1.6.2) > Cannot make MLlib model predictions in Spark streaming with checkpoi

[jira] [Resolved] (SPARK-18448) SparkSession should implement java.lang.AutoCloseable like JavaSparkContext

[ https://issues.apache.org/jira/browse/SPARK-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-18448. --- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 15932 [https://github.co