[jira] [Commented] (SPARK-21489) Update release docs to point out Python 2.6 support is removed.

[ https://issues.apache.org/jira/browse/SPARK-21489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095873#comment-16095873 ] Holden Karau commented on SPARK-21489: -- Ah yes it was, was updating the docs from a

[jira] [Commented] (SPARK-21498) quick start -> one py demo have some bug in code

[ https://issues.apache.org/jira/browse/SPARK-21498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095877#comment-16095877 ] LiZhaochuan commented on SPARK-21498: - ha, ,i continue my leaning --

[jira] [Commented] (SPARK-21486) Fail when using aliased column of a aliased table from a subquery

[ https://issues.apache.org/jira/browse/SPARK-21486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095903#comment-16095903 ] Miguel Angel Fernandez Diaz commented on SPARK-21486: - Fair enough. T

[jira] [Created] (SPARK-21499) Support creating persistent function for Spark UDAF(UserDefinedAggregateFunction)

Xiao Li created SPARK-21499: --- Summary: Support creating persistent function for Spark UDAF(UserDefinedAggregateFunction) Key: SPARK-21499 URL: https://issues.apache.org/jira/browse/SPARK-21499 Project: Spar

[jira] [Assigned] (SPARK-21499) Support creating persistent function for Spark UDAF(UserDefinedAggregateFunction)

[ https://issues.apache.org/jira/browse/SPARK-21499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21499: Assignee: Xiao Li (was: Apache Spark) > Support creating persistent function for Spark >

[jira] [Commented] (SPARK-21499) Support creating persistent function for Spark UDAF(UserDefinedAggregateFunction)

[ https://issues.apache.org/jira/browse/SPARK-21499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095931#comment-16095931 ] Apache Spark commented on SPARK-21499: -- User 'gatorsmile' has created a pull request

[jira] [Assigned] (SPARK-21499) Support creating persistent function for Spark UDAF(UserDefinedAggregateFunction)

[ https://issues.apache.org/jira/browse/SPARK-21499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21499: Assignee: Apache Spark (was: Xiao Li) > Support creating persistent function for Spark >

[jira] [Commented] (SPARK-21472) Introduce ArrowColumnVector as a reader for Arrow vectors.

[ https://issues.apache.org/jira/browse/SPARK-21472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095971#comment-16095971 ] Apache Spark commented on SPARK-21472: -- User 'ueshin' has created a pull request for

[jira] [Commented] (SPARK-18226) SparkR displaying vector columns in incorrect way

[ https://issues.apache.org/jira/browse/SPARK-18226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096007#comment-16096007 ] Kirti commented on SPARK-18226: --- This code retrieves values of probability column: fileP <-

[jira] [Assigned] (SPARK-21485) API Documentation for Spark SQL functions

[ https://issues.apache.org/jira/browse/SPARK-21485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21485: Assignee: (was: Apache Spark) > API Documentation for Spark SQL functions > --

[jira] [Commented] (SPARK-21485) API Documentation for Spark SQL functions

[ https://issues.apache.org/jira/browse/SPARK-21485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096011#comment-16096011 ] Apache Spark commented on SPARK-21485: -- User 'HyukjinKwon' has created a pull reques

[jira] [Assigned] (SPARK-21485) API Documentation for Spark SQL functions

[ https://issues.apache.org/jira/browse/SPARK-21485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21485: Assignee: Apache Spark > API Documentation for Spark SQL functions > -

[jira] [Commented] (SPARK-21491) Performance enhancement: eliminate creation of intermediate collections

[ https://issues.apache.org/jira/browse/SPARK-21491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096055#comment-16096055 ] Sean Owen commented on SPARK-21491: --- I searched for other instances of map followed by

[jira] [Commented] (SPARK-21491) Performance enhancement: eliminate creation of intermediate collections

[ https://issues.apache.org/jira/browse/SPARK-21491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096079#comment-16096079 ] Hyukjin Kwon commented on SPARK-21491: -- For me, I won't (don't) fix these but am fin

[jira] [Comment Edited] (SPARK-21491) Performance enhancement: eliminate creation of intermediate collections

[ https://issues.apache.org/jira/browse/SPARK-21491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096079#comment-16096079 ] Hyukjin Kwon edited comment on SPARK-21491 at 7/21/17 10:01 AM: ---

[jira] [Commented] (SPARK-19810) Remove support for Scala 2.10

[ https://issues.apache.org/jira/browse/SPARK-19810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096091#comment-16096091 ] Apache Spark commented on SPARK-19810: -- User 'srowen' has created a pull request for

[jira] [Commented] (SPARK-21402) Java encoders - switch fields on collectAsList

[ https://issues.apache.org/jira/browse/SPARK-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096162#comment-16096162 ] Tom commented on SPARK-21402: - I have a small insight regarding this possible issue as I bump

[jira] [Commented] (SPARK-20783) Enhance ColumnVector to support compressed representation

[ https://issues.apache.org/jira/browse/SPARK-20783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096306#comment-16096306 ] Apache Spark commented on SPARK-20783: -- User 'kiszk' has created a pull request for

[jira] [Updated] (SPARK-21488) Make saveAsTable() and createOrReplaceTempView() return dataframe of created table/ created view

[ https://issues.apache.org/jira/browse/SPARK-21488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruslan Dautkhanov updated SPARK-21488: -- Target Version/s: 3.0.0 > Make saveAsTable() and createOrReplaceTempView() return dataf

[jira] [Commented] (SPARK-21488) Make saveAsTable() and createOrReplaceTempView() return dataframe of created table/ created view

[ https://issues.apache.org/jira/browse/SPARK-21488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096354#comment-16096354 ] Ruslan Dautkhanov commented on SPARK-21488: --- Makes sense [~zsxwing], thank you.

[jira] [Created] (SPARK-21500) Update Shuffle Fetch bookkeeping immediately after receiving remote block

Dhruve Ashar created SPARK-21500: Summary: Update Shuffle Fetch bookkeeping immediately after receiving remote block Key: SPARK-21500 URL: https://issues.apache.org/jira/browse/SPARK-21500 Project: Sp

[jira] [Commented] (SPARK-20389) Upgrade kryo to fix NegativeArraySizeException

[ https://issues.apache.org/jira/browse/SPARK-20389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096431#comment-16096431 ] Louis Bergelson commented on SPARK-20389: - [~srowen] What would I have to do to

[jira] [Created] (SPARK-21501) Spark shuffle index cache size should be memory based

Thomas Graves created SPARK-21501: - Summary: Spark shuffle index cache size should be memory based Key: SPARK-21501 URL: https://issues.apache.org/jira/browse/SPARK-21501 Project: Spark Issue

[jira] [Created] (SPARK-21502) --supervise causing frameworkId conflicts in mesos cluster mode

Stavros Kontopoulos created SPARK-21502: --- Summary: --supervise causing frameworkId conflicts in mesos cluster mode Key: SPARK-21502 URL: https://issues.apache.org/jira/browse/SPARK-21502 Project

[jira] [Commented] (SPARK-21502) --supervise causing frameworkId conflicts in mesos cluster mode

[ https://issues.apache.org/jira/browse/SPARK-21502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096450#comment-16096450 ] Stavros Kontopoulos commented on SPARK-21502: - I have a patch I will submit a

[jira] [Updated] (SPARK-21502) --supervise causing frameworkId conflicts in mesos cluster mode

[ https://issues.apache.org/jira/browse/SPARK-21502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stavros Kontopoulos updated SPARK-21502: Description: This issue is described here: https://dcosjira.atlassian.net/browse/SP

[jira] [Commented] (SPARK-21494) Spark 2.2.0 AES encryption not working with External shuffle

[ https://issues.apache.org/jira/browse/SPARK-21494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096453#comment-16096453 ] Marcelo Vanzin commented on SPARK-21494: [~srowen] I already asked all these ques

[jira] [Assigned] (SPARK-21502) --supervise causing frameworkId conflicts in mesos cluster mode

[ https://issues.apache.org/jira/browse/SPARK-21502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21502: Assignee: Apache Spark > --supervise causing frameworkId conflicts in mesos cluster mode >

[jira] [Commented] (SPARK-21502) --supervise causing frameworkId conflicts in mesos cluster mode

[ https://issues.apache.org/jira/browse/SPARK-21502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096556#comment-16096556 ] Apache Spark commented on SPARK-21502: -- User 'skonto' has created a pull request for

[jira] [Assigned] (SPARK-21502) --supervise causing frameworkId conflicts in mesos cluster mode

[ https://issues.apache.org/jira/browse/SPARK-21502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21502: Assignee: (was: Apache Spark) > --supervise causing frameworkId conflicts in mesos clu

[jira] [Assigned] (SPARK-21494) Spark 2.2.0 AES encryption not working with External shuffle

[ https://issues.apache.org/jira/browse/SPARK-21494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21494: Assignee: (was: Apache Spark) > Spark 2.2.0 AES encryption not working with External s

[jira] [Commented] (SPARK-21494) Spark 2.2.0 AES encryption not working with External shuffle

[ https://issues.apache.org/jira/browse/SPARK-21494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096615#comment-16096615 ] Apache Spark commented on SPARK-21494: -- User 'vanzin' has created a pull request for

[jira] [Updated] (SPARK-21498) quick start -> one py demo have some bug in code

[ https://issues.apache.org/jira/browse/SPARK-21498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21498: -- Target Version/s: (was: 2.2.0) Fix Version/s: (was: 2.2.0) Do you want to make a pull requ

[jira] [Assigned] (SPARK-21494) Spark 2.2.0 AES encryption not working with External shuffle

[ https://issues.apache.org/jira/browse/SPARK-21494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21494: Assignee: Apache Spark > Spark 2.2.0 AES encryption not working with External shuffle > --

[jira] [Updated] (SPARK-21243) Limit the number of maps in a single shuffle fetch

[ https://issues.apache.org/jira/browse/SPARK-21243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21243: -- Fix Version/s: 2.2.1 > Limit the number of maps in a single shuffle fetch > ---

[jira] [Created] (SPARK-21503) Spark UI shows incorrect task status for a killed Executor Process

Parth Gandhi created SPARK-21503: Summary: Spark UI shows incorrect task status for a killed Executor Process Key: SPARK-21503 URL: https://issues.apache.org/jira/browse/SPARK-21503 Project: Spark

[jira] [Commented] (SPARK-21503) Spark UI shows incorrect task status for a killed Executor Process

[ https://issues.apache.org/jira/browse/SPARK-21503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096846#comment-16096846 ] Parth Gandhi commented on SPARK-21503: -- Currently working on the fix, will file a pu

[jira] [Commented] (SPARK-20389) Upgrade kryo to fix NegativeArraySizeException

[ https://issues.apache.org/jira/browse/SPARK-20389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096889#comment-16096889 ] Sean Owen commented on SPARK-20389: --- I think the question is whether it would cause any

[jira] [Assigned] (SPARK-21503) Spark UI shows incorrect task status for a killed Executor Process

[ https://issues.apache.org/jira/browse/SPARK-21503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21503: Assignee: Apache Spark > Spark UI shows incorrect task status for a killed Executor Proces

[jira] [Commented] (SPARK-21503) Spark UI shows incorrect task status for a killed Executor Process

[ https://issues.apache.org/jira/browse/SPARK-21503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096896#comment-16096896 ] Apache Spark commented on SPARK-21503: -- User 'pgandhi999' has created a pull request

[jira] [Assigned] (SPARK-21503) Spark UI shows incorrect task status for a killed Executor Process

[ https://issues.apache.org/jira/browse/SPARK-21503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21503: Assignee: (was: Apache Spark) > Spark UI shows incorrect task status for a killed Exec

[jira] [Assigned] (SPARK-21339) spark-shell --packages option does not add jars to classpath on windows

[ https://issues.apache.org/jira/browse/SPARK-21339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21339: Assignee: Apache Spark > spark-shell --packages option does not add jars to classpath on w

[jira] [Assigned] (SPARK-21339) spark-shell --packages option does not add jars to classpath on windows

[ https://issues.apache.org/jira/browse/SPARK-21339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21339: Assignee: (was: Apache Spark) > spark-shell --packages option does not add jars to cla

[jira] [Commented] (SPARK-21339) spark-shell --packages option does not add jars to classpath on windows

[ https://issues.apache.org/jira/browse/SPARK-21339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096925#comment-16096925 ] Apache Spark commented on SPARK-21339: -- User 'devaraj-kavali' has created a pull req

[jira] [Created] (SPARK-21504) Add spark version info in table metadata

Xiao Li created SPARK-21504: --- Summary: Add spark version info in table metadata Key: SPARK-21504 URL: https://issues.apache.org/jira/browse/SPARK-21504 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-21504) Add spark version info in table metadata

[ https://issues.apache.org/jira/browse/SPARK-21504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21504: Assignee: Apache Spark (was: Xiao Li) > Add spark version info in table metadata > --

[jira] [Assigned] (SPARK-21504) Add spark version info in table metadata

[ https://issues.apache.org/jira/browse/SPARK-21504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21504: Assignee: Xiao Li (was: Apache Spark) > Add spark version info in table metadata > --

[jira] [Commented] (SPARK-21504) Add spark version info in table metadata

[ https://issues.apache.org/jira/browse/SPARK-21504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096948#comment-16096948 ] Apache Spark commented on SPARK-21504: -- User 'gatorsmile' has created a pull request

[jira] [Commented] (SPARK-21498) quick start -> one py demo have some bug in code

[ https://issues.apache.org/jira/browse/SPARK-21498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096965#comment-16096965 ] LiZhaochuan commented on SPARK-21498: - yes,that's very good. how should i to do? ---

[jira] [Resolved] (SPARK-21489) Update release docs to point out Python 2.6 support is removed.

[ https://issues.apache.org/jira/browse/SPARK-21489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-21489. - Resolution: Resolved Assignee: Hyukjin Kwon Fix Version/s: 2.2.1 Fixed in https://github

[jira] [Assigned] (SPARK-21434) Add PySpark pip documentation

[ https://issues.apache.org/jira/browse/SPARK-21434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk reassigned SPARK-21434: --- Assignee: holdenk > Add PySpark pip documentation > - > >

[jira] [Resolved] (SPARK-21434) Add PySpark pip documentation

[ https://issues.apache.org/jira/browse/SPARK-21434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-21434. - Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 Issue resolved by pull request 186

[jira] [Commented] (SPARK-21086) CrossValidator, TrainValidationSplit should preserve all models after fitting

[ https://issues.apache.org/jira/browse/SPARK-21086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16097062#comment-16097062 ] yuhao yang commented on SPARK-21086: sure, indices sounds fine. For the driver memor

[jira] [Commented] (SPARK-16599) java.util.NoSuchElementException: None.get at at org.apache.spark.storage.BlockInfoManager.releaseAllLocksForTask(BlockInfoManager.scala:343)

[ https://issues.apache.org/jira/browse/SPARK-16599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16097070#comment-16097070 ] Ryan Williams commented on SPARK-16599: --- I've been running into this lately, and I

[jira] [Created] (SPARK-21505) A dynamic join operator to improve the join reliability

Lin created SPARK-21505: --- Summary: A dynamic join operator to improve the join reliability Key: SPARK-21505 URL: https://issues.apache.org/jira/browse/SPARK-21505 Project: Spark Issue Type: New Feature