[jira] [Updated] (SPARK-18040) Improve R handling or messaging of JVM exception

2016-10-20 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-18040: - Description: Similar to SPARK-17838, there are a few cases where an exception can be thrown

[jira] [Updated] (SPARK-17254) Filter operator should have “stop if false” semantics for sorted data

2016-10-20 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-17254: Attachment: (was: stop-after-physical-plan.pdf) > Filter operator should have “stop if

[jira] [Updated] (SPARK-17254) Filter operator should have “stop if false” semantics for sorted data

2016-10-20 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-17254: Attachment: stop-after-physical-plan.pdf > Filter operator should have “stop if false”

[jira] [Created] (SPARK-18041) activedrivers section in http:sparkMasterurl/json is missing Main class information

2016-10-20 Thread sudheesh k s (JIRA)
sudheesh k s created SPARK-18041: Summary: activedrivers section in http:sparkMasterurl/json is missing Main class information Key: SPARK-18041 URL: https://issues.apache.org/jira/browse/SPARK-18041

[jira] [Created] (SPARK-18040) Improve R handling or messaging of JVM exception

2016-10-20 Thread Felix Cheung (JIRA)
Felix Cheung created SPARK-18040: Summary: Improve R handling or messaging of JVM exception Key: SPARK-18040 URL: https://issues.apache.org/jira/browse/SPARK-18040 Project: Spark Issue Type:

[jira] [Commented] (SPARK-17275) Flaky test: org.apache.spark.deploy.RPackageUtilsSuite.jars that don't exist are skipped and print warning

2016-10-20 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15594080#comment-15594080 ] Felix Cheung commented on SPARK-17275: -- is this still a problem? > Flaky test:

[jira] [Commented] (SPARK-17916) CSV data source treats empty string as null no matter what nullValue option is

2016-10-20 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15594046#comment-15594046 ] Felix Cheung commented on SPARK-17916: -- So here's what happen. First, R read.csv has clearly

[jira] [Resolved] (SPARK-18029) PruneFileSourcePartitions should not change the output of LogicalRelation

2016-10-20 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-18029. - Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 15569

[jira] [Comment Edited] (SPARK-18022) java.lang.NullPointerException instead of real exception when saving DF to MySQL

2016-10-20 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-18022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593935#comment-15593935 ] Maciej Bryński edited comment on SPARK-18022 at 10/21/16 4:24 AM: -- I

[jira] [Commented] (SPARK-18022) java.lang.NullPointerException instead of real exception when saving DF to MySQL

2016-10-20 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-18022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593935#comment-15593935 ] Maciej Bryński commented on SPARK-18022: I think the problem is in this PR.

[jira] [Created] (SPARK-18039) ReceiverTracker run dummyjob too fast cause receiver scheduling unbalaced

2016-10-20 Thread astralidea (JIRA)
astralidea created SPARK-18039: -- Summary: ReceiverTracker run dummyjob too fast cause receiver scheduling unbalaced Key: SPARK-18039 URL: https://issues.apache.org/jira/browse/SPARK-18039 Project: Spark

[jira] [Comment Edited] (SPARK-18022) java.lang.NullPointerException instead of real exception when saving DF to MySQL

2016-10-20 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-18022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593846#comment-15593846 ] Maciej Bryński edited comment on SPARK-18022 at 10/21/16 3:56 AM: -- Only

[jira] [Comment Edited] (SPARK-18022) java.lang.NullPointerException instead of real exception when saving DF to MySQL

2016-10-20 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-18022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593846#comment-15593846 ] Maciej Bryński edited comment on SPARK-18022 at 10/21/16 3:39 AM: -- Only

[jira] [Commented] (SPARK-18022) java.lang.NullPointerException instead of real exception when saving DF to MySQL

2016-10-20 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-18022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593846#comment-15593846 ] Maciej Bryński commented on SPARK-18022: Only improvement in error handling. >

[jira] [Comment Edited] (SPARK-15765) Make continuous Parquet writes consistent with non-continuous Parquet writes

2016-10-20 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593782#comment-15593782 ] Liwei Lin edited comment on SPARK-15765 at 10/21/16 3:07 AM: - I'm closing

[jira] [Commented] (SPARK-15765) Make continuous Parquet writes consistent with non-continuous Parquet writes

2016-10-20 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593782#comment-15593782 ] Liwei Lin commented on SPARK-15765: --- I'm closing this in favor of SPARK-18025 > Make continuous

[jira] [Closed] (SPARK-15765) Make continuous Parquet writes consistent with non-continuous Parquet writes

2016-10-20 Thread Liwei Lin(Inactive) (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liwei Lin(Inactive) closed SPARK-15765. --- Resolution: Duplicate > Make continuous Parquet writes consistent with

[jira] [Commented] (SPARK-16845) org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering" grows beyond 64 KB

2016-10-20 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593760#comment-15593760 ] Liwei Lin commented on SPARK-16845: --- Oh thanks for the feedback; it's helpful! The branch you're

[jira] [Commented] (SPARK-17829) Stable format for offset log

2016-10-20 Thread Tyson Condie (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593706#comment-15593706 ] Tyson Condie commented on SPARK-17829: -- Had a conversation with Michael about how to offset

[jira] [Commented] (SPARK-17891) SQL-based three column join loses first column

2016-10-20 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593685#comment-15593685 ] Yuming Wang commented on SPARK-17891: - *Workaround:* # Disable BroadcastHashJoin by setting

[jira] [Comment Edited] (SPARK-882) Have link for feedback/suggestions in docs

2016-10-20 Thread Deron Eriksson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593573#comment-15593573 ] Deron Eriksson edited comment on SPARK-882 at 10/21/16 1:37 AM: I don't see

[jira] [Commented] (SPARK-18038) Move output partitioning definition from UnaryNodeExec to its children

2016-10-20 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593583#comment-15593583 ] Reynold Xin commented on SPARK-18038: - It definitely does. > Move output partitioning definition

[jira] [Commented] (SPARK-882) Have link for feedback/suggestions in docs

2016-10-20 Thread Deron Eriksson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593573#comment-15593573 ] Deron Eriksson commented on SPARK-882: -- I don't see any activity, so mind if I take a crack at this

[jira] [Comment Edited] (SPARK-7146) Should ML sharedParams be a public API?

2016-10-20 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593539#comment-15593539 ] Joseph K. Bradley edited comment on SPARK-7146 at 10/21/16 12:53 AM: -

[jira] [Commented] (SPARK-7146) Should ML sharedParams be a public API?

2016-10-20 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593539#comment-15593539 ] Joseph K. Bradley commented on SPARK-7146: -- Update: We may need to make Java interfaces for

[jira] [Assigned] (SPARK-18030) Flaky test: org.apache.spark.sql.streaming.FileStreamSourceSuite

2016-10-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18030: Assignee: Apache Spark (was: Tathagata Das) > Flaky test:

[jira] [Commented] (SPARK-18030) Flaky test: org.apache.spark.sql.streaming.FileStreamSourceSuite

2016-10-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593493#comment-15593493 ] Apache Spark commented on SPARK-18030: -- User 'zsxwing' has created a pull request for this issue:

[jira] [Assigned] (SPARK-18030) Flaky test: org.apache.spark.sql.streaming.FileStreamSourceSuite

2016-10-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18030: Assignee: Tathagata Das (was: Apache Spark) > Flaky test:

[jira] [Assigned] (SPARK-17674) Warnings from SparkR tests being ignored without redirecting to errors

2016-10-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17674: Assignee: Apache Spark > Warnings from SparkR tests being ignored without redirecting to

[jira] [Commented] (SPARK-17674) Warnings from SparkR tests being ignored without redirecting to errors

2016-10-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593484#comment-15593484 ] Apache Spark commented on SPARK-17674: -- User 'felixcheung' has created a pull request for this

[jira] [Assigned] (SPARK-17674) Warnings from SparkR tests being ignored without redirecting to errors

2016-10-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17674: Assignee: (was: Apache Spark) > Warnings from SparkR tests being ignored without

[jira] [Assigned] (SPARK-18038) Move output partitioning definition from UnaryNodeExec to its children

2016-10-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18038: Assignee: (was: Apache Spark) > Move output partitioning definition from

[jira] [Commented] (SPARK-18038) Move output partitioning definition from UnaryNodeExec to its children

2016-10-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593479#comment-15593479 ] Apache Spark commented on SPARK-18038: -- User 'tejasapatil' has created a pull request for this

[jira] [Assigned] (SPARK-18038) Move output partitioning definition from UnaryNodeExec to its children

2016-10-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18038: Assignee: Apache Spark > Move output partitioning definition from UnaryNodeExec to its

[jira] [Commented] (SPARK-18038) Move output partitioning definition from UnaryNodeExec to its children

2016-10-20 Thread Tejas Patil (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593478#comment-15593478 ] Tejas Patil commented on SPARK-18038: - Not sure if this deserves a jira but created one. This is a

[jira] [Created] (SPARK-18038) Move output partitioning definition from UnaryNodeExec to its children

2016-10-20 Thread Tejas Patil (JIRA)
Tejas Patil created SPARK-18038: --- Summary: Move output partitioning definition from UnaryNodeExec to its children Key: SPARK-18038 URL: https://issues.apache.org/jira/browse/SPARK-18038 Project: Spark

[jira] [Commented] (SPARK-13955) Spark in yarn mode fails

2016-10-20 Thread Tzach Zohar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593453#comment-15593453 ] Tzach Zohar commented on SPARK-13955: - [~saisai_shao] can you clarify regarding option #1: when you

[jira] [Commented] (SPARK-18037) Event listener should be aware of multiple tries of same stage

2016-10-20 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593347#comment-15593347 ] Josh Rosen commented on SPARK-18037: Ahhh, I remember there being other JIRAs related to a negative

[jira] [Created] (SPARK-18037) Event listener should be aware of multiple tries of same stage

2016-10-20 Thread Davies Liu (JIRA)
Davies Liu created SPARK-18037: -- Summary: Event listener should be aware of multiple tries of same stage Key: SPARK-18037 URL: https://issues.apache.org/jira/browse/SPARK-18037 Project: Spark

[jira] [Assigned] (SPARK-18019) Log instrumentation in GBTs

2016-10-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18019: Assignee: (was: Apache Spark) > Log instrumentation in GBTs >

[jira] [Commented] (SPARK-18019) Log instrumentation in GBTs

2016-10-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593306#comment-15593306 ] Apache Spark commented on SPARK-18019: -- User 'sethah' has created a pull request for this issue:

[jira] [Assigned] (SPARK-18019) Log instrumentation in GBTs

2016-10-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18019: Assignee: Apache Spark > Log instrumentation in GBTs > --- > >

[jira] [Commented] (SPARK-17916) CSV data source treats empty string as null no matter what nullValue option is

2016-10-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593300#comment-15593300 ] Hyukjin Kwon commented on SPARK-17916: -- Could I please ask what you think? cc [~felixcheung] > CSV

[jira] [Commented] (SPARK-17916) CSV data source treats empty string as null no matter what nullValue option is

2016-10-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593271#comment-15593271 ] Hyukjin Kwon commented on SPARK-17916: -- Oh, yes sure. I just thought the root problem is to

[jira] [Created] (SPARK-18036) Decision Trees do not handle edge cases

2016-10-20 Thread Seth Hendrickson (JIRA)
Seth Hendrickson created SPARK-18036: Summary: Decision Trees do not handle edge cases Key: SPARK-18036 URL: https://issues.apache.org/jira/browse/SPARK-18036 Project: Spark Issue Type:

[jira] [Commented] (SPARK-15777) Catalog federation

2016-10-20 Thread Yan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593229#comment-15593229 ] Yan commented on SPARK-15777: - One approach could be first tagging a subtree as specific to a data source,

[jira] [Assigned] (SPARK-18035) Unwrapping java maps in HiveInspectors allocates unnecessary buffer

2016-10-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18035: Assignee: Apache Spark > Unwrapping java maps in HiveInspectors allocates unnecessary

[jira] [Commented] (SPARK-18035) Unwrapping java maps in HiveInspectors allocates unnecessary buffer

2016-10-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593190#comment-15593190 ] Apache Spark commented on SPARK-18035: -- User 'tejasapatil' has created a pull request for this

[jira] [Assigned] (SPARK-18035) Unwrapping java maps in HiveInspectors allocates unnecessary buffer

2016-10-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18035: Assignee: (was: Apache Spark) > Unwrapping java maps in HiveInspectors allocates

[jira] [Created] (SPARK-18035) Unwrapping java maps in HiveInspectors allocates unnecessary buffer

2016-10-20 Thread Tejas Patil (JIRA)
Tejas Patil created SPARK-18035: --- Summary: Unwrapping java maps in HiveInspectors allocates unnecessary buffer Key: SPARK-18035 URL: https://issues.apache.org/jira/browse/SPARK-18035 Project: Spark

[jira] [Updated] (SPARK-18035) Unwrapping java maps in HiveInspectors allocates unnecessary buffer

2016-10-20 Thread Tejas Patil (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tejas Patil updated SPARK-18035: Description: In HiveInspectors, I saw that converting Java map to Spark's `ArrayBasedMapData`

[jira] [Commented] (SPARK-17916) CSV data source treats empty string as null no matter what nullValue option is

2016-10-20 Thread Suresh Thalamati (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593119#comment-15593119 ] Suresh Thalamati commented on SPARK-17916: -- Thank you for trying out the different scenarios. I

[jira] [Commented] (SPARK-18022) java.lang.NullPointerException instead of real exception when saving DF to MySQL

2016-10-20 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593008#comment-15593008 ] Dongjoon Hyun commented on SPARK-18022: --- Or, what you want is just general improvement for error

[jira] [Commented] (SPARK-18022) java.lang.NullPointerException instead of real exception when saving DF to MySQL

2016-10-20 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593005#comment-15593005 ] Dongjoon Hyun commented on SPARK-18022: --- Hi, [~maver1ck]. Could you give us more information to

[jira] [Commented] (SPARK-17829) Stable format for offset log

2016-10-20 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593000#comment-15593000 ] Cody Koeninger commented on SPARK-17829: At least with regard to kafka offsets, it might be good

[jira] [Updated] (SPARK-17829) Stable format for offset log

2016-10-20 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-17829: - Assignee: Tyson Condie > Stable format for offset log > > >

[jira] [Commented] (SPARK-16845) org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering" grows beyond 64 KB

2016-10-20 Thread Don Drake (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592862#comment-15592862 ] Don Drake commented on SPARK-16845: --- I compiled your branch and ran my large job and it finished

[jira] [Assigned] (SPARK-18034) Upgrade to MiMa 0.1.11

2016-10-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18034: Assignee: Josh Rosen (was: Apache Spark) > Upgrade to MiMa 0.1.11 >

[jira] [Assigned] (SPARK-18034) Upgrade to MiMa 0.1.11

2016-10-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18034: Assignee: Apache Spark (was: Josh Rosen) > Upgrade to MiMa 0.1.11 >

[jira] [Commented] (SPARK-18034) Upgrade to MiMa 0.1.11

2016-10-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592845#comment-15592845 ] Apache Spark commented on SPARK-18034: -- User 'JoshRosen' has created a pull request for this issue:

[jira] [Commented] (SPARK-10915) Add support for UDAFs in Python

2016-10-20 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592839#comment-15592839 ] Reynold Xin commented on SPARK-10915: - The current implementation of collect_list isn't going to work

[jira] [Created] (SPARK-18034) Upgrade to MiMa 0.1.11

2016-10-20 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-18034: -- Summary: Upgrade to MiMa 0.1.11 Key: SPARK-18034 URL: https://issues.apache.org/jira/browse/SPARK-18034 Project: Spark Issue Type: Bug Components:

[jira] [Commented] (SPARK-10915) Add support for UDAFs in Python

2016-10-20 Thread Jason White (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592831#comment-15592831 ] Jason White commented on SPARK-10915: - At the moment, we use .repartitionAndSortWithinPartitions to

[jira] [Updated] (SPARK-2629) Improved state management for Spark Streaming (mapWithState)

2016-10-20 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-2629: - Summary: Improved state management for Spark Streaming (mapWithState) (was: Improved state

[jira] [Resolved] (SPARK-18021) Refactor file name specification for data sources

2016-10-20 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18021. - Resolution: Fixed Fix Version/s: 2.1.0 > Refactor file name specification for data

[jira] [Created] (SPARK-18033) Deprecate TaskContext.partitionId

2016-10-20 Thread Cody Koeninger (JIRA)
Cody Koeninger created SPARK-18033: -- Summary: Deprecate TaskContext.partitionId Key: SPARK-18033 URL: https://issues.apache.org/jira/browse/SPARK-18033 Project: Spark Issue Type:

[jira] [Commented] (SPARK-10915) Add support for UDAFs in Python

2016-10-20 Thread Jason White (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592534#comment-15592534 ] Jason White commented on SPARK-10915: - That's unfortunate. Materializing a list somewhere is exactly

[jira] [Commented] (SPARK-10915) Add support for UDAFs in Python

2016-10-20 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592544#comment-15592544 ] Reynold Xin commented on SPARK-10915: - But if you need strict ordering guarantees, materializing them

[jira] [Commented] (SPARK-10915) Add support for UDAFs in Python

2016-10-20 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592514#comment-15592514 ] Davies Liu commented on SPARK-10915: [~jason.white] When a aggregate function is applied, the order

[jira] [Updated] (SPARK-17999) Add getPreferredLocations for KafkaSourceRDD

2016-10-20 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-17999: - Assignee: Saisai Shao > Add getPreferredLocations for KafkaSourceRDD >

[jira] [Resolved] (SPARK-17999) Add getPreferredLocations for KafkaSourceRDD

2016-10-20 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-17999. -- Resolution: Fixed Fix Version/s: 2.1.0 2.0.2 > Add

[jira] [Created] (SPARK-18032) Spark test failed as OOM in jenkins

2016-10-20 Thread Davies Liu (JIRA)
Davies Liu created SPARK-18032: -- Summary: Spark test failed as OOM in jenkins Key: SPARK-18032 URL: https://issues.apache.org/jira/browse/SPARK-18032 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-18031) Flaky test: org.apache.spark.streaming.scheduler.ExecutorAllocationManagerSuite basic functionality

2016-10-20 Thread Davies Liu (JIRA)
Davies Liu created SPARK-18031: -- Summary: Flaky test: org.apache.spark.streaming.scheduler.ExecutorAllocationManagerSuite basic functionality Key: SPARK-18031 URL: https://issues.apache.org/jira/browse/SPARK-18031

[jira] [Created] (SPARK-18030) Flaky test: org.apache.spark.sql.streaming.FileStreamSourceSuite

2016-10-20 Thread Davies Liu (JIRA)
Davies Liu created SPARK-18030: -- Summary: Flaky test: org.apache.spark.sql.streaming.FileStreamSourceSuite Key: SPARK-18030 URL: https://issues.apache.org/jira/browse/SPARK-18030 Project: Spark

[jira] [Commented] (SPARK-15687) Columnar execution engine

2016-10-20 Thread Evan Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592429#comment-15592429 ] Evan Chan commented on SPARK-15687: --- [~kiszk] thanks for the PR... would you mind pointing me to the

[jira] [Resolved] (SPARK-15780) Support mapValues on KeyValueGroupedDataset

2016-10-20 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-15780. - Resolution: Fixed Assignee: Koert Kuipers Fix Version/s: 2.1.0 > Support

[jira] [Resolved] (SPARK-17698) Join predicates should not contain filter clauses

2016-10-20 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-17698. - Resolution: Fixed Assignee: Tejas Patil Fix Version/s: 2.1.0 > Join predicates

[jira] [Commented] (SPARK-17904) Add a wrapper function to install R packages on each executors.

2016-10-20 Thread Piotr Smolinski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592272#comment-15592272 ] Piotr Smolinski commented on SPARK-17904: - Would it work at all? I have been looking recently on

[jira] [Comment Edited] (SPARK-17048) ML model read for custom transformers in a pipeline does not work

2016-10-20 Thread Nicolas Long (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592141#comment-15592141 ] Nicolas Long edited comment on SPARK-17048 at 10/20/16 3:32 PM: I hit

[jira] [Commented] (SPARK-17048) ML model read for custom transformers in a pipeline does not work

2016-10-20 Thread Nicolas Long (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592141#comment-15592141 ] Nicolas Long commented on SPARK-17048: -- I hit this today too. The Scala workaround is simply to

[jira] [Commented] (SPARK-15777) Catalog federation

2016-10-20 Thread Nattavut Sutyanyong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592092#comment-15592092 ] Nattavut Sutyanyong commented on SPARK-15777: - How do we test that a rule added in one data

[jira] [Assigned] (SPARK-18029) PruneFileSourcePartitions should not change the output of LogicalRelation

2016-10-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18029: Assignee: Apache Spark (was: Wenchen Fan) > PruneFileSourcePartitions should not change

[jira] [Commented] (SPARK-18029) PruneFileSourcePartitions should not change the output of LogicalRelation

2016-10-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592067#comment-15592067 ] Apache Spark commented on SPARK-18029: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-18029) PruneFileSourcePartitions should not change the output of LogicalRelation

2016-10-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18029: Assignee: Wenchen Fan (was: Apache Spark) > PruneFileSourcePartitions should not change

[jira] [Created] (SPARK-18029) PruneFileSourcePartitions should not change the output of LogicalRelation

2016-10-20 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-18029: --- Summary: PruneFileSourcePartitions should not change the output of LogicalRelation Key: SPARK-18029 URL: https://issues.apache.org/jira/browse/SPARK-18029 Project:

[jira] [Commented] (SPARK-9219) ClassCastException in instance of org.apache.spark.rdd.MapPartitionsRDD

2016-10-20 Thread Nick Orka (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592030#comment-15592030 ] Nick Orka commented on SPARK-9219: -- I've made a CLONE for the JIRA ticket here

[jira] [Commented] (SPARK-9219) ClassCastException in instance of org.apache.spark.rdd.MapPartitionsRDD

2016-10-20 Thread Nick Orka (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592041#comment-15592041 ] Nick Orka commented on SPARK-9219: -- I'm using IntelliJ Idea. Here is whole dependency tree (IML file)

[jira] [Issue Comment Deleted] (SPARK-9219) ClassCastException in instance of org.apache.spark.rdd.MapPartitionsRDD

2016-10-20 Thread Nick Orka (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Orka updated SPARK-9219: - Comment: was deleted (was: I've made a CLONE for the JIRA ticket here

[jira] [Updated] (SPARK-18016) Code Generation: Constant Pool Past Limit for Wide/Nested Dataset

2016-10-20 Thread Aleksander Eskilson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksander Eskilson updated SPARK-18016: Description: When attempting to encode collections of large Java objects to

[jira] [Comment Edited] (SPARK-17131) Code generation fails when running SQL expressions against a wide dataset (thousands of columns)

2016-10-20 Thread Aleksander Eskilson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592000#comment-15592000 ] Aleksander Eskilson edited comment on SPARK-17131 at 10/20/16 2:45 PM:

[jira] [Commented] (SPARK-17131) Code generation fails when running SQL expressions against a wide dataset (thousands of columns)

2016-10-20 Thread Aleksander Eskilson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592000#comment-15592000 ] Aleksander Eskilson commented on SPARK-17131: - Yeah, that makes sense. So far, what I

[jira] [Commented] (SPARK-17131) Code generation fails when running SQL expressions against a wide dataset (thousands of columns)

2016-10-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15591953#comment-15591953 ] Sean Owen commented on SPARK-17131: --- OK well I think it's fine to leave one copy of the "0x" issue

[jira] [Updated] (SPARK-18016) Code Generation: Constant Pool Past Limit for Wide/Nested Dataset

2016-10-20 Thread Aleksander Eskilson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksander Eskilson updated SPARK-18016: Description: When attempting to encode collections of large Java objects to

[jira] [Updated] (SPARK-18016) Code Generation: Constant Pool Past Limit for Wide/Nested Dataset

2016-10-20 Thread Aleksander Eskilson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksander Eskilson updated SPARK-18016: Summary: Code Generation: Constant Pool Past Limit for Wide/Nested Dataset (was:

[jira] [Resolved] (SPARK-18016) Code Generation Fails When Encoding Large Object to Wide Dataset

2016-10-20 Thread Aleksander Eskilson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksander Eskilson resolved SPARK-18016. - Resolution: Duplicate > Code Generation Fails When Encoding Large Object to Wide

[jira] [Commented] (SPARK-18016) Code Generation Fails When Encoding Large Object to Wide Dataset

2016-10-20 Thread Aleksander Eskilson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15591814#comment-15591814 ] Aleksander Eskilson commented on SPARK-18016: - As per some discussion in SPARK-17131, marking

[jira] [Commented] (SPARK-17131) Code generation fails when running SQL expressions against a wide dataset (thousands of columns)

2016-10-20 Thread Aleksander Eskilson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15591810#comment-15591810 ] Aleksander Eskilson commented on SPARK-17131: - Sure, I apologize for that. I'll also mark it

[jira] [Assigned] (SPARK-18028) simplify TableFileCatalog

2016-10-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18028: Assignee: Apache Spark (was: Wenchen Fan) > simplify TableFileCatalog >

[jira] [Commented] (SPARK-18028) simplify TableFileCatalog

2016-10-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15591794#comment-15591794 ] Apache Spark commented on SPARK-18028: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-18028) simplify TableFileCatalog

2016-10-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18028: Assignee: Wenchen Fan (was: Apache Spark) > simplify TableFileCatalog >

  1   2   >