[jira] [Commented] (SPARK-17654) Propagate bucketing information for Hive tables to / from Catalog

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518468#comment-15518468 ] Apache Spark commented on SPARK-17654: -- User 'tejasapatil' has created a pull request for this

[jira] [Assigned] (SPARK-17654) Propagate bucketing information for Hive tables to / from Catalog

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17654: Assignee: (was: Apache Spark) > Propagate bucketing information for Hive tables to /

[jira] [Assigned] (SPARK-17654) Propagate bucketing information for Hive tables to / from Catalog

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17654: Assignee: Apache Spark > Propagate bucketing information for Hive tables to / from

[jira] [Commented] (SPARK-17654) Propagate bucketing information for Hive tables to / from Catalog

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518203#comment-15518203 ] Apache Spark commented on SPARK-17654: -- User 'tejasapatil' has created a pull request for this

[jira] [Updated] (SPARK-17655) Remove unused variables declarations and definations in a WholeStageCodeGened stage

2016-09-23 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-17655: - Summary: Remove unused variables declarations and definations in a WholeStageCodeGened stage (was:

[jira] [Commented] (SPARK-17655) [SQL]Remove unused variables declarations and definations in a WholeStageCodeGened stage

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518175#comment-15518175 ] Apache Spark commented on SPARK-17655: -- User 'yaooqinn' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17655) [SQL]Remove unused variables declarations and definations in a WholeStageCodeGened stage

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17655: Assignee: (was: Apache Spark) > [SQL]Remove unused variables declarations and

[jira] [Assigned] (SPARK-17655) [SQL]Remove unused variables declarations and definations in a WholeStageCodeGened stage

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17655: Assignee: Apache Spark > [SQL]Remove unused variables declarations and definations in a

[jira] [Created] (SPARK-17655) [SQL]Remove unused variables declarations and definations in a WholeStageCodeGened stage

2016-09-23 Thread Kent Yao (JIRA)
Kent Yao created SPARK-17655: Summary: [SQL]Remove unused variables declarations and definations in a WholeStageCodeGened stage Key: SPARK-17655 URL: https://issues.apache.org/jira/browse/SPARK-17655

[jira] [Created] (SPARK-17654) Propagate bucketing information for Hive tables to / from Catalog

2016-09-23 Thread Tejas Patil (JIRA)
Tejas Patil created SPARK-17654: --- Summary: Propagate bucketing information for Hive tables to / from Catalog Key: SPARK-17654 URL: https://issues.apache.org/jira/browse/SPARK-17654 Project: Spark

[jira] [Updated] (SPARK-17653) Optimizer should remove unnecessary distincts (in multiple unions)

2016-09-23 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-17653: Description: Query: {code} select 1 a union select 2 b union select 3 c {code} Explain plan:

[jira] [Created] (SPARK-17653) Optimizer should remove unnecessary distincts (in multiple unions)

2016-09-23 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-17653: --- Summary: Optimizer should remove unnecessary distincts (in multiple unions) Key: SPARK-17653 URL: https://issues.apache.org/jira/browse/SPARK-17653 Project: Spark

[jira] [Commented] (SPARK-17649) Log how many Spark events got dropped in LiveListenerBus

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517949#comment-15517949 ] Apache Spark commented on SPARK-17649: -- User 'zsxwing' has created a pull request for this issue:

[jira] [Commented] (SPARK-17634) Spark job hangs when using dapply

2016-09-23 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517839#comment-15517839 ] Felix Cheung commented on SPARK-17634: -- Also it would be great if you have a shareable example to

[jira] [Commented] (SPARK-17634) Spark job hangs when using dapply

2016-09-23 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517837#comment-15517837 ] Felix Cheung commented on SPARK-17634: -- I see. Do you know if the partitions are evenly distributed

[jira] [Assigned] (SPARK-17652) Fix confusing exception message while reserving capacity

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17652: Assignee: (was: Apache Spark) > Fix confusing exception message while reserving

[jira] [Commented] (SPARK-17652) Fix confusing exception message while reserving capacity

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517723#comment-15517723 ] Apache Spark commented on SPARK-17652: -- User 'sameeragarwal' has created a pull request for this

[jira] [Assigned] (SPARK-17652) Fix confusing exception message while reserving capacity

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17652: Assignee: Apache Spark > Fix confusing exception message while reserving capacity >

[jira] [Created] (SPARK-17652) Fix confusing exception message while reserving capacity

2016-09-23 Thread Sameer Agarwal (JIRA)
Sameer Agarwal created SPARK-17652: -- Summary: Fix confusing exception message while reserving capacity Key: SPARK-17652 URL: https://issues.apache.org/jira/browse/SPARK-17652 Project: Spark

[jira] [Updated] (SPARK-15703) Make ListenerBus event queue size configurable

2016-09-23 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-15703: - Fix Version/s: 2.0.2 > Make ListenerBus event queue size configurable >

[jira] [Comment Edited] (SPARK-17647) SQL LIKE/RLIKE do not handle backslashes correctly

2016-09-23 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517656#comment-15517656 ] Josh Rosen edited comment on SPARK-17647 at 9/23/16 9:52 PM: - Another piece

[jira] [Comment Edited] (SPARK-10713) SPARK_DIST_CLASSPATH ignored on Mesos executors

2016-09-23 Thread Michael McCarthy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517638#comment-15517638 ] Michael McCarthy edited comment on SPARK-10713 at 9/23/16 9:48 PM: --- I'm

[jira] [Commented] (SPARK-17647) SQL LIKE/RLIKE do not handle backslashes correctly

2016-09-23 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517656#comment-15517656 ] Josh Rosen commented on SPARK-17647: Another piece of evidence to help untangle this: In MySQL,

[jira] [Resolved] (SPARK-17651) Automate Spark version update for documentations

2016-09-23 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-17651. - Resolution: Fixed Assignee: Shivaram Venkataraman Fix Version/s: 2.1.0

[jira] [Commented] (SPARK-10713) SPARK_DIST_CLASSPATH ignored on Mesos executors

2016-09-23 Thread Michael McCarthy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517638#comment-15517638 ] Michael McCarthy commented on SPARK-10713: -- I'm also seeing this issue. Executors get lost with

[jira] [Commented] (SPARK-17647) SQL LIKE/RLIKE do not handle backslashes correctly

2016-09-23 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517636#comment-15517636 ] Josh Rosen commented on SPARK-17647: On the other hand, running {code} select '' like '%\\%'

[jira] [Commented] (SPARK-17651) Automate Spark version update for documentations

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517626#comment-15517626 ] Apache Spark commented on SPARK-17651: -- User 'shivaram' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17651) Automate Spark version update for documentations

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17651: Assignee: (was: Apache Spark) > Automate Spark version update for documentations >

[jira] [Assigned] (SPARK-17651) Automate Spark version update for documentations

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17651: Assignee: Apache Spark > Automate Spark version update for documentations >

[jira] [Assigned] (SPARK-17650) Adding a malformed URL to sc.addJar and/or sc.addFile bricks Executors

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17650: Assignee: Apache Spark > Adding a malformed URL to sc.addJar and/or sc.addFile bricks

[jira] [Commented] (SPARK-17650) Adding a malformed URL to sc.addJar and/or sc.addFile bricks Executors

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517591#comment-15517591 ] Apache Spark commented on SPARK-17650: -- User 'brkyvz' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17650) Adding a malformed URL to sc.addJar and/or sc.addFile bricks Executors

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17650: Assignee: (was: Apache Spark) > Adding a malformed URL to sc.addJar and/or sc.addFile

[jira] [Commented] (SPARK-17647) SQL LIKE/RLIKE do not handle backslashes correctly

2016-09-23 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517589#comment-15517589 ] Josh Rosen commented on SPARK-17647: I think that the first case is clearly a bug (and have a fix)

[jira] [Resolved] (SPARK-12221) Add CPU time metric to TaskMetrics

2016-09-23 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-12221. Resolution: Fixed Assignee: Jisoo Kim Fix Version/s: 2.1.0 > Add CPU time

[jira] [Created] (SPARK-17651) Automate Spark version update for documentations

2016-09-23 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-17651: --- Summary: Automate Spark version update for documentations Key: SPARK-17651 URL: https://issues.apache.org/jira/browse/SPARK-17651 Project: Spark Issue Type:

[jira] [Commented] (SPARK-17577) SparkR support add files to Spark job and get by executors

2016-09-23 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517408#comment-15517408 ] Kousuke Saruta commented on SPARK-17577: I add 2.0.1 as a fix version but we are in voting for

[jira] [Updated] (SPARK-17577) SparkR support add files to Spark job and get by executors

2016-09-23 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-17577: --- Fix Version/s: 2.0.1 > SparkR support add files to Spark job and get by executors >

[jira] [Commented] (SPARK-15703) Make ListenerBus event queue size configurable

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517377#comment-15517377 ] Apache Spark commented on SPARK-15703: -- User 'zsxwing' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17648) TaskSchedulerImpl.resourceOffers should take an IndexedSeq, not a Seq

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17648: Assignee: Imran Rashid (was: Apache Spark) > TaskSchedulerImpl.resourceOffers should

[jira] [Commented] (SPARK-17648) TaskSchedulerImpl.resourceOffers should take an IndexedSeq, not a Seq

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517361#comment-15517361 ] Apache Spark commented on SPARK-17648: -- User 'squito' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17648) TaskSchedulerImpl.resourceOffers should take an IndexedSeq, not a Seq

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17648: Assignee: Apache Spark (was: Imran Rashid) > TaskSchedulerImpl.resourceOffers should

[jira] [Commented] (SPARK-17648) TaskSchedulerImpl.resourceOffers should take an IndexedSeq, not a Seq

2016-09-23 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517344#comment-15517344 ] Imran Rashid commented on SPARK-17648: -- in this particular case, the index is used because there are

[jira] [Updated] (SPARK-17648) TaskSchedulerImpl.resourceOffers should take an IndexedSeq, not a Seq

2016-09-23 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-17648: - Description: {{TaskSchedulerImpl.resourceOffer}} takes in a {{Seq[WorkerOffer]}}. however,

[jira] [Resolved] (SPARK-17643) Remove comparable requirement from Offset

2016-09-23 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-17643. --- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 15207

[jira] [Updated] (SPARK-17650) Adding a malformed URL to sc.addJar and/or sc.addFile bricks Executors

2016-09-23 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17650: --- Component/s: Spark Core > Adding a malformed URL to sc.addJar and/or sc.addFile bricks Executors >

[jira] [Created] (SPARK-17650) Adding a malformed URL to sc.addJar and/or sc.addFile bricks Executors

2016-09-23 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-17650: --- Summary: Adding a malformed URL to sc.addJar and/or sc.addFile bricks Executors Key: SPARK-17650 URL: https://issues.apache.org/jira/browse/SPARK-17650 Project: Spark

[jira] [Assigned] (SPARK-17649) Log how many Spark events got dropped in LiveListenerBus

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17649: Assignee: Shixiong Zhu (was: Apache Spark) > Log how many Spark events got dropped in

[jira] [Commented] (SPARK-17649) Log how many Spark events got dropped in LiveListenerBus

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517248#comment-15517248 ] Apache Spark commented on SPARK-17649: -- User 'zsxwing' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17649) Log how many Spark events got dropped in LiveListenerBus

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17649: Assignee: Apache Spark (was: Shixiong Zhu) > Log how many Spark events got dropped in

[jira] [Updated] (SPARK-17649) Log how many Spark events got dropped in LiveListenerBus

2016-09-23 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-17649: - Priority: Minor (was: Major) > Log how many Spark events got dropped in LiveListenerBus >

[jira] [Commented] (SPARK-17210) sparkr.zip is not distributed to executors when run sparkr in RStudio

2016-09-23 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517236#comment-15517236 ] Reynold Xin commented on SPARK-17210: - [~felixcheung] please set the version to 2.0.2 from now on,

[jira] [Updated] (SPARK-17210) sparkr.zip is not distributed to executors when run sparkr in RStudio

2016-09-23 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-17210: Target Version/s: (was: 2.0.1, 2.1.0) > sparkr.zip is not distributed to executors when run

[jira] [Updated] (SPARK-17210) sparkr.zip is not distributed to executors when run sparkr in RStudio

2016-09-23 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-17210: Fix Version/s: (was: 2.0.1) 2.0.2 > sparkr.zip is not distributed to

[jira] [Created] (SPARK-17649) Log how many Spark events got dropped in LiveListenerBus

2016-09-23 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-17649: Summary: Log how many Spark events got dropped in LiveListenerBus Key: SPARK-17649 URL: https://issues.apache.org/jira/browse/SPARK-17649 Project: Spark

[jira] [Commented] (SPARK-17210) sparkr.zip is not distributed to executors when run sparkr in RStudio

2016-09-23 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517224#comment-15517224 ] Felix Cheung commented on SPARK-17210: -- cc [~rxin] - this is merged to master and branch-2.0. If we

[jira] [Comment Edited] (SPARK-17210) sparkr.zip is not distributed to executors when run sparkr in RStudio

2016-09-23 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517224#comment-15517224 ] Felix Cheung edited comment on SPARK-17210 at 9/23/16 6:41 PM: --- cc [~rxin]

[jira] [Resolved] (SPARK-17210) sparkr.zip is not distributed to executors when run sparkr in RStudio

2016-09-23 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-17210. -- Resolution: Fixed Assignee: Jeff Zhang Fix Version/s: 2.1.0

[jira] [Commented] (SPARK-14098) Generate code that get a float/double value in each column from CachedBatch when DataFrame.cache() is called

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517209#comment-15517209 ] Apache Spark commented on SPARK-14098: -- User 'kiszk' has created a pull request for this issue:

[jira] [Updated] (SPARK-17636) Parquet filter push down doesn't handle struct fields

2016-09-23 Thread Mitesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mitesh updated SPARK-17636: --- Affects Version/s: 1.6.3 > Parquet filter push down doesn't handle struct fields >

[jira] [Resolved] (SPARK-17499) make the default params in sparkR spark.mlp consistent with MultilayerPerceptronClassifier

2016-09-23 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-17499. -- Resolution: Fixed Assignee: Weichen Xu Fix Version/s: 2.1.0 Target

[jira] [Resolved] (SPARK-17646) SparkType::add method does not work in 2.0.0 (in Java)

2016-09-23 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-17646. Resolution: Not A Problem Yep, this is the intended behavior. From the Scaladoc: {code} /**

[jira] [Updated] (SPARK-16240) model loading backward compatibility for ml.clustering.LDA

2016-09-23 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-16240: --- Fix Version/s: (was: 2.0.1) 2.0.2 > model loading backward compatibility for

[jira] [Updated] (SPARK-14387) Enable Hive-1.x ORC compatibility with spark.sql.hive.convertMetastoreOrc

2016-09-23 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-14387: --- Target Version/s: 2.0.2 (was: 2.0.1) > Enable Hive-1.x ORC compatibility with

[jira] [Commented] (SPARK-17648) TaskSchedulerImpl.resourceOffers should take an IndexedSeq, not a Seq

2016-09-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517063#comment-15517063 ] Sean Owen commented on SPARK-17648: --- We recently cleaned up a lot of other instances of this type of

[jira] [Created] (SPARK-17648) TaskSchedulerImpl.resourceOffers should take an IndexedSeq, not a Seq

2016-09-23 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-17648: Summary: TaskSchedulerImpl.resourceOffers should take an IndexedSeq, not a Seq Key: SPARK-17648 URL: https://issues.apache.org/jira/browse/SPARK-17648 Project: Spark

[jira] [Commented] (SPARK-17454) Add option to specify Mesos resource offer constraints

2016-09-23 Thread Michael Gummelt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517007#comment-15517007 ] Michael Gummelt commented on SPARK-17454: - So you're trying to only launch executors on nodes

[jira] [Commented] (SPARK-17634) Spark job hangs when using dapply

2016-09-23 Thread Thomas Powell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516975#comment-15516975 ] Thomas Powell commented on SPARK-17634: --- Several hours. I modified {{worker.R}} so that it logs the

[jira] [Commented] (SPARK-17619) To add support for pattern matching in ArrayContains Expression.

2016-09-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516905#comment-15516905 ] Hyukjin Kwon commented on SPARK-17619: -- Do you mind if I ask you mean {{array_contains}} function?

[jira] [Commented] (SPARK-17646) SparkType::add method does not work in 2.0.0 (in Java)

2016-09-23 Thread Pawel Skorupinski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516876#comment-15516876 ] Pawel Skorupinski commented on SPARK-17646: --- [~srowen] Correct. If this is an assumed

[jira] [Assigned] (SPARK-17637) Packed scheduling for Spark tasks across executors

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17637: Assignee: (was: Apache Spark) > Packed scheduling for Spark tasks across executors >

[jira] [Assigned] (SPARK-17637) Packed scheduling for Spark tasks across executors

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17637: Assignee: Apache Spark > Packed scheduling for Spark tasks across executors >

[jira] [Commented] (SPARK-17637) Packed scheduling for Spark tasks across executors

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516881#comment-15516881 ] Apache Spark commented on SPARK-17637: -- User 'zhzhan' has created a pull request for this issue:

[jira] [Comment Edited] (SPARK-17646) SparkType::add method does not work in 2.0.0 (in Java)

2016-09-23 Thread Pawel Skorupinski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516876#comment-15516876 ] Pawel Skorupinski edited comment on SPARK-17646 at 9/23/16 4:17 PM:

[jira] [Commented] (SPARK-17637) Packed scheduling for Spark tasks across executors

2016-09-23 Thread Zhan Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516851#comment-15516851 ] Zhan Zhang commented on SPARK-17637: [~jerryshao] The idea is straightforward. Instead of doing round

[jira] [Updated] (SPARK-17647) SQL LIKE/RLIKE do not handle backslashes correctly

2016-09-23 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-17647: -- Labels: correctness (was: ) > SQL LIKE/RLIKE do not handle backslashes correctly >

[jira] [Updated] (SPARK-17647) SQL LIKE/RLIKE do not handle backslashes correctly

2016-09-23 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-17647: -- Description: Try the following in SQL shell: {code} select '' like '%\\%'; select ''

[jira] [Created] (SPARK-17647) SQL LIKE/RLIKE do not handle backslashes correctly

2016-09-23 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-17647: - Summary: SQL LIKE/RLIKE do not handle backslashes correctly Key: SPARK-17647 URL: https://issues.apache.org/jira/browse/SPARK-17647 Project: Spark Issue

[jira] [Updated] (SPARK-17647) SQL LIKE/RLIKE do not handle backslashes correctly

2016-09-23 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-17647: -- Description: Try the following in SQL shell: {code} select '' like '%\\%'; select ''

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516772#comment-15516772 ] Fei Wang commented on SPARK-17556: -- [~viirya] in this case how about notify driver to re-persist the

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516732#comment-15516732 ] Fei Wang commented on SPARK-17556: -- That's a good point! In your solution, the broadcast rdd must

[jira] [Commented] (SPARK-17454) Add option to specify Mesos resource offer constraints

2016-09-23 Thread Chris Bannister (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516685#comment-15516685 ] Chris Bannister commented on SPARK-17454: - The scheduler will write the to mesos disk sandbox if

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516668#comment-15516668 ] Liang-Chi Hsieh commented on SPARK-17556: - No. It doesn't. I think the point is not only the

[jira] [Comment Edited] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516639#comment-15516639 ] Fei Wang edited comment on SPARK-17556 at 9/23/16 2:50 PM: --- Yes, the main

[jira] [Commented] (SPARK-17577) SparkR support add files to Spark job and get by executors

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516658#comment-15516658 ] Apache Spark commented on SPARK-17577: -- User 'yanboliang' has created a pull request for this issue:

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516639#comment-15516639 ] Fei Wang commented on SPARK-17556: -- Yes, the main different is is does not introduce overhead to driver,

[jira] [Commented] (SPARK-17577) SparkR support add files to Spark job and get by executors

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516606#comment-15516606 ] Apache Spark commented on SPARK-17577: -- User 'yanboliang' has created a pull request for this issue:

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516577#comment-15516577 ] Liang-Chi Hsieh commented on SPARK-17556: - In other words, from the jira description we say "the

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516568#comment-15516568 ] Liang-Chi Hsieh commented on SPARK-17556: - OK. You create the broadcast object on one executor.

[jira] [Issue Comment Deleted] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-17556: - Comment: was deleted (was: Not correct, I just collect the broadcast ref to the driver but not the

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516545#comment-15516545 ] Fei Wang commented on SPARK-17556: -- Not correct, I just collect the broadcast ref to the driver but not

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516546#comment-15516546 ] Fei Wang commented on SPARK-17556: -- Not correct, I just collect the broadcast ref to the driver but not

[jira] [Comment Edited] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516490#comment-15516490 ] Liang-Chi Hsieh edited comment on SPARK-17556 at 9/23/16 1:49 PM: --

[jira] [Comment Edited] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516490#comment-15516490 ] Liang-Chi Hsieh edited comment on SPARK-17556 at 9/23/16 1:48 PM: --

[jira] [Comment Edited] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516478#comment-15516478 ] Liang-Chi Hsieh edited comment on SPARK-17556 at 9/23/16 1:45 PM: --

[jira] [Comment Edited] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516490#comment-15516490 ] Liang-Chi Hsieh edited comment on SPARK-17556 at 9/23/16 1:45 PM: --

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516490#comment-15516490 ] Liang-Chi Hsieh commented on SPARK-17556: - [~Fei Wang] I quickly go through your design doc.

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516478#comment-15516478 ] Liang-Chi Hsieh commented on SPARK-17556: - [~scwf]I already submitted a PR for this. Can you also

[jira] [Comment Edited] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516386#comment-15516386 ] Fei Wang edited comment on SPARK-17556 at 9/23/16 1:15 PM: --- [~rxin] attached a

[jira] [Comment Edited] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516386#comment-15516386 ] Fei Wang edited comment on SPARK-17556 at 9/23/16 1:05 PM: --- [~rxin] attached a

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516386#comment-15516386 ] Fei Wang commented on SPARK-17556: -- [~rxin] attached a design doc for the executor based broadcast. Will

  1   2   >