[jira] [Commented] (SPARK-20954) DESCRIBE showing 1 extra row of "| # col_name | data_type | comment |"

2017-06-08 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043973#comment-16043973 ] Dongjoon Hyun commented on SPARK-20954: --- ^^ > DESCRIBE showing 1 extra row of "| # col_name |

[jira] [Commented] (SPARK-20954) DESCRIBE showing 1 extra row of "| # col_name | data_type | comment |"

2017-06-08 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043972#comment-16043972 ] Wenchen Fan commented on SPARK-20954: - oh sorry I misclicked... > DESCRIBE showing 1 extra row of "|

[jira] [Assigned] (SPARK-20954) DESCRIBE showing 1 extra row of "| # col_name | data_type | comment |"

2017-06-08 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-20954: --- Assignee: Dongjoon Hyun (was: Liang-Chi Hsieh) > DESCRIBE showing 1 extra row of "| #

[jira] [Commented] (SPARK-20954) DESCRIBE showing 1 extra row of "| # col_name | data_type | comment |"

2017-06-08 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043948#comment-16043948 ] Dongjoon Hyun commented on SPARK-20954: --- Hi, Wenchen. I'm Dongjoon. :) > DESCRIBE showing 1 extra

[jira] [Created] (SPARK-21030) extend hint syntax to support any expression for Python and R

2017-06-08 Thread Felix Cheung (JIRA)
Felix Cheung created SPARK-21030: Summary: extend hint syntax to support any expression for Python and R Key: SPARK-21030 URL: https://issues.apache.org/jira/browse/SPARK-21030 Project: Spark

[jira] [Created] (SPARK-21029) All StreamingQuery should be stopped when the SparkSession is stopped

2017-06-08 Thread Felix Cheung (JIRA)
Felix Cheung created SPARK-21029: Summary: All StreamingQuery should be stopped when the SparkSession is stopped Key: SPARK-21029 URL: https://issues.apache.org/jira/browse/SPARK-21029 Project: Spark

[jira] [Commented] (SPARK-20510) SparkR 2.2 QA: Update user guide for new features & APIs

2017-06-08 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043895#comment-16043895 ] Felix Cheung commented on SPARK-20510: -- credit to SPARK-20208 SPARK-20849 SPARK-20477 SPARK-20478

[jira] [Commented] (SPARK-20511) SparkR 2.2 QA: Check for new R APIs requiring example code

2017-06-08 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043894#comment-16043894 ] Felix Cheung commented on SPARK-20511: -- credit to SPARK-20208 SPARK-20849 SPARK-20477 SPARK-20478

[jira] [Commented] (SPARK-20513) Update SparkR website for 2.2

2017-06-08 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043892#comment-16043892 ] Felix Cheung commented on SPARK-20513: -- right, I don't think there's a site for R

[jira] [Assigned] (SPARK-20954) DESCRIBE showing 1 extra row of "| # col_name | data_type | comment |"

2017-06-08 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-20954: --- Assignee: Liang-Chi Hsieh > DESCRIBE showing 1 extra row of "| # col_name | data_type |

[jira] [Resolved] (SPARK-20954) DESCRIBE showing 1 extra row of "| # col_name | data_type | comment |"

2017-06-08 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-20954. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 18245

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

2017-06-08 Thread Fei Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043877#comment-16043877 ] Fei Shao commented on SPARK-20589: -- Tasks are assigned to executors. If we set the number of executors

[jira] [Commented] (SPARK-21001) Staging folders from Hive table are not being cleared.

2017-06-08 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043869#comment-16043869 ] Liang-Chi Hsieh commented on SPARK-21001: - No, I mean the current 2.0 branch in git. I think

[jira] [Comment Edited] (SPARK-20952) TaskContext should be an InheritableThreadLocal

2017-06-08 Thread Robert Kruszewski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043714#comment-16043714 ] Robert Kruszewski edited comment on SPARK-20952 at 6/9/17 2:12 AM: ---

[jira] [Comment Edited] (SPARK-20952) TaskContext should be an InheritableThreadLocal

2017-06-08 Thread Robert Kruszewski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043714#comment-16043714 ] Robert Kruszewski edited comment on SPARK-20952 at 6/9/17 2:13 AM: ---

[jira] [Commented] (SPARK-18075) UDF doesn't work on non-local spark

2017-06-08 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043778#comment-16043778 ] Wenchen Fan commented on SPARK-18075: - for development/testing, you can special the spark master as

[jira] [Resolved] (SPARK-20863) Add metrics/instrumentation to LiveListenerBus

2017-06-08 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-20863. - Resolution: Fixed Fix Version/s: 2.3.0 > Add metrics/instrumentation to LiveListenerBus >

[jira] [Assigned] (SPARK-13933) hadoop-2.7 profile's curator version should be 2.7.1

2017-06-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13933: Assignee: (was: Apache Spark) > hadoop-2.7 profile's curator version should be 2.7.1

[jira] [Commented] (SPARK-13933) hadoop-2.7 profile's curator version should be 2.7.1

2017-06-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043748#comment-16043748 ] Apache Spark commented on SPARK-13933: -- User 'wangyum' has created a pull request for this issue:

[jira] [Assigned] (SPARK-13933) hadoop-2.7 profile's curator version should be 2.7.1

2017-06-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13933: Assignee: Apache Spark > hadoop-2.7 profile's curator version should be 2.7.1 >

[jira] [Updated] (SPARK-21027) Parallel One vs. Rest Classifier

2017-06-08 Thread Ajay Saini (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Saini updated SPARK-21027: --- Description: Adding a class called ParOneVsRest that includes support for a parallelism parameter in

[jira] [Created] (SPARK-21028) Parallel One vs. Rest Classifier Scala

2017-06-08 Thread Ajay Saini (JIRA)
Ajay Saini created SPARK-21028: -- Summary: Parallel One vs. Rest Classifier Scala Key: SPARK-21028 URL: https://issues.apache.org/jira/browse/SPARK-21028 Project: Spark Issue Type: New Feature

[jira] [Updated] (SPARK-21027) Parallel One vs. Rest Classifier

2017-06-08 Thread Ajay Saini (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Saini updated SPARK-21027: --- Description: Adding a class called ParOneVsRest that includes support for a parallelism parameter in

[jira] [Created] (SPARK-21027) Parallel One vs. Rest Classifier

2017-06-08 Thread Ajay Saini (JIRA)
Ajay Saini created SPARK-21027: -- Summary: Parallel One vs. Rest Classifier Key: SPARK-21027 URL: https://issues.apache.org/jira/browse/SPARK-21027 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-20952) TaskContext should be an InheritableThreadLocal

2017-06-08 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043732#comment-16043732 ] Shixiong Zhu commented on SPARK-20952: -- For `ParquetFileFormat#readFootersInParallel`, I would

[jira] [Comment Edited] (SPARK-20952) TaskContext should be an InheritableThreadLocal

2017-06-08 Thread Robert Kruszewski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043714#comment-16043714 ] Robert Kruszewski edited comment on SPARK-20952 at 6/9/17 12:28 AM:

[jira] [Commented] (SPARK-20952) TaskContext should be an InheritableThreadLocal

2017-06-08 Thread Robert Kruszewski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043714#comment-16043714 ] Robert Kruszewski commented on SPARK-20952: --- Right, but how do I pass it downstream? >

[jira] [Commented] (SPARK-20952) TaskContext should be an InheritableThreadLocal

2017-06-08 Thread Robert Kruszewski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043712#comment-16043712 ] Robert Kruszewski commented on SPARK-20952: --- It doesn't but things underneath it do. It's weird

[jira] [Commented] (SPARK-20952) TaskContext should be an InheritableThreadLocal

2017-06-08 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043710#comment-16043710 ] Shixiong Zhu commented on SPARK-20952: -- Although I don't know what you plan to do, you can save the

[jira] [Commented] (SPARK-20952) TaskContext should be an InheritableThreadLocal

2017-06-08 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043708#comment-16043708 ] Shixiong Zhu commented on SPARK-20952: -- Why it needs TaskContext? > TaskContext should be an

[jira] [Commented] (SPARK-20952) TaskContext should be an InheritableThreadLocal

2017-06-08 Thread Robert Kruszewski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043704#comment-16043704 ] Robert Kruszewski commented on SPARK-20952: --- No modifications, it's this code

[jira] [Commented] (SPARK-20954) DESCRIBE showing 1 extra row of "| # col_name | data_type | comment |"

2017-06-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043700#comment-16043700 ] Apache Spark commented on SPARK-20954: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Commented] (SPARK-20952) TaskContext should be an InheritableThreadLocal

2017-06-08 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043694#comment-16043694 ] Shixiong Zhu commented on SPARK-20952: -- [~robert3005] could you show me your codes? Are you

[jira] [Commented] (SPARK-20952) TaskContext should be an InheritableThreadLocal

2017-06-08 Thread Robert Kruszewski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043669#comment-16043669 ] Robert Kruszewski commented on SPARK-20952: --- I am not really attached to the solution. Would be

[jira] [Commented] (SPARK-20952) TaskContext should be an InheritableThreadLocal

2017-06-08 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043655#comment-16043655 ] Shixiong Zhu commented on SPARK-20952: -- If TaskContext is not inheritable, we can always find a way

[jira] [Commented] (SPARK-20211) `1 > 0.0001` throws Decimal scale (0) cannot be greater than precision (-2) exception

2017-06-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043639#comment-16043639 ] Apache Spark commented on SPARK-20211: -- User 'gatorsmile' has created a pull request for this issue:

[jira] [Commented] (SPARK-20178) Improve Scheduler fetch failures

2017-06-08 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043622#comment-16043622 ] Josh Rosen commented on SPARK-20178: Update: I commented over on

[jira] [Commented] (SPARK-20952) TaskContext should be an InheritableThreadLocal

2017-06-08 Thread Robert Kruszewski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043623#comment-16043623 ] Robert Kruszewski commented on SPARK-20952: --- This is already an issue though on driver side

[jira] [Commented] (SPARK-20953) Add hash map metrics to aggregate and join

2017-06-08 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043581#comment-16043581 ] Reynold Xin commented on SPARK-20953: - I'd show the avg in the UI if possible. As a matter of fact

[jira] [Commented] (SPARK-10795) FileNotFoundException while deploying pyspark job on cluster

2017-06-08 Thread Nico Pappagianis (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043550#comment-16043550 ] Nico Pappagianis commented on SPARK-10795: -- [~HackerWilson] Were you able to resolve this? I'm

[jira] [Commented] (SPARK-20958) Roll back parquet-mr 1.8.2 to parquet-1.8.1

2017-06-08 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043478#comment-16043478 ] Cheng Lian commented on SPARK-20958: [~marmbrus], here is the draft release note entry: {quote}

[jira] [Created] (SPARK-21026) Document jenkins plug-ins assumed by the spark documentation build

2017-06-08 Thread Erik Erlandson (JIRA)
Erik Erlandson created SPARK-21026: -- Summary: Document jenkins plug-ins assumed by the spark documentation build Key: SPARK-21026 URL: https://issues.apache.org/jira/browse/SPARK-21026 Project:

[jira] [Resolved] (SPARK-21018) "Completed Jobs" and "Completed Stages" support pagination

2017-06-08 Thread Alex Bozarth (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Bozarth resolved SPARK-21018. -- Resolution: Duplicate This was added in Spark 2.1 by SPARK-15590 > "Completed Jobs" and

[jira] [Updated] (SPARK-20958) Roll back parquet-mr 1.8.2 to parquet-1.8.1

2017-06-08 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-20958: --- Labels: release-notes release_notes releasenotes (was: release-notes) > Roll back parquet-mr 1.8.2

[jira] [Issue Comment Deleted] (SPARK-21025) missing data in jsc.union

2017-06-08 Thread meng xi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] meng xi updated SPARK-21025: Comment: was deleted (was: I attached the Java file) > missing data in jsc.union >

[jira] [Reopened] (SPARK-21025) missing data in jsc.union

2017-06-08 Thread meng xi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] meng xi reopened SPARK-21025: - I attached the Java file > missing data in jsc.union > - > > Key:

[jira] [Commented] (SPARK-21025) missing data in jsc.union

2017-06-08 Thread meng xi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043390#comment-16043390 ] meng xi commented on SPARK-21025: - I attached the JAVA file, which does not have the format issue >

[jira] [Updated] (SPARK-21025) missing data in jsc.union

2017-06-08 Thread meng xi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] meng xi updated SPARK-21025: Attachment: SparkTest.java > missing data in jsc.union > - > >

[jira] [Updated] (SPARK-21025) missing data in jsc.union

2017-06-08 Thread meng xi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] meng xi updated SPARK-21025: Description: we are using an iterator of RDD for some special data processing, and then using union to

[jira] [Updated] (SPARK-21025) missing data in jsc.union

2017-06-08 Thread meng xi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] meng xi updated SPARK-21025: Description: we are using an iterator of RDD for some special data processing, and then using union to

[jira] [Updated] (SPARK-21025) missing data in jsc.union

2017-06-08 Thread meng xi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] meng xi updated SPARK-21025: Description: we are using an iterator of RDD for some special data processing, and then using union to

[jira] [Updated] (SPARK-21025) missing data in jsc.union

2017-06-08 Thread meng xi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] meng xi updated SPARK-21025: Description: we are using an iterator of RDD for some special data processing, and then using union to

[jira] [Commented] (SPARK-21025) missing data in jsc.union

2017-06-08 Thread meng xi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043379#comment-16043379 ] meng xi commented on SPARK-21025: - no, I just comment out one line, the system incorrectly format my code

[jira] [Closed] (SPARK-21022) RDD.foreach swallows exceptions

2017-06-08 Thread Colin Woodbury (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Woodbury closed SPARK-21022. -- Resolution: Invalid Wasn't actually a bug - `foreach` _doesn't_ actually swallow exceptions.

[jira] [Commented] (SPARK-21022) RDD.foreach swallows exceptions

2017-06-08 Thread Colin Woodbury (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043368#comment-16043368 ] Colin Woodbury commented on SPARK-21022: Ah ok, that makes sense for `foreachPartition`. And

[jira] [Commented] (SPARK-21024) CSV parse mode handles Univocity parser exceptions

2017-06-08 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043310#comment-16043310 ] Xiao Li commented on SPARK-21024: - Yes! We should fix it. Thanks! > CSV parse mode handles Univocity

[jira] [Issue Comment Deleted] (SPARK-21022) RDD.foreach swallows exceptions

2017-06-08 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-21022: - Comment: was deleted (was: ~~Good catch...~~) > RDD.foreach swallows exceptions >

[jira] [Commented] (SPARK-21022) RDD.foreach swallows exceptions

2017-06-08 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043266#comment-16043266 ] Shixiong Zhu commented on SPARK-21022: -- Wait. I also checked `foreach` method. It does throw the

[jira] [Resolved] (SPARK-21025) missing data in jsc.union

2017-06-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21025. --- Resolution: Invalid Oh, and I realized the mistake here. You put the body of the loop on one line

[jira] [Resolved] (SPARK-20976) Unify Error Messages for FAILFAST mode.

2017-06-08 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-20976. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18196

[jira] [Comment Edited] (SPARK-21022) RDD.foreach swallows exceptions

2017-06-08 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043223#comment-16043223 ] Shixiong Zhu edited comment on SPARK-21022 at 6/8/17 7:05 PM: -- ~~Good

[jira] [Commented] (SPARK-21022) RDD.foreach swallows exceptions

2017-06-08 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043247#comment-16043247 ] Shixiong Zhu commented on SPARK-21022: -- By the way, `foreachPartition` doesn't have the issue. It's

[jira] [Commented] (SPARK-21025) missing data in jsc.union

2017-06-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043229#comment-16043229 ] Sean Owen commented on SPARK-21025: --- It's not clear why you're parallelizing 'src' to begin with, or

[jira] [Assigned] (SPARK-21022) RDD.foreach swallows exceptions

2017-06-08 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-21022: Assignee: Shixiong Zhu > RDD.foreach swallows exceptions >

[jira] [Commented] (SPARK-21022) RDD.foreach swallows exceptions

2017-06-08 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043223#comment-16043223 ] Shixiong Zhu commented on SPARK-21022: -- Good catch... > RDD.foreach swallows exceptions >

[jira] [Commented] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-08 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043209#comment-16043209 ] Marcelo Vanzin commented on SPARK-21023: Then your best bet is a new command line option that

[jira] [Commented] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043191#comment-16043191 ] Lantao Jin commented on SPARK-21023: I think {{--conf}} couldn't help this. Because from the view of

[jira] [Commented] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043179#comment-16043179 ] Lantao Jin commented on SPARK-21023: {quote} it may break existing applications {quote} I really know

[jira] [Commented] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-08 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043176#comment-16043176 ] Marcelo Vanzin commented on SPARK-21023: bq. When and where the new config option be set?

[jira] [Commented] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043172#comment-16043172 ] Lantao Jin commented on SPARK-21023: {quote} Another option is to have a config option {quote} Oh,

[jira] [Commented] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-08 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043168#comment-16043168 ] Marcelo Vanzin commented on SPARK-21023: bq. The purpose is making the default configuration

[jira] [Created] (SPARK-21025) missing data in jsc.union

2017-06-08 Thread meng xi (JIRA)
meng xi created SPARK-21025: --- Summary: missing data in jsc.union Key: SPARK-21025 URL: https://issues.apache.org/jira/browse/SPARK-21025 Project: Spark Issue Type: Bug Components: Java

[jira] [Commented] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043164#comment-16043164 ] Lantao Jin commented on SPARK-21023: {quote} Another option is to have a config option {quote} LGTM

[jira] [Commented] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043160#comment-16043160 ] Lantao Jin commented on SPARK-21023: *The purpose is making the default configuration loaded

[jira] [Comment Edited] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043160#comment-16043160 ] Lantao Jin edited comment on SPARK-21023 at 6/8/17 6:06 PM: The purpose is

[jira] [Commented] (SPARK-20971) Purge the metadata log for FileStreamSource

2017-06-08 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043150#comment-16043150 ] Shixiong Zhu commented on SPARK-20971: -- FileStreamSource saves the seen files in the disk/HDFS, we

[jira] [Commented] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-08 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043132#comment-16043132 ] Marcelo Vanzin commented on SPARK-21023: Another option is to have a config option that controls

[jira] [Commented] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-08 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043131#comment-16043131 ] Marcelo Vanzin commented on SPARK-21023: bq. I suggest to change the current behavior Yes, and

[jira] [Commented] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043124#comment-16043124 ] Lantao Jin commented on SPARK-21023: [~vanzin] I suggest to change the current behavior and offer a

[jira] [Comment Edited] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-08 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043124#comment-16043124 ] Lantao Jin edited comment on SPARK-21023 at 6/8/17 5:45 PM: [~vanzin] I

[jira] [Assigned] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21023: Assignee: Apache Spark > Ignore to load default properties file is not a good choice from

[jira] [Commented] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043097#comment-16043097 ] Apache Spark commented on SPARK-21023: -- User 'LantaoJin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21023: Assignee: (was: Apache Spark) > Ignore to load default properties file is not a good

[jira] [Comment Edited] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-08 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043053#comment-16043053 ] Marcelo Vanzin edited comment on SPARK-21023 at 6/8/17 5:11 PM: I thought

[jira] [Commented] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-08 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043053#comment-16043053 ] Marcelo Vanzin commented on SPARK-21023: I thought we have an issue for adding a user-specific

[jira] [Commented] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043036#comment-16043036 ] Sean Owen commented on SPARK-21023: --- Maybe, but it would be a behavior change now. There are equal

[jira] [Commented] (SPARK-19185) ConcurrentModificationExceptions with CachedKafkaConsumers when Windowing

2017-06-08 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043030#comment-16043030 ] Marcelo Vanzin commented on SPARK-19185: I merged Mark's patch above to master and branch-2.2,

[jira] [Comment Edited] (SPARK-21001) Staging folders from Hive table are not being cleared.

2017-06-08 Thread Ajay Cherukuri (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042981#comment-16042981 ] Ajay Cherukuri edited comment on SPARK-21001 at 6/8/17 4:51 PM: Hi Liang,

[jira] [Resolved] (SPARK-21013) Spark History Server does not show the logs of completed Yarn Jobs

2017-06-08 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-21013. Resolution: Duplicate You need the MR history server for aggregated logs to show. This is

[jira] [Commented] (SPARK-21001) Staging folders from Hive table are not being cleared.

2017-06-08 Thread Ajay Cherukuri (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042981#comment-16042981 ] Ajay Cherukuri commented on SPARK-21001: do you mean 2.0.3? > Staging folders from Hive table

[jira] [Commented] (SPARK-21024) CSV parse mode handles Univocity parser exceptions

2017-06-08 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042925#comment-16042925 ] Takeshi Yamamuro commented on SPARK-21024: -- Is it worth fixing this? (I feel it is a kind of

[jira] [Created] (SPARK-21024) CSV parse mode handles Univocity parser exceptions

2017-06-08 Thread Takeshi Yamamuro (JIRA)
Takeshi Yamamuro created SPARK-21024: Summary: CSV parse mode handles Univocity parser exceptions Key: SPARK-21024 URL: https://issues.apache.org/jira/browse/SPARK-21024 Project: Spark

[jira] [Created] (SPARK-21023) Ignore to load default properties file is not a good choice from the perspective of system

2017-06-08 Thread Lantao Jin (JIRA)
Lantao Jin created SPARK-21023: -- Summary: Ignore to load default properties file is not a good choice from the perspective of system Key: SPARK-21023 URL: https://issues.apache.org/jira/browse/SPARK-21023

[jira] [Updated] (SPARK-21022) RDD.foreach swallows exceptions

2017-06-08 Thread Colin Woodbury (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Woodbury updated SPARK-21022: --- Summary: RDD.foreach swallows exceptions (was: foreach swallows exceptions) > RDD.foreach

[jira] [Updated] (SPARK-21022) foreach swallows exceptions

2017-06-08 Thread Colin Woodbury (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Woodbury updated SPARK-21022: --- Description: A `RDD.foreach` or `RDD.foreachPartition` call will swallow Exceptions thrown

[jira] [Created] (SPARK-21022) foreach swallows exceptions

2017-06-08 Thread Colin Woodbury (JIRA)
Colin Woodbury created SPARK-21022: -- Summary: foreach swallows exceptions Key: SPARK-21022 URL: https://issues.apache.org/jira/browse/SPARK-21022 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-13736) Big-Endian plataform issues

2017-06-08 Thread Luciano Resende (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved SPARK-13736. - Resolution: Invalid We are using the big-endian label to track platform related issues.

[jira] [Updated] (SPARK-20984) Reading back from ORC format gives error on big endian systems.

2017-06-08 Thread Luciano Resende (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende updated SPARK-20984: Labels: big-endian (was: ) > Reading back from ORC format gives error on big endian

[jira] [Commented] (SPARK-17237) DataFrame fill after pivot causing org.apache.spark.sql.AnalysisException

2017-06-08 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042713#comment-16042713 ] Takeshi Yamamuro commented on SPARK-17237: -- sorry for confusing you, but I originally mean

[jira] [Updated] (SPARK-21021) Reading partitioned parquet does not respect specified schema column order

2017-06-08 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michel Lemay updated SPARK-21021: - Description: When reading back a partitioned parquet folder, column order gets messed up.

  1   2   >