[GitHub] spark pull request: [SPARK-2087] [SQL] [WIP] Multiple thriftserver...

2015-02-05 Thread mallman
Github user mallman commented on the pull request: https://github.com/apache/spark/pull/4382#issuecomment-73081758 FWIW I'd like to add my two cents. The main piece of functionality the installation at my company would benefit from is independent user sessions. I'm not familiar

[GitHub] spark pull request: [SPARK-1442][SQL][WIP] Initial window function...

2015-03-19 Thread mallman
Github user mallman commented on the pull request: https://github.com/apache/spark/pull/3703#issuecomment-83691288 I'm confused. Why was this PR abruptly closed? Was there another active PR for window functions? --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: [SPARK-8401] [Build] Scala version switching b...

2015-06-18 Thread mallman
Github user mallman commented on the pull request: https://github.com/apache/spark/pull/6832#issuecomment-113206728 Indeed the build does generate the scaladoc in the right location, but the `docs/_plugin/copy_api_dirs.rb` is currently hardcoded to always look for the api docs

[GitHub] spark pull request: [SPARK-8401] [Build] Scala version switching b...

2015-06-19 Thread mallman
Github user mallman commented on the pull request: https://github.com/apache/spark/pull/6832#issuecomment-113548582 Sure thing. FYI, I'm leaving for Iceland tomorrow (Saturday), and I'll be away for two weeks. I will probably be incommunicado during this time. If you need something

[GitHub] spark pull request: Scala version switching build enhancements

2015-06-16 Thread mallman
Github user mallman commented on the pull request: https://github.com/apache/spark/pull/6832#issuecomment-112477102 @srowen Will create a Jira ticket. @ScrapCodes This is what I get with (presumably BSD) sed on OS X: ``` [msa@Michaels-MacBook-Pro spark-1.4]$ ./dev

[GitHub] spark pull request: Scala version switching build enhancements

2015-06-16 Thread mallman
Github user mallman commented on the pull request: https://github.com/apache/spark/pull/6832#issuecomment-112478712 @srowen Should I create one Jira ticket for this or multiple? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [SPARK-8401] [Build] Scala version switching b...

2015-06-18 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/6832#discussion_r32703376 --- Diff: dev/change-scala-version.sh --- @@ -0,0 +1,63 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under

[GitHub] spark pull request: [SPARK-8401] [Build] Scala version switching b...

2015-06-17 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/6832#discussion_r32649538 --- Diff: dev/change-scala-version.sh --- @@ -0,0 +1,63 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under

[GitHub] spark pull request: [SPARK-8401] [Build] Scala version switching b...

2015-06-17 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/6832#discussion_r32650306 --- Diff: dev/change-scala-version.sh --- @@ -0,0 +1,63 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under

[GitHub] spark pull request: [SPARK-8401] [Build] Scala version switching b...

2015-06-16 Thread mallman
Github user mallman commented on the pull request: https://github.com/apache/spark/pull/6832#issuecomment-112608540 @srowen I created the Jira ticket which shows the problem with the current version changing scripts. --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: Scala version switching build enhancements

2015-06-15 Thread mallman
GitHub user mallman opened a pull request: https://github.com/apache/spark/pull/6832 Scala version switching build enhancements These commits address a few minor issues in the Scala cross-version support in the build: 1. Correct two missing `${scala.binary.version}` pom

[GitHub] spark pull request: [SPARK-8401] [Build] Scala version switching b...

2015-07-06 Thread mallman
Github user mallman commented on the pull request: https://github.com/apache/spark/pull/6832#issuecomment-118999888 I've spent some more time googling around this problem. Unsurprisingly, there's plenty of discussion/frustration around finding a cross-platform solution. There doesn't

[GitHub] spark pull request: [SPARK-8401] [Build] Scala version switching b...

2015-07-06 Thread mallman
Github user mallman commented on the pull request: https://github.com/apache/spark/pull/6832#issuecomment-118743436 @srowen I just returned from my vacation abroad and am catching up. Sorry for the wait. I'll take a look at this tomorrow. Cheers. --- If your project is set up

[GitHub] spark pull request: [SPARK-8401] [Build] Scala version switching b...

2015-07-07 Thread mallman
Github user mallman commented on the pull request: https://github.com/apache/spark/pull/6832#issuecomment-119266285 I'll work on a revision following along the lines of what @ScrapCodes did and push it to this PR. Incidentally, I was going to suggest we use `mktemp` to create

[GitHub] spark pull request: [SPARK-9304] [BUILD] Improve backwards compati...

2015-07-24 Thread mallman
Github user mallman commented on the pull request: https://github.com/apache/spark/pull/7639#issuecomment-124568478 Thanks for the fix @srowen. It was my oversight to assume it was safe to remove these scripts in the first place. --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-8401] [Build] Scala version switching b...

2015-07-16 Thread mallman
Github user mallman commented on the pull request: https://github.com/apache/spark/pull/6832#issuecomment-122112493 @srowen Sorry, I've been swamped. I think I can get this done by Saturday if you want to wait. --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: [SPARK-8401] [Build] Scala version switching b...

2015-07-13 Thread mallman
Github user mallman commented on the pull request: https://github.com/apache/spark/pull/6832#issuecomment-120988236 @srowen @ScrapCodes Let me know if you'd like me to take on those additional tasks. Cheers. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: [SPARK-8401] [Build] Scala version switching b...

2015-07-20 Thread mallman
Github user mallman commented on the pull request: https://github.com/apache/spark/pull/6832#issuecomment-122957103 @srowen I've pushed a new commit to replace usage of `dev/change-version-to-*.sh` scripts with `dev/change-scala-version.sh version`. I also modified the latter so

[GitHub] spark pull request: [SPARK-8401] [Build] Scala version switching b...

2015-07-20 Thread mallman
Github user mallman commented on the pull request: https://github.com/apache/spark/pull/6832#issuecomment-122946797 @srowen I'm working on this now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-8401] [Build] Scala version switching b...

2015-07-07 Thread mallman
Github user mallman commented on the pull request: https://github.com/apache/spark/pull/6832#issuecomment-119333018 I've run into a roadblock. This syntax: sed -e '0,/scala\.binary\.version2.10/s//scala.binary.version2.11/' pom.xml doesn't work with my Mac's sed

[GitHub] spark pull request: [SPARK-8401] [Build] Scala version switching b...

2015-07-07 Thread mallman
Github user mallman commented on the pull request: https://github.com/apache/spark/pull/6832#issuecomment-119375569 The original code replaces only the first instance of `scala.binary.version2.10` in the file, which is the desired behavior. The code you presented replaces all of them

[GitHub] spark pull request: [SPARK-8401] [Build] Scala version switching b...

2015-07-08 Thread mallman
Github user mallman commented on the pull request: https://github.com/apache/spark/pull/6832#issuecomment-119708081 Thanks for the tip, @srowen. That works. I now have a version following approach (2) which I've verified works on OS X with its built-in sed. I'll test on a GNU

[GitHub] spark pull request: [SPARK-8401] [Build] Scala version switching b...

2015-07-09 Thread mallman
Github user mallman commented on the pull request: https://github.com/apache/spark/pull/6832#issuecomment-120059239 I've pushed a commit to implement the second strategy. I've tested this script successfully on OS X Yosemite and Ubuntu 14. --- If your project is set up for it, you

[GitHub] spark pull request: [SPARK-8982][Core] Worker hostnames not showin...

2015-12-17 Thread mallman
Github user mallman commented on the pull request: https://github.com/apache/spark/pull/7345#issuecomment-165532682 To add my two cents, I think that to call this change "cosmetic" is strictly true but underrates its value. In our case we have additional monitori

[GitHub] spark pull request: [SPARK-8982][Core] Worker hostnames not showin...

2015-12-17 Thread mallman
Github user mallman commented on the pull request: https://github.com/apache/spark/pull/7345#issuecomment-165559632 @andrewor14 We've put this in production. Everything looks good. Hostnames show up in the UI as expected. No broken links. --- If your project is set up for it, you

[GitHub] spark pull request #13686: [SPARK-15968][SQL] HiveMetastoreCatalog does not ...

2016-06-15 Thread mallman
GitHub user mallman opened a pull request: https://github.com/apache/spark/pull/13686 [SPARK-15968][SQL] HiveMetastoreCatalog does not correctly validate ## What changes were proposed in this pull request? The `getCached` method of `HiveMetastoreCatalog` computes

[GitHub] spark pull request #13818: [SPARK-15968][SQL] Nonempty partitioned metastore...

2016-06-22 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/13818#discussion_r68064203 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/parquetSuites.scala --- @@ -425,6 +425,28 @@ class ParquetMetastoreSuite extends

[GitHub] spark issue #13686: [SPARK-15968][SQL] HiveMetastoreCatalog does not correct...

2016-06-15 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/13686 Actually, let me think about this some more... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #13686: [SPARK-15968][SQL] HiveMetastoreCatalog does not correct...

2016-06-15 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/13686 Aaaak! Some unit tests are failing on my build. Sorry, I will re-examine and submit a new commit. Ugh. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark issue #13686: [SPARK-15968][SQL] HiveMetastoreCatalog does not correct...

2016-06-15 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/13686 @hvanhovell Sounds like a good idea, but I don't know how to unit test this without opening up some of this caching api to at least the `private[hive]` access level. Would that be acceptable? I'm

[GitHub] spark issue #13686: [SPARK-15968][SQL] HiveMetastoreCatalog does not correct...

2016-06-15 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/13686 I'm going to close this PR and open a new one when I've fixed the test failures. My bad. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request #13686: [SPARK-15968][SQL] HiveMetastoreCatalog does not ...

2016-06-15 Thread mallman
Github user mallman closed the pull request at: https://github.com/apache/spark/pull/13686 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request #13818: [SPARK-15968][SQL] Nonempty partitioned metastore...

2016-06-21 Thread mallman
GitHub user mallman opened a pull request: https://github.com/apache/spark/pull/13818 [SPARK-15968][SQL] Nonempty partitioned metastore tables are not cached (Please note this is a revision of PR #13686, which has been closed in favor of this PR.) ## What changes were

[GitHub] spark issue #13818: [SPARK-15968][SQL] Nonempty partitioned metastore tables...

2016-06-21 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/13818 @hvanhovell I'm mentioning you here because you commented on my previous PR for this Jira issue. In response to your original question, yes, I have added a unit test for this patch. --- If your

[GitHub] spark pull request: [SPARK-6950][CORE] Stop the event logger befor...

2016-01-11 Thread mallman
Github user mallman commented on the pull request: https://github.com/apache/spark/pull/10700#issuecomment-170659870 Changed Jira ref from SPARK-6950 to SPARK-12755. SPARK-6950 is an older, defunct ticket. Oops. --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: [SPARK-6950][CORE] Stop the event logger befor...

2016-01-11 Thread mallman
GitHub user mallman opened a pull request: https://github.com/apache/spark/pull/10700 [SPARK-6950][CORE] Stop the event logger before the DAG scheduler [SPARK-6950][CORE] Stop the event logger before the DAG scheduler to avoid a race condition where the standalone master attempts

[GitHub] spark pull request: [SPARK-12755][CORE] Stop the event logger befo...

2016-01-18 Thread mallman
Github user mallman commented on the pull request: https://github.com/apache/spark/pull/10700#issuecomment-172564021 Sorry guys. I bungled the ordering of the `stop()` calls. That's what I get for doing a manual patch from a manual diff from another branch-1.5... :disappointed

[GitHub] spark pull request: [SPARK-12755][CORE] Stop the event logger befo...

2016-01-17 Thread mallman
Github user mallman commented on the pull request: https://github.com/apache/spark/pull/10700#issuecomment-172440507 Hi Josh, Good questions. I may have submitted this PR incorrectly. Perhaps you can guide me in the right direction. I submitted this PR for merging

[GitHub] spark pull request: [SPARK-12755][CORE] Stop the event logger befo...

2016-01-17 Thread mallman
Github user mallman commented on the pull request: https://github.com/apache/spark/pull/10700#issuecomment-172443532 I should also state that my original motivation in submitting this patch was to address the confusing log messages Application ... is still in progress

[GitHub] spark pull request: [SPARK-12755][CORE] Stop the event logger befo...

2016-01-21 Thread mallman
Github user mallman commented on the pull request: https://github.com/apache/spark/pull/10700#issuecomment-173490509 Here are my current thoughts. Josh says this functionality is going to be removed in Spark 2.0. The bug this PR is designed to address manifests itself in Spark 1.5

[GitHub] spark pull request: [SPARK-12755][CORE] Stop the event logger befo...

2016-01-25 Thread mallman
Github user mallman commented on the pull request: https://github.com/apache/spark/pull/10700#issuecomment-174599806 Thanks, @srowen. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request #14551: [SPARK-16961][CORE] Fixed off-by-one error that b...

2016-08-10 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/14551#discussion_r74276138 --- Diff: core/src/test/scala/org/apache/spark/util/UtilsSuite.scala --- @@ -874,4 +874,38 @@ class UtilsSuite extends SparkFunSuite

[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

2016-08-12 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/14537 @rajeshbalamohan We'll need a committer to review your patch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #14031: [SPARK-16353][BUILD][DOC] Missing javadoc options for ja...

2016-07-04 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/14031 Thank you. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #13818: [SPARK-15968][SQL] Nonempty partitioned metastore tables...

2016-07-05 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/13818 @zsxwing I was able to do following without error: git clone g...@github.com:apache/spark.git spark-master cd spark-master ./dev/change-scala-version.sh 2.10

[GitHub] spark issue #13818: [SPARK-15968][SQL] Nonempty partitioned metastore tables...

2016-07-05 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/13818 > I have a few questions. > > Is it a regression from 1.6? Looks like not? I don't know about 1.6. I know it's a regression from 1.5. > Is it a corre

[GitHub] spark issue #14064: [SPARK-15968][SQL] Nonempty partitioned metastore tables...

2016-07-06 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/14064 @cloud-fan Muchas gracias! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #14031: [SPARK-16353][BUILD][DOC] Missing javadoc options...

2016-07-04 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/14031#discussion_r69479805 --- Diff: project/SparkBuild.scala --- @@ -723,8 +723,8 @@ object Unidoc { .map(_.filterNot(_.getCanonicalPath.contains("org/apache/h

[GitHub] spark issue #13818: [SPARK-15968][SQL] Nonempty partitioned metastore tables...

2016-07-04 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/13818 I believe I've addressed @liancheng's style issues in my new unit test, along with the same in the two tests from which it was copy-pasta'd (boy scout rule). Hopefully I didn't cock it up

[GitHub] spark pull request #13818: [SPARK-15968][SQL] Nonempty partitioned metastore...

2016-08-08 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/13818#discussion_r73892580 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala --- @@ -298,6 +298,7 @@ case class InsertIntoHiveTable

[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

2016-08-09 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/14537#discussion_r74003788 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -294,7 +294,9 @@ private[hive] class HiveMetastoreCatalog

[GitHub] spark issue #14537: [SPARK-16948][SQL] Querying empty partitioned orc tables...

2016-08-09 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/14537 @rajeshbalamohan, the changes to `HiveMetastoreCatalog.scala` look reasonable. This mirrors the behavior of this method before the `if (fileType.equals("parquet"))` expression was

[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

2016-08-09 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/14537#discussion_r74092457 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -287,14 +287,14 @@ private[hive] class HiveMetastoreCatalog

[GitHub] spark pull request #14537: [SPARK-16948][SQL] Querying empty partitioned orc...

2016-08-09 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/14537#discussion_r74092780 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -287,14 +287,14 @@ private[hive] class HiveMetastoreCatalog

[GitHub] spark pull request #13818: [SPARK-15968][SQL] Nonempty partitioned metastore...

2016-06-30 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/13818#discussion_r69106546 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -200,7 +201,6 @@ private[hive] class HiveMetastoreCatalog

[GitHub] spark pull request #13818: [SPARK-15968][SQL] Nonempty partitioned metastore...

2016-06-30 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/13818#discussion_r69107723 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -191,6 +191,7 @@ private[hive] class HiveMetastoreCatalog

[GitHub] spark pull request #13818: [SPARK-15968][SQL] Nonempty partitioned metastore...

2016-06-30 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/13818#discussion_r69159655 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -200,7 +201,6 @@ private[hive] class HiveMetastoreCatalog

[GitHub] spark pull request #13818: [SPARK-15968][SQL] Nonempty partitioned metastore...

2016-06-30 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/13818#discussion_r69233431 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -265,9 +265,12 @@ private[hive] class HiveMetastoreCatalog

[GitHub] spark pull request #13818: [SPARK-15968][SQL] Nonempty partitioned metastore...

2016-06-30 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/13818#discussion_r69230833 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -265,9 +265,12 @@ private[hive] class HiveMetastoreCatalog

[GitHub] spark pull request #13818: [SPARK-15968][SQL] Nonempty partitioned metastore...

2016-06-30 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/13818#discussion_r69231754 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala --- @@ -298,6 +298,7 @@ case class InsertIntoHiveTable

[GitHub] spark issue #13818: [SPARK-15968][SQL] Nonempty partitioned metastore tables...

2016-06-30 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/13818 You are very welcome. Thank you for taking time to review it! 😃 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request #14031: [SPARK-16353][BUILD][DOC] Missing javadoc options...

2016-07-02 Thread mallman
GitHub user mallman opened a pull request: https://github.com/apache/spark/pull/14031 [SPARK-16353][BUILD][DOC] Missing javadoc options for java unidoc ## What changes were proposed in this pull request? The javadoc options for the java unidoc generation are ignored when

[GitHub] spark pull request #14031: [SPARK-16353][BUILD][DOC] Missing javadoc options...

2016-07-02 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/14031#discussion_r69382212 --- Diff: project/SparkBuild.scala --- @@ -723,8 +723,8 @@ object Unidoc { .map(_.filterNot(_.getCanonicalPath.contains("org/apache/h

[GitHub] spark pull request #14031: [SPARK-16353][BUILD][DOC] Missing javadoc options...

2016-07-02 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/14031#discussion_r69385673 --- Diff: project/SparkBuild.scala --- @@ -723,8 +723,8 @@ object Unidoc { .map(_.filterNot(_.getCanonicalPath.contains("org/apache/h

[GitHub] spark pull request #14690: [SPARK-16980][SQL] Load only catalog table partit...

2016-08-17 Thread mallman
GitHub user mallman opened a pull request: https://github.com/apache/spark/pull/14690 [SPARK-16980][SQL] Load only catalog table partition metadata required to answer a query (This PR addresses https://issues.apache.org/jira/browse/SPARK-16980.) (N.B. I'm submitting

[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning

2017-01-31 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/16578#discussion_r98819150 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/GetStructField2.scala --- @@ -0,0 +1,33 @@ +/* + * Licensed

[GitHub] spark issue #16281: [SPARK-13127][SQL] Update Parquet to 1.9.0

2017-01-31 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/16281 FYI, we've been using 1.9.0 patched with a fix for https://issues.apache.org/jira/browse/PARQUET-783 without problem. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark issue #16751: [SPARK-19409][BUILD] Bump parquet version to 1.8.2

2017-01-31 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/16751 FYI, there are at least two workarounds in the Spark codebase which can potentially be removed as a consequence of this upgrade. For example: https://github.com/apache/spark/blob

[GitHub] spark issue #16797: [SPARK-19455][SQL] Add option for case-insensitive Parqu...

2017-02-06 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/16797 The proposal to restore schema inference with finer grained control on when it is performed sounds reasonable to me. The case I'm most interested in is turning off schema inference entirely

[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning

2017-02-08 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/16578#discussion_r100229300 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/SelectedField.scala --- @@ -0,0 +1,76 @@ +/* + * Licensed

[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning

2017-02-08 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/16578#discussion_r100229358 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/GetStructField2.scala --- @@ -0,0 +1,33 @@ +/* + * Licensed

[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning

2017-02-01 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/16578#discussion_r98920657 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/GetStructField2.scala --- @@ -0,0 +1,33 @@ +/* + * Licensed

[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning

2017-02-02 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/16578#discussion_r99174674 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/SelectedField.scala --- @@ -0,0 +1,76 @@ +/* + * Licensed

[GitHub] spark pull request #15125: [SPARK-5484][GraphX] Periodically do checkpoint i...

2017-02-20 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/15125#discussion_r102056438 --- Diff: docs/graphx-programming-guide.md --- @@ -708,7 +708,9 @@ messages remaining. > messaging function. These constraints allow additio

[GitHub] spark issue #15125: [SPARK-5484][GraphX] Periodically do checkpoint in Prege...

2017-02-20 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/15125 @dding3, thank you for your continued patience and dedication to this PR, despite the continued change requests. We are getting closer to a merge. --- If your project is set up for it, you can

[GitHub] spark pull request #15125: [SPARK-5484][GraphX] Periodically do checkpoint i...

2017-02-20 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/15125#discussion_r102057156 --- Diff: docs/graphx-programming-guide.md --- @@ -708,7 +708,9 @@ messages remaining. > messaging function. These constraints allow additio

[GitHub] spark pull request #15125: [SPARK-5484][GraphX] Periodically do checkpoint i...

2017-02-20 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/15125#discussion_r102053462 --- Diff: graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala --- @@ -154,7 +169,9 @@ object Pregel extends Logging { // count

[GitHub] spark pull request #16499: [SPARK-17204][CORE] Fix replicated off heap stora...

2017-02-17 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/16499#discussion_r101809872 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -1018,7 +1025,9 @@ private[spark] class BlockManager( try

[GitHub] spark pull request #15125: [SPARK-5484][GraphX] Periodically do checkpoint i...

2017-02-17 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/15125#discussion_r101819321 --- Diff: graphx/src/main/scala/org/apache/spark/graphx/util/PeriodicGraphCheckpointer.scala --- @@ -87,10 +87,10 @@ private[mllib] class

[GitHub] spark pull request #16499: [SPARK-17204][CORE] Fix replicated off heap stora...

2017-02-16 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/16499#discussion_r101675576 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -813,7 +813,14 @@ private[spark] class BlockManager

[GitHub] spark issue #15125: [SPARK-5484][GraphX] Periodically do checkpoint in Prege...

2017-02-16 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/15125 LGTM. @felixcheung are we good to merge? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #16499: [SPARK-17204][CORE] Fix replicated off heap stora...

2017-02-16 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/16499#discussion_r101675669 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -1018,7 +1025,9 @@ private[spark] class BlockManager( try

[GitHub] spark issue #15125: [SPARK-5484][GraphX] Periodically do checkpoint in Prege...

2017-02-16 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/15125 @dding3 I submitted a PR against your `cp2_pregel` branch. If you merge that PR into your branch, it will be reflected in this PR. This is my PR: https://github.com/dding3/spark/pull/1

[GitHub] spark pull request #16499: [SPARK-17204][CORE] Fix replicated off heap stora...

2017-02-17 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/16499#discussion_r101809099 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -813,7 +813,14 @@ private[spark] class BlockManager

[GitHub] spark pull request #15125: [SPARK-5484][GraphX] Periodically do checkpoint i...

2017-02-17 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/15125#discussion_r101818789 --- Diff: graphx/src/main/scala/org/apache/spark/graphx/GraphOps.scala --- @@ -362,12 +362,14 @@ class GraphOps[VD: ClassTag, ED: ClassTag](graph: Graph[VD

[GitHub] spark pull request #16499: [SPARK-17204][CORE] Fix replicated off heap stora...

2017-02-21 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/16499#discussion_r102271763 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -1018,7 +1025,9 @@ private[spark] class BlockManager( try

[GitHub] spark pull request #16499: [SPARK-17204][CORE] Fix replicated off heap stora...

2017-02-21 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/16499#discussion_r102272981 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -813,7 +813,14 @@ private[spark] class BlockManager

[GitHub] spark pull request #15125: [SPARK-5484][GraphX] Periodically do checkpoint i...

2017-02-21 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/15125#discussion_r102290972 --- Diff: docs/graphx-programming-guide.md --- @@ -708,7 +708,9 @@ messages remaining. > messaging function. These constraints allow additio

[GitHub] spark pull request #15125: [SPARK-5484][GraphX] Periodically do checkpoint i...

2017-02-21 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/15125#discussion_r102292537 --- Diff: graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala --- @@ -122,27 +125,39 @@ object Pregel extends Logging { require

[GitHub] spark pull request #16499: [SPARK-17204][CORE] Fix replicated off heap stora...

2017-02-21 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/16499#discussion_r102293219 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -843,7 +852,15 @@ private[spark] class BlockManager

[GitHub] spark pull request #16499: [SPARK-17204][CORE] Fix replicated off heap stora...

2017-02-21 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/16499#discussion_r102293681 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -317,6 +317,9 @@ private[spark] class BlockManager

[GitHub] spark issue #15125: [SPARK-5484][GraphX] Periodically do checkpoint in Prege...

2017-02-14 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/15125 Our connected components computation completed successfully, with performance as expected. I've created a PR against @dding3's PR branch to incorporate a couple simple things. Then I think we're

[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning

2017-02-14 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/16578 @viirya I've added a commit to address some of your feedback. I will have another commit to address the others, but I'm not sure when I'll have it in. Hopefully by the end of next week

[GitHub] spark issue #16797: [SPARK-19455][SQL] Add option for case-insensitive Parqu...

2017-02-09 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/16797 BTW @budde, given that this represents a regression in behavior from previous versions of Spark, I think it is too generous of you to label the Jira issue as an "improvement" instead of

[GitHub] spark pull request #15125: [SPARK-5484][GraphX] Periodically do checkpoint i...

2017-02-10 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/15125#discussion_r100609529 --- Diff: graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala --- @@ -123,16 +127,25 @@ object Pregel extends Logging { s" bu

[GitHub] spark pull request #15125: [SPARK-5484][GraphX] Periodically do checkpoint i...

2017-02-10 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/15125#discussion_r100612840 --- Diff: graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala --- @@ -123,16 +127,25 @@ object Pregel extends Logging { s" bu

[GitHub] spark pull request #15125: [SPARK-5484][GraphX] Periodically do checkpoint i...

2017-02-10 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/15125#discussion_r100608839 --- Diff: graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala --- @@ -123,16 +127,25 @@ object Pregel extends Logging { s" bu

[GitHub] spark pull request #15125: [SPARK-5484][GraphX] Periodically do checkpoint i...

2017-02-10 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/15125#discussion_r100638292 --- Diff: graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala --- @@ -123,16 +127,25 @@ object Pregel extends Logging { s" bu

[GitHub] spark pull request #15125: [SPARK-5484][GraphX] Periodically do checkpoint i...

2017-02-10 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/15125#discussion_r100641170 --- Diff: graphx/src/main/scala/org/apache/spark/graphx/util/PeriodicGraphCheckpointer.scala --- @@ -87,10 +88,7 @@ private[mllib] class

[GitHub] spark pull request #15125: [SPARK-5484][GraphX] Periodically do checkpoint i...

2017-02-10 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/15125#discussion_r100632148 --- Diff: mllib/src/test/scala/org/apache/spark/mllib/impl/PeriodicRDDCheckpointerSuite.scala --- @@ -23,7 +23,7 @@ import org.apache.spark.{SparkContext

  1   2   3   4   5   6   7   >