[GitHub] spark pull request #13751: [SPARK-15159][SPARKR] SparkSession roxygen2 doc, ...

2016-06-18 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13751#discussion_r67604883 --- Diff: docs/sparkr.md --- @@ -263,19 +256,19 @@ head(df) ## Running SQL Queries from SparkR -A SparkR DataFrame can also

[GitHub] spark issue #13872: [SPARK-16164][SQL] Update `CombineFilters` to try to con...

2016-06-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13872 Thank you, @rxin . I hope so, too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #13872: [SPARK-16164][SQL] Update `CombineFilters` to try to con...

2016-06-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13872 Thank you, @mengxr , @liancheng , and @rxin . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request #13876: [SPARK-16174][SQL] Add RemoveLiteralRepetitionFro...

2016-06-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13876#discussion_r68328083 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -793,6 +794,20 @@ object ConstantFolding extends

[GitHub] spark pull request #13876: [SPARK-16174][SQL] Add RemoveLiteralRepetitionFro...

2016-06-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13876#discussion_r68329169 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -793,6 +794,20 @@ object ConstantFolding extends

[GitHub] spark issue #13870: [SPARK-16165][SQL] Fix the update logic for InMemoryTabl...

2016-06-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13870 Thank you for merging, @liancheng and @davies . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #13884: [SPARK-16181][SQL] outer join with isNull filter may ret...

2016-06-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13884 Hi, @cloud-fan . LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #13765: [SPARK-16052][SQL] Add CollapseRepartitionBy optimizer

2016-06-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13765 Hi, @cloud-fan . Could you review this optimizer? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request #13930: [SPARK-16228][SQL] HiveSessionCatalog should retu...

2016-06-27 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13930#discussion_r68682430 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala --- @@ -174,6 +175,18 @@ private[sql] class HiveSessionCatalog

[GitHub] spark pull request #13914: [SPARK-16111][SQL][DOC] Hide SparkOrcNewRecordRea...

2016-06-27 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13914#discussion_r68681583 --- Diff: project/SparkBuild.scala --- @@ -720,6 +720,7 @@ object Unidoc { // Skip class names containing $ and some internal packages

[GitHub] spark issue #13930: [SPARK-16228][SQL] HiveSessionCatalog should return `dou...

2016-06-27 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13930 Hi, @hvanhovell . Could you review this PR again? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request #13906: [SPARK-16208][SQL] Add `CollapseEmptyPlan` optimi...

2016-06-25 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/13906 [SPARK-16208][SQL] Add `CollapseEmptyPlan` optimizer ## What changes were proposed in this pull request? This PR adds a new logical optimizer, `CollapseEmptyPlan`, to collapse

[GitHub] spark pull request #13906: [SPARK-16208][SQL] Add `CollapseEmptyPlan` optimi...

2016-06-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13906#discussion_r68495438 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -1053,6 +1055,34 @@ object PruneFilters extends

[GitHub] spark pull request #13905: [SPARK-16208][SQL] Add `CollapseEmptyPlan` optimi...

2016-06-25 Thread dongjoon-hyun
Github user dongjoon-hyun closed the pull request at: https://github.com/apache/spark/pull/13905 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #13906: [SPARK-16208][SQL] Add `CollapseEmptyPlan` optimi...

2016-06-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13906#discussion_r68495422 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -1053,6 +1055,34 @@ object PruneFilters extends

[GitHub] spark issue #13913: [SPARK-10591][SQL][TEST] Add a testcase to ensure if `ch...

2016-06-27 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13913 Thank you for merging, @liancheng ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #13765: [SPARK-16052][SQL] Improve `CollapseRepartition` optimiz...

2016-06-27 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13765 Hi, @cloud-fan . Now, this PR can handle all combinations of Repartition and RepartitionBy. I updated the description of PR and JIRA, too. Thank you so much for making this PR much

[GitHub] spark issue #13918: [SPARK-16221][SQL] Redirect Parquet JUL logger via SLF4J...

2016-06-27 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13918 Hi, @liancheng . Now, it passes the Jenkins again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #13915: [SPARK-16081][BUILD] Disallow using `l` as variable name

2016-06-27 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13915 Hi, @rxin and @srowen . I also worry about merge conflicts. It's really annoying for committers. So, what about the stepwise approach? We have 2395 files and found 90 files

[GitHub] spark issue #13918: [SPARK-16221][SQL] Redirect Parquet JUL logger via SLF4J...

2016-06-27 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13918 Thank you, @liancheng ! I fixed it. By the way, you already finished all before. :) I think some updates on ParquetWriter seem to modify after that. --- If your project is set up

[GitHub] spark issue #13730: [SPARK-16006][SQL] Attemping to write empty DataFrame wi...

2016-06-27 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13730 Ping @tdas --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #13765: [SPARK-16052][SQL] Add `CollapseRepartitionBy` op...

2016-06-27 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13765#discussion_r68610179 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -547,6 +548,16 @@ object CollapseRepartition

[GitHub] spark issue #13918: [SPARK-16221][SQL] Redirect Parquet JUL logger via SLF4J...

2016-06-27 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13918 Thank you for merging, @liancheng ! :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #13914: [SPARK-16111][SQL][DOC] Hide SparkOrcNewRecordReader in ...

2016-06-27 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13914 Thank you for merging, @rxin . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #13906: [SPARK-16208][SQL] Add `CollapseEmptyPlan` optimi...

2016-06-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13906#discussion_r68701793 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/CollapseEmptyPlanSuite.scala --- @@ -0,0 +1,133

[GitHub] spark pull request #13930: [SPARK-16228][SQL] HiveSessionCatalog should retu...

2016-06-27 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13930#discussion_r68700034 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala --- @@ -174,6 +175,18 @@ private[sql] class HiveSessionCatalog

[GitHub] spark pull request #13930: [SPARK-16228][SQL] HiveSessionCatalog should retu...

2016-06-27 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13930#discussion_r68700984 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala --- @@ -174,6 +175,18 @@ private[sql] class HiveSessionCatalog

[GitHub] spark pull request #13930: [SPARK-16228][SQL] HiveSessionCatalog should retu...

2016-06-27 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13930#discussion_r68697699 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala --- @@ -174,6 +175,18 @@ private[sql] class HiveSessionCatalog

[GitHub] spark issue #13906: [SPARK-16208][SQL] Add `CollapseEmptyPlan` optimizer

2016-06-27 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13906 Hi, @rxin . I just remembered this PR while looking your whitelist PR. :) Any advice for this PR? --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request #13930: [SPARK-16228][SQL] HiveSessionCatalog should retu...

2016-06-27 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13930#discussion_r68700695 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala --- @@ -174,6 +175,18 @@ private[sql] class HiveSessionCatalog

[GitHub] spark pull request #13930: [SPARK-16228][SQL] HiveSessionCatalog should retu...

2016-06-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13930#discussion_r68702737 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala --- @@ -174,6 +175,18 @@ private[sql] class HiveSessionCatalog

[GitHub] spark issue #13915: [SPARK-16081][BUILD] Disallow using `l` as variable name

2016-06-27 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13915 @mengxr 's idea sounds good to me, too. May I update this PR, @rxin ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request #13930: [SPARK-16228][SQL] HiveSessionCatalog should retu...

2016-06-27 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13930#discussion_r68700193 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala --- @@ -174,6 +175,18 @@ private[sql] class HiveSessionCatalog

[GitHub] spark pull request #13906: [SPARK-16208][SQL] Add `CollapseEmptyPlan` optimi...

2016-06-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13906#discussion_r68703430 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/CollapseEmptyPlanSuite.scala --- @@ -0,0 +1,133

[GitHub] spark pull request #13930: [SPARK-16228][SQL] HiveSessionCatalog should retu...

2016-06-27 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13930#discussion_r68700636 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala --- @@ -174,6 +175,18 @@ private[sql] class HiveSessionCatalog

[GitHub] spark pull request #13939: [SPARK-16248][SQL] Whitelist the list of Hive fal...

2016-06-27 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13939#discussion_r68701105 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala --- @@ -221,4 +214,18 @@ private[sql] class HiveSessionCatalog

[GitHub] spark pull request #13906: [SPARK-16208][SQL] Add `CollapseEmptyPlan` optimi...

2016-06-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13906#discussion_r68701978 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -1053,6 +1055,41 @@ object PruneFilters extends

[GitHub] spark issue #13906: [SPARK-16208][SQL] Add `CollapseEmptyPlan` optimizer

2016-06-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13906 Anyway, thank you for review again, @rxin ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request #13930: [SPARK-16228][SQL] Support a fallback lookup for ...

2016-06-27 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/13930 [SPARK-16228][SQL] Support a fallback lookup for external functions with `double`-type parameter only ## What changes were proposed in this pull request? This PR supports a fallback

[GitHub] spark issue #13914: [SPARK-16111][SQL][DOC] Hide SparkOrcNewRecordReader in ...

2016-06-27 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13914 Hi, @mengxr . Could you review this PR when you have some time? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request #13930: [SPARK-16228][SQL] Support a fallback lookup for ...

2016-06-27 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13930#discussion_r68648985 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -881,7 +881,16 @@ class Analyzer

[GitHub] spark pull request #13930: [SPARK-16228][SQL] Support a fallback lookup for ...

2016-06-27 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13930#discussion_r68649440 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -881,7 +881,16 @@ class Analyzer

[GitHub] spark pull request #13914: [SPARK-16111][DOC] Hide SparkOrcNewRecordReader i...

2016-06-26 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/13914 [SPARK-16111][DOC] Hide SparkOrcNewRecordReader in API docs ## What changes were proposed in this pull request? This PR hides `SparkOrcNewRecordReader` from API docs. ## How

[GitHub] spark issue #13914: [SPARK-16111][DOC] Hide SparkOrcNewRecordReader in API d...

2016-06-26 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13914 Hi, @mengxr . Could you review this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request #13914: [SPARK-16111][DOC] Hide SparkOrcNewRecordReader i...

2016-06-26 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13914#discussion_r68502088 --- Diff: project/SparkBuild.scala --- @@ -733,7 +734,8 @@ object Unidoc { unidocSourceBase := s"https://github.com/apache/spark/t

[GitHub] spark issue #13913: [SPARK-10591][TEST] Add a testcase to ensure if `checkAn...

2016-06-26 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13913 Hi, @liancheng . When I check [SPARK-10591](https://issues.apache.org/jira/browse/SPARK-10591) today, it is handled by `Row.equals` correctly. I just make this PR for ensuring

[GitHub] spark pull request #13914: [SPARK-16111][DOC] Hide SparkOrcNewRecordReader i...

2016-06-26 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13914#discussion_r68502066 --- Diff: project/SparkBuild.scala --- @@ -720,6 +720,7 @@ object Unidoc { // Skip class names containing $ and some internal packages

[GitHub] spark pull request #13913: [SPARK-10591][TEST] Add a testcase to ensure if `...

2016-06-26 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/13913 [SPARK-10591][TEST] Add a testcase to ensure if `checkAnswer` handles map correctly ## What changes were proposed in this pull request? This PR adds a testcase to ensure

[GitHub] spark issue #13900: [SPARK-16173][SQL] Can't join describe() of DataFrame in...

2016-06-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13900 cc @davies --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #13900: [SPARK-16173][SQL] Can't join describe() of DataF...

2016-06-24 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/13900 [SPARK-16173][SQL] Can't join describe() of DataFrame in Scala 2.10 ## What changes were proposed in this pull request? This PR fixes `DataFrame.describe()` by forcing

[GitHub] spark issue #13900: [SPARK-16173][SQL] Can't join describe() of DataFrame in...

2016-06-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13900 @davies . I added the comment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #13887: [SPARK-16186][SQL] Support partition batch pruning with ...

2016-06-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13887 Okay. Let's summarize before updating PR. 1. In general, `a IN (expression)` will be `a = expression`. `OptimizerIn` optimizer will take care of this. 2. In general, `a IN (2001, 2002

[GitHub] spark issue #13887: [SPARK-16186][SQL] Support partition batch pruning with ...

2016-06-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13887 Anyway, thank you in advance, @davies . :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #13887: [SPARK-16186][SQL] Support partition batch pruning with ...

2016-06-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13887 Oh, you meant adding additional constraints by using **min** and **max**. I see. By the way, I have one question. If there are many predicates, does Spark use the predicate

[GitHub] spark issue #13887: [SPARK-16186][SQL] Support partition batch pruning with ...

2016-06-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13887 Oh, right. It's pending Jenkins. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #13887: [SPARK-16186][SQL] Support partition batch pruning with ...

2016-06-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13887 The some of frequent TPC-DS usages were STATE, ZIP, Color strings. The min/max of these values doesn't have much meaning. --- If your project is set up for it, you can reply to this email

[GitHub] spark issue #13887: [SPARK-16186][SQL] Support partition batch pruning with ...

2016-06-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13887 Hi, @davies . I removes the option-related stuff from the code/PR description/JIRA description according to your advice. Thank you for review! --- If your project is set up

[GitHub] spark issue #13887: [SPARK-16186][SQL] Support partition batch pruning with ...

2016-06-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13887 Maybe, you are confused with https://github.com/apache/spark/pull/13900 . It passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request #13900: [SPARK-16173][SQL] Can't join describe() of DataF...

2016-06-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13900#discussion_r68469199 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -1908,7 +1908,7 @@ class Dataset[T] private[sql]( // All columns

[GitHub] spark issue #13900: [SPARK-16173][SQL] Can't join describe() of DataFrame in...

2016-06-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13900 Thank you, @davies ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #13887: [SPARK-16186][SQL] Support partition batch pruning with ...

2016-06-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13887 I'm not sure, but it's just my hope. :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #13887: [SPARK-16186][SQL] Support partition batch pruning with ...

2016-06-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13887 Hmm. The general idea is good. But, I still think this PR and the idea seem to be complementary to each other. Sorry, but, if possible, can we proceed that general idea in another PR

[GitHub] spark pull request #13887: [SPARK-16186][SQL] Support partition batch prunin...

2016-06-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13887#discussion_r68475255 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala --- @@ -79,6 +79,11 @@ private[sql] case class

[GitHub] spark issue #13900: [SPARK-16173][SQL] Can't join describe() of DataFrame in...

2016-06-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13900 Thank you for merging. Sure. I will make a patch for 1.6. Should I make a patch for 1.5, too? --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request #13915: [SPARK-16081] Disallow using `l` as variable name

2016-06-26 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/13915 [SPARK-16081] Disallow using `l` as variable name ## What changes were proposed in this pull request? This PR adds a ScalaStyle custom rule, `DisallowMisleadingVariableName

[GitHub] spark pull request #13902: [SPARK-16173] [SQL] Can't join describe() of Data...

2016-06-24 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/13902 [SPARK-16173] [SQL] Can't join describe() of DataFrame in Scala 2.10 ## What changes were proposed in this pull request? This PR fixes `DataFrame.describe()` by forcing

[GitHub] spark issue #13902: [SPARK-16173] [SQL] Can't join describe() of DataFrame i...

2016-06-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13902 cc @davies . This is a PR for branch 1.6. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #13887: [SPARK-16186][SQL] Support partition batch pruning with ...

2016-06-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13887 Hi, @davies . Now, it passed. If there is anything for me to do, please let me know. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark issue #13902: [SPARK-16173] [SQL] Can't join describe() of DataFrame i...

2016-06-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13902 Thank you, @davies ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #13900: [SPARK-16173][SQL] Can't join describe() of DataFrame in...

2016-06-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13900 Oh, great! I didn't notice that is mergeable for 1.5 branch. Thank you! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark issue #13876: [SPARK-16174][SQL] Improve OptimizeIn optimizer to remov...

2016-06-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13876 Hi, @rxin . For this `OptimizeIn` PR, please let me know if we need further optimization. Thank you always. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark issue #13915: [SPARK-16081][BUILD] Disallow using `l` as variable name

2016-06-26 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13915 Thank you, @HyukjinKwon . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #13902: [SPARK-16173] [SQL] Can't join describe() of Data...

2016-06-25 Thread dongjoon-hyun
Github user dongjoon-hyun closed the pull request at: https://github.com/apache/spark/pull/13902 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #13902: [SPARK-16173] [SQL] Can't join describe() of DataFrame i...

2016-06-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13902 Oh, sure. I forgot that it is not close automatically. Thank you for pining me, @rxin . --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request #13905: [SPARK-16208][SQL] Add `CollapseEmptyPlan` optimi...

2016-06-25 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/13905 [SPARK-16208][SQL] Add `CollapseEmptyPlan` optimizer ## What changes were proposed in this pull request? This PR adds a new logical optimizer, `CollapseEmptyPlan`, to collapse

[GitHub] spark pull request #13918: [SPARK-16221][SQL] Redirect Parquet JUL logger vi...

2016-06-26 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13918#discussion_r68522055 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala --- @@ -915,15 +917,14 @@ private[sql

[GitHub] spark pull request #13918: [SPARK-16221][SQL] Redirect Parquet JUL logger vi...

2016-06-26 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/13918 [SPARK-16221][SQL] Redirect Parquet JUL logger via SLF4J for WRITE operations ## What changes were proposed in this pull request? [SPARK-8118](https://github.com/apache/spark/pull

[GitHub] spark issue #13765: [SPARK-16052][SQL] Add `CollapseRepartitionBy` optimizer

2016-06-26 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13765 Hi, @cloud-fan . Could you review this `CollapseRepartitionBy` optimizer when you have some time? --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark issue #13915: [SPARK-16081][BUILD] Disallow using `l` as variable name

2016-06-26 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13915 cc @rxin and @srowen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #13930: [SPARK-16228][SQL] Support a fallback lookup for externa...

2016-06-27 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13930 Hi, @hvanhovell . I updated this PR according to your comments. Definitely, this issue was only about `HiveSessionCatalog`. Thank you! --- If your project is set up for it, you

[GitHub] spark issue #13876: [SPARK-16174][SQL] Improve OptimizeIn optimizer to remov...

2016-06-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13876 Hi, @rxin . Now, variable `l` is replaced with `list`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #13918: [SPARK-16221][SQL] Redirect Parquet JUL logger via SLF4J...

2016-06-27 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13918 Hi, @liancheng . Could you review this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #13721: [SPARK-16005][R] Add `randomSplit` to SparkR

2016-06-16 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13721 Hi, @shivaram . The followings are updated and become ready for review again. - The param description is improved. - The size and ratio of returned list is compared with those

[GitHub] spark pull request #13730: [SPARK-16006][SQL] Attemping to write empty DataF...

2016-06-16 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/13730 [SPARK-16006][SQL] Attemping to write empty DataFrame with no fields throws non-intuitive exception ## What changes were proposed in this pull request? This PR fixes the error

[GitHub] spark pull request #13730: [SPARK-16006][SQL] Attemping to write empty DataF...

2016-06-16 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13730#discussion_r67462488 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala --- @@ -339,6 +339,9 @@ private[sql] object

[GitHub] spark issue #13730: [SPARK-16006][SQL] Attemping to write empty DataFrame wi...

2016-06-16 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13730 Hi, @tdas . This is the PR to care the reported corner case. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request #13684: [SPARK-15908][R] Add varargs-type dropDuplicates(...

2016-06-15 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13684#discussion_r67280096 --- Diff: R/pkg/R/DataFrame.R --- @@ -1869,14 +1871,22 @@ setMethod("where", #' path <- "path/to/file.json"

[GitHub] spark issue #13684: [SPARK-15908][R] Add varargs-type dropDuplicates() funct...

2016-06-15 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13684 Thank you, @sun-rui . Now, this PR checks all parameters' type correctly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request #13734: [SPARK-14995][R] Add `since` tag in Roxygen docum...

2016-06-17 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/13734 [SPARK-14995][R] Add `since` tag in Roxygen documentation for SparkR API methods ## What changes were proposed in this pull request? This PR adds `since` tags to Roxygen

[GitHub] spark issue #13734: [SPARK-14995][R] Add `since` tag in Roxygen documentatio...

2016-06-17 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13734 Hi, @shivaram , @felixcheung , @sun-rui . It's the first draft. There is a little ambiguity like the following. - `SparkDataFrame` is marked as `@note since 2.0.0` because

[GitHub] spark issue #13730: [SPARK-16006][SQL] Attemping to write empty DataFrame wi...

2016-06-17 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13730 Hi, @tdas . At first look, I thought this corner case throws exceptions. But, after considering more carefully, I want to allow `emptyDataFrame.write`. That is more natural way

[GitHub] spark pull request #13967: [SPARK-16278][SPARK-16279][SQL] Implement map_key...

2016-06-29 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/13967 [SPARK-16278][SPARK-16279][SQL] Implement map_keys/map_values SQL functions ## What changes were proposed in this pull request? This PR adds `map_keys` and `map_values` SQL functions

[GitHub] spark issue #13967: [SPARK-16278][SPARK-16279][SQL] Implement map_keys/map_v...

2016-06-29 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13967 cc @rxin and @cloud-fan . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #13956: [SPARK-16267][TEST] Replace deprecated `CREATE TEMPORARY...

2016-06-29 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13956 Could you review this PR, @srowen ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-11381][DOCS] Replace example code in ml...

2016-02-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/11320#issuecomment-187953829 Test build 51808 is running now. Let's see the result. :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [SPARK-11381][DOCS] Replace example code in ml...

2016-02-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/11320#issuecomment-187953113 I've done! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-11381][DOCS] Replace example code in ml...

2016-02-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/11320#issuecomment-187959828 According to Jenkins, other PRs also suffer from this. I think `retriggering` is not helpful at this time. --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-11381][DOCS] Replace example code in ml...

2016-02-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/11320#issuecomment-187958660 Hmm, it fails again due to Github. ``` ERROR: Timeout after 15 minutes ERROR: Error fetching remote repo 'origin' ``` --- If your project is set

[GitHub] spark pull request: [SPARK-11381][DOCS] Replace example code in ml...

2016-02-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/11320#issuecomment-187821324 Thank you for reviewing, @noprom . This PR is the similar to #11053 (merged yesterday.) Hi, @yinxusen, @mengxr . Could you review this PR

[GitHub] spark pull request: [SPARK-11381][DOCS] Replace example code in ml...

2016-02-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/11320#issuecomment-188002229 Finally! Now, it's ready to be reviewed again. :) Thank you, all. --- If your project is set up for it, you can reply to this email and have your reply

<    1   2   3   4   5   6   7   8   9   10   >