[GitHub] spark pull request: [SPARK-4233] [SQL] WIP:Simplify the UDAF API (...

2015-04-05 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/3247#discussion_r27782127 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregates.scala --- @@ -17,285 +17,159 @@ package

[GitHub] spark pull request: [SPARK-6521][Core]executors in the same node r...

2015-04-05 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/5178#discussion_r27781922 --- Diff: core/src/main/scala/org/apache/spark/shuffle/FileShuffleBlockManager.scala --- @@ -180,7 +180,8 @@ class FileShuffleBlockManager(conf: SparkConf

[GitHub] spark pull request: [SPARK-6747][SQL] Support List as a return t...

2015-04-07 Thread maropu
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/5395 [SPARK-6747][SQL] Support List as a return type in Hive UDF This patch supports List as a return type in Hive UDF. We assume an UDF below; public class UDFToListString extends UDF

[GitHub] spark pull request: [SPARK-6747][SQL] Support List as a return t...

2015-04-07 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/5395#discussion_r27938425 --- Diff: sql/hive/src/test/java/org/apache/spark/sql/hive/execution/UDFToListString.java --- @@ -0,0 +1,29 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-6747][SQL] Support List as a return t...

2015-04-07 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/5395#issuecomment-90784532 Ok, I will look into the implementation and the documentation of Hive for that. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [SPARK-6521][Core]executors in the same node r...

2015-04-05 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/5178#issuecomment-89735823 Understood. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-6734] [SQL] Add UDTF.close support in G...

2015-04-08 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/5383#discussion_r27995414 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/Generate.scala --- @@ -74,10 +84,15 @@ case class Generate( } else

[GitHub] spark pull request: [SPARK-6734] [SQL] Add UDTF.close support in G...

2015-04-09 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/5383#issuecomment-91172886 I found one issue; the current implementation of HiveGenericUdtf always calls `terminate()` though, it does not call `initialize()` in some cases because of lazy

[GitHub] spark pull request: [SPARK-4950] Delete obsolete mapReduceTripelet...

2015-06-19 Thread maropu
Github user maropu closed the pull request at: https://github.com/apache/spark/pull/3782 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-5623][GraphX] Replace an obsolete mapRe...

2015-06-23 Thread maropu
Github user maropu closed the pull request at: https://github.com/apache/spark/pull/4402 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-6747] [SQL] Support List as a return ...

2015-05-26 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/6179#discussion_r31096485 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUdfSuite.scala --- @@ -133,6 +134,41 @@ class HiveUdfSuite extends QueryTest

[GitHub] spark pull request: [SPARK-6747] [SQL] Support List as a return ...

2015-07-06 Thread maropu
Github user maropu closed the pull request at: https://github.com/apache/spark/pull/6179 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-6747] [SQL] Throw an AnalysisException ...

2015-07-06 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/7248#issuecomment-119040198 @marmbrus Ok and thanks. After this patch merged, I'll make a same patch for Map because it has the same issue. --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-6747] [SQL] Throw an AnalysisException ...

2015-07-06 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/7248#issuecomment-119036207 @marmbrus Through the discussion of #5395, I think it is hard to support java List types in SparkSQL because of type erasure. ISTM that if udf developers use this type

[GitHub] spark pull request: [SPARK-6747] [SQL] Throw an AnalysisException ...

2015-07-06 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/7248#discussion_r34000400 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala --- @@ -133,6 +133,32 @@ class HiveUDFSuite extends QueryTest

[GitHub] spark pull request: [SPARK-6747] [SQL] Throw an AnalysisException ...

2015-07-06 Thread maropu
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/7248 [SPARK-6747] [SQL] Throw an AnalysisException when unsupported Java list types used in Hive UDF The current implementation can't handle List as a return type in Hive UDF and throws meaningless

[GitHub] spark pull request: [SPARK-6747] [SQL] Support List as a return ...

2015-05-21 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/6179#issuecomment-104236074 @marmbrus please merge it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-6912][SQL] Throw an AnalysisException w...

2015-07-07 Thread maropu
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/7257 [SPARK-6912][SQL] Throw an AnalysisException when unsupported Java MapK,V types used in Hive UDF To make UDF developers understood, throw an exception when unsupported MapK,V types used in Hive UDF

[GitHub] spark pull request: [SPARK-6912][SQL] Throw an AnalysisException w...

2015-07-07 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/7257#issuecomment-119110618 @marmbrus plz review it, thx. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-9867][SQL] Move utilities for binary da...

2015-08-20 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/8122#issuecomment-132992396 cc: @rxin it's a just remainder. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-9360][SQL] Support BinaryType in Prefix...

2015-08-03 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/7676#issuecomment-127459623 thanks, I'll fix it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-9360][SQL] Support BinaryType in Prefix...

2015-08-03 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/7676#issuecomment-127482382 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-9360][SQL] Support BinaryType in Prefix...

2015-08-04 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/7676#issuecomment-127638905 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-8930][SQL] Support a star '*' in genera...

2015-08-04 Thread maropu
Github user maropu closed the pull request at: https://github.com/apache/spark/pull/7305 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-9361][SQL] Refactor new aggregation cod...

2015-07-29 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/7677#discussion_r35736333 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/utils.scala --- @@ -0,0 +1,167 @@ +/* + * Licensed

[GitHub] spark pull request: [SPARK-9361][SQL] Refactor new aggregation cod...

2015-07-29 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/7677#issuecomment-125878935 This pr is reasonable to me though, what's actual performance differences when this patch applied? --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: [SPARK-7460] [SQL] Support DataFrame.zip to me...

2015-07-29 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/7474#discussion_r35733516 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala --- @@ -1365,6 +1365,20 @@ class DataFrame private[sql]( def foreachPartition(f

[GitHub] spark pull request: [SPARK-7460] [SQL] Support DataFrame.zip to me...

2015-07-29 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/7474#discussion_r35733198 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala --- @@ -1365,6 +1365,20 @@ class DataFrame private[sql]( def foreachPartition(f

[GitHub] spark pull request: [SPARK-8263][SQL] substr/substring should also...

2015-07-29 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/7641#discussion_r35731817 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringOperations.scala --- @@ -699,8 +732,12 @@ case class Substring(str

[GitHub] spark pull request: [SPARK-8263][SQL] substr/substring should also...

2015-07-29 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/7641#issuecomment-125856142 ISTM that `Concat` also needs to support binary types according to (Hive)[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF]. So, how about

[GitHub] spark pull request: [SPARK-9370][SQL] Add DecimalType support in U...

2015-07-29 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/7744#discussion_r35745491 --- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java --- @@ -92,7 +92,8 @@ public static int

[GitHub] spark pull request: [SPARK-9370][SQL] Add DecimalType support in U...

2015-07-29 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/7744#issuecomment-125905838 We must change these getter/setter interfaces? ISTM that `setDecimal(i: ordinal, value: Decimal)` serializes the value with the precision/scale that the input decimal

[GitHub] spark pull request: [SPARK-9360][SQL] Support BinaryType in Prefix...

2015-08-05 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/7676#discussion_r36275502 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SortOrder.scala --- @@ -76,6 +78,7 @@ case class SortPrefix(child: SortOrder

[GitHub] spark pull request: [SPARK-9360][SQL] Support BinaryType in Prefix...

2015-08-05 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/7676#discussion_r36275517 --- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/PrefixComparators.java --- @@ -52,6 +59,38 @@ public int compare(long bPrefix, long

[GitHub] spark pull request: [SPARK-9360][SQL] Support BinaryType in Prefix...

2015-08-05 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/7676#issuecomment-127898753 @davies @rxin ok, all the comments applied. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-9360][SQL] Support BinaryType in Prefix...

2015-08-04 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/7676#discussion_r36260211 --- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/PrefixComparators.java --- @@ -52,6 +59,38 @@ public int compare(long bPrefix, long

[GitHub] spark pull request: [SPARK-9360][SQL] Support BinaryType in Prefix...

2015-08-04 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/7676#issuecomment-127815853 @rxin The tests failed though, ISTM the ``hive-thriftserver`` tests are not related to this pr. This failure is correct, or not? --- If your project is set up

[GitHub] spark pull request: [SPARK-8263][SQL] substr/substring should also...

2015-07-30 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/7641#discussion_r35841474 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringOperations.scala --- @@ -699,8 +732,12 @@ case class Substring(str

[GitHub] spark pull request: [SPARK-9816][SQL] Support BinaryType in Concat

2015-08-11 Thread maropu
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/8098 [SPARK-9816][SQL] Support BinaryType in Concat Support BinaryType in catalyst Concat according to hive behaviours. https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF

[GitHub] spark pull request: [SPARK-9816][SQL] Support BinaryType in Concat

2015-08-11 Thread maropu
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/8099 [SPARK-9816][SQL] Support BinaryType in Concat Support BinaryType in catalyst Concat according to hive behaviours. https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF

[GitHub] spark pull request: [SPARK-9816][SQL] Support BinaryType in Concat

2015-08-11 Thread maropu
Github user maropu closed the pull request at: https://github.com/apache/spark/pull/8098 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-7542][SQL][WIP] Support off-heap buffer...

2015-08-10 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/8068#discussion_r36711245 --- Diff: core/src/main/java/org/apache/spark/util/collection/TimSort.java --- @@ -914,7 +915,7 @@ private void mergeHi(int base1, int len1, int base2, int

[GitHub] spark pull request: [SPARK-9867][SQL] Move utilities for binary da...

2015-08-13 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/8122#discussion_r36993560 --- Diff: unsafe/src/main/java/org/apache/spark/unsafe/types/ByteArray.java --- @@ -29,4 +31,45 @@ public static void writeToMemory(byte[] src, Object

[GitHub] spark pull request: [SPARK-9867][SQL] Move utilities for binary da...

2015-08-14 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/8122#issuecomment-131117757 If no problem, could you merge this? cc: @rxin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-9816][SQL] Support BinaryType in Concat

2015-08-11 Thread maropu
Github user maropu closed the pull request at: https://github.com/apache/spark/pull/8099 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-9867][SQL] Move utilities for binary da...

2015-08-12 Thread maropu
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/8122 [SPARK-9867][SQL] Move utilities for binary data into ByteArray The utilities such as Substring#substringBinarySQL and BinaryPrefixComparator#computePrefix for binary data are put together

[GitHub] spark pull request: [SPARK-9867][SQL] Move utilities for binary da...

2015-08-12 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/8122#issuecomment-130251509 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-9867][SQL] Move utilities for binary da...

2015-08-12 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/8122#issuecomment-130300499 @rxin Could you review this? The unit tests failed though, ISTM these failures also happen in master. --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-8930][SQL] Support a star '*' in genera...

2015-07-24 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/7305#issuecomment-124343855 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-9360][SQL] Support BinaryType in Prefix...

2015-07-26 Thread maropu
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/7676 [SPARK-9360][SQL] Support BinaryType in PrefixComparators for UnsafeExternalSort The current implementation of UnsafeExternalSort uses NoOpPrefixComparator for binary-typed data. So, we need

[GitHub] spark pull request: [SPARK-8867][SQL][WIP] Support list / describe...

2015-07-24 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/7259#issuecomment-124571393 This patch's very useful for users :)) I leave some comments; I think we can add a function name together in ExpressionDescription according to Hive `Description

[GitHub] spark pull request: [SPARK-8867][SQL] Support list / describe func...

2015-07-26 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/7259#issuecomment-125067194 Ok, thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [Spark-8668][SQL][WIP] Adding expr to function...

2015-07-22 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/7606#discussion_r35290606 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -1720,4 +1720,11 @@ object functions { UnresolvedFunction(udfName

[GitHub] spark pull request: [SPARK-9020][SQL] Support mutable state in cod...

2015-07-14 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/7392#discussion_r34590505 --- Diff: core/src/main/scala/org/apache/spark/TaskContext.scala --- @@ -32,7 +32,13 @@ object TaskContext { */ def get(): TaskContext

[GitHub] spark pull request: [SPARK-9020][SQL] Support mutable state in cod...

2015-07-14 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/7392#discussion_r34591393 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -56,6 +56,16 @@ class CodeGenContext

[GitHub] spark pull request: [SPARK-8955][SQL] Replace a duplicated initial...

2015-07-13 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/7324#discussion_r34524246 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUDFs.scala --- @@ -501,7 +501,15 @@ private[hive] case class HiveGenericUDTF( protected

[GitHub] spark pull request: [SPARK-8955][SQL] Replace a duplicated initial...

2015-07-13 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/7324#issuecomment-121097682 @marmbrus Could you check and merge this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK 5280] RDF Loader added + documentation

2015-07-13 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/4650#issuecomment-121105412 @emir-munoz @blankdots This PR is totally stale, so it'd better to refactor ithis if you're interested in. Also, ISTM this kind of loader extensions should

[GitHub] spark pull request: [SPARK-5352][GraphX] Add getPartitionStrategy ...

2015-07-21 Thread maropu
Github user maropu closed the pull request at: https://github.com/apache/spark/pull/5549 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-6281][GraphX] Support incremental updat...

2015-07-21 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/5067#issuecomment-123231061 Ok, I'll close it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-6281][GraphX] Support incremental updat...

2015-07-21 Thread maropu
Github user maropu closed the pull request at: https://github.com/apache/spark/pull/5067 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-5450][GraphX] Add APIs to save a graph ...

2015-07-21 Thread maropu
Github user maropu closed the pull request at: https://github.com/apache/spark/pull/4244 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-8930][SQL] Support a star '*' in genera...

2015-07-15 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/7305#issuecomment-121659767 I understood that Hive ``explode`` only has a single expression though, should we apply the same limitation into ``UserDefinedGenerator`` used in the DataFrame#explode

[GitHub] spark pull request: [SPARK-8955][SQL] Replace a duplicated initial...

2015-07-15 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/7324#issuecomment-121606700 @marmbrus This is just a reminder. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-9058][SQL] Split projectionCode if it i...

2015-07-15 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/7418#issuecomment-121654917 This patch seems to be duplicated with #7076. Why you make a new PR? --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request: [SPARK-9058][SQL] Split projectionCode if it i...

2015-07-15 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/7418#discussion_r34692458 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateMutableProjection.scala --- @@ -45,7 +45,28 @@ object

[GitHub] spark pull request: [SPARK-8955][SQL] Replace a duplicated initial...

2015-07-15 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/7324#issuecomment-121645978 Ok, I'll close this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-8955][SQL] Replace a duplicated initial...

2015-07-15 Thread maropu
Github user maropu closed the pull request at: https://github.com/apache/spark/pull/7324 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-2827][GraphX] Add collectDegreeDist to ...

2015-10-29 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/4399#issuecomment-152342718 Ok, I'll fix it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-10186][SQL] Array types using JDBCRDD a...

2015-10-28 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/9137#discussion_r43347874 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala --- @@ -171,21 +187,9 @@ object JdbcUtils extends Logging

[GitHub] spark pull request: [SPARK-10186][SQL] Array types using JDBCRDD a...

2015-10-28 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/9137#discussion_r43347707 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala --- @@ -121,6 +122,21 @@ object JdbcUtils extends Logging

[GitHub] spark pull request: [SPARK-10186][SQL] Array types using JDBCRDD a...

2015-10-28 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/9137#issuecomment-152073314 Great work! I left some review comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-10186][SQL] Array types using JDBCRDD a...

2015-10-28 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/9137#discussion_r43348162 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala --- @@ -207,6 +225,25 @@ case object PostgresDialect extends JdbcDialect

[GitHub] spark pull request: [SPARK-10186][SQL] Array types using JDBCRDD a...

2015-10-28 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/9137#discussion_r43347951 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala --- @@ -72,7 +72,7 @@ abstract class JdbcDialect { * or null

[GitHub] spark pull request: [SPARK-11394][SQL] Throw IllegalArgumentExcept...

2015-10-29 Thread maropu
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/9350 [SPARK-11394][SQL] Throw IllegalArgumentException for unsupported types in postgresql If DataFrame has BYTE types, throws an exception: org.postgresql.util.PSQLException: ERROR: type "byte&

[GitHub] spark pull request: [SPARK-10101] [SQL] Add maxlength to JDBC fiel...

2015-10-29 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/8374#discussion_r43359470 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala --- @@ -278,3 +285,59 @@ case object MsSqlServerDialect extends JdbcDialect

[GitHub] spark pull request: [SPARK-10101] [SQL] Add maxlength to JDBC fiel...

2015-10-29 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/8374#issuecomment-152111324 Great work! I left some trivial comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-10186][SQL] Array types using JDBCRDD a...

2015-10-29 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/9137#discussion_r43472350 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala --- @@ -207,6 +225,25 @@ case object PostgresDialect extends JdbcDialect

[GitHub] spark pull request: [SPARK-6521][Core] Bypass unnecessary network ...

2015-11-04 Thread maropu
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/9478 [SPARK-6521][Core] Bypass unnecessary network access if block managers share an identical host Refactored #5178 and added unit tests. You can merge this pull request into a Git repository

[GitHub] spark pull request: [SPARK-10101] [SQL] Add maxlength to JDBC fiel...

2015-11-03 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/8374#discussion_r43833399 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala --- @@ -82,6 +82,14 @@ abstract class JdbcDialect { def getJDBCType(dt

[GitHub] spark pull request: [SPARK-6521][Core] Bypass unnecessary network ...

2015-11-05 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/9478#issuecomment-154231492 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-6521][Core] Bypass unnecessary network ...

2015-11-05 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/9478#issuecomment-154231203 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-6521][Core] Bypass unnecessary network ...

2015-11-05 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/9478#issuecomment-154270938 @andrewor14 Could you review this and give some suggestions? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [SPARK-2827][GraphX] Add collectDegreeDist to ...

2015-10-29 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/4399#issuecomment-152381853 @ankurdave @andrewor14 Fixed and could you merge this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [SPARK-8955][SQL] Replace a duplicated initial...

2015-07-09 Thread maropu
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/7324 [SPARK-8955][SQL] Replace a duplicated initialize() in HiveGenericUDTF with new one HiveGenericUDTF#initialize(ObjectInspector[] argOIs) in v0.13.1 is duplicated, so it needs to be replaced

[GitHub] spark pull request: [SPARK-8443][SQL] Split GenerateMutableProject...

2015-07-10 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/7076#issuecomment-120247243 This fix is kind of hack things to me. It'd be better to check the code size and, if it is over 64KB (the janino limitation), throw an exception to fall back

[GitHub] spark pull request: [SPARK-8443][SQL] Split GenerateMutableProject...

2015-07-11 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/7076#issuecomment-120623834 Yes, and falling back into normal expressions turns off unsafe optimization. I feel concerned that this fix is less meaningful for most users because

[GitHub] spark pull request: [SPARK-8930][SQL] Support a star '*' in genera...

2015-07-08 Thread maropu
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/7305 [SPARK-8930][SQL] Support a star '*' in generator function arguments The current implementation throws an exception if generators contain a star '*' like codes blow; val df = Seq((1, 1,2), (2, 4

[GitHub] spark pull request: [SPARK-8930][SQL] Support a star '*' in genera...

2015-07-12 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/7305#issuecomment-120815118 @chenghao-intel So I'll fix this PR to throw an AnalysisException if Generate has '*'. Is it ok? --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: [SPARK-8930][SQL] Support a star '*' in genera...

2015-07-12 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/7305#issuecomment-120816055 @chenghao-intel Ok, thanks. Is the limitation only applied to ``Explode`` and I mean that can other generator functions have multiple expressions? Anyway, I'll

[GitHub] spark pull request: [SPARK-10259] [ML] Add @since annotation to ml...

2015-12-03 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/8534#issuecomment-161548934 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-12392] Optimize a location order of bro...

2015-12-16 Thread maropu
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/10346 [SPARK-12392] Optimize a location order of broadcast blocks by considering preferred local hosts When multiple workers exist in a host, we can bypass unnecessary remote access for broadcasts

[GitHub] spark pull request: [SPARK-2827][GraphX] Add collectDegreeDist to ...

2015-12-16 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/4399#issuecomment-165357980 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-12392] Optimize a location order of bro...

2015-12-16 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/10346#issuecomment-165349453 I did quick benchmarks for large broadcasts; - aws m4.x4large x 4, 4 works in a host - elapsed time: -- w/opt.: 6.887943434s, w/o opt.: 11.738593435s

[GitHub] spark pull request: [SPARK-5882][GraphX] Add a test for GraphLoade...

2015-12-16 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/4674#issuecomment-165358131 @andrewor14 @ankurdave Fixed. Also, could you merge #4399? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [SPARK-5882][GraphX] Add a test for GraphLoade...

2015-12-16 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/4674#issuecomment-165354761 @andrewor14 okay --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-12392][Core] Optimize a location order ...

2015-12-17 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/10346#issuecomment-165705964 @andrewor14 Could you review this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...

2016-01-04 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-168651319 @yuhai Oh... my bad :(( Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...

2016-01-04 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-168618638 @yuhai Could you review it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-2827][GraphX] Add collectDegreeDist to ...

2016-01-04 Thread maropu
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/10521#issuecomment-168618986 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

<    1   2   3   4   5   6   7   8   9   10   >