Re: Change nullable property in Dataset schema

2016-08-17 Thread Kazuaki Ishizaki
branches. For example, in the above URL, we can say the condition at line 45 is always false since the result of map() is never null by using our schema. As a result, we can eliminate assignments at lines 52 and 56, and conditional branches at lines 55 and 61. Kazuaki Ishizaki From: Koert Kuipers

[jira] [Commented] (SPARK-15285) Generated SpecificSafeProjection.apply method grows beyond 64 KB

2016-08-16 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15422757#comment-15422757 ] Kazuaki Ishizaki commented on SPARK-15285: -- This is because the original pull request breaks

[jira] [Reopened] (SPARK-15285) Generated SpecificSafeProjection.apply method grows beyond 64 KB

2016-08-16 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki reopened SPARK-15285: -- This problem can be reproduced at commit id 7de30d6e9e5d3020d2ba8c2ce08893d9cd822b56

Re: Spark 2.0.0 JaninoRuntimeException

2016-08-16 Thread Kazuaki Ishizaki
I just realized it since it broken a build with Scala 2.10. https://github.com/apache/spark/commit/fa244e5a90690d6a31be50f2aa203ae1a2e9a1cf I can reproduce the problem in SPARK-15285 with master branch. Should we reopen SPARK-15285? Best Regards, Kazuaki Ishizaki, From: Ted Yu <yuz

Re: Change nullable property in Dataset schema

2016-08-10 Thread Kazuaki Ishizaki
alse), nullable = false))) .as(newDoubleArrayEncoder) ds1.printSchema ds2.printSchema } } root |-- value: array (nullable = true) ||-- element: integer (containsNull = false) root |-- value: array (nullable = false) ||-- element: integer (containsNull = false) K

[jira] [Commented] (SPARK-16913) [SQL] Better codegen where querying nested struct

2016-08-07 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15410840#comment-15410840 ] Kazuaki Ishizaki commented on SPARK-16913: -- It seems to copy each elements in a struct. Since

[jira] [Created] (SPARK-16902) Custom ExpressionEncoder for primitive array is not effective

2016-08-04 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-16902: Summary: Custom ExpressionEncoder for primitive array is not effective Key: SPARK-16902 URL: https://issues.apache.org/jira/browse/SPARK-16902 Project: Spark

Change nullable property in Dataset schema

2016-08-03 Thread Kazuaki Ishizaki
a ds2.printSchema } } root |-- value: array (nullable = true) ||-- element: integer (containsNull = false) root |-- value: array (nullable = true) // Expected (nullable = false) ||-- element: integer (containsNull = false) Kazuaki Ishizaki

Re: Spark GraphFrames

2016-08-02 Thread Kazuaki Ishizaki
Sorry Please ignore this mail. Sorry for misinterpretation of GraphFrame in Spark. I thought that Frame Graph for profiling tool. Kazuaki Ishizaki, From: Kazuaki Ishizaki/Japan/IBM@IBMJP To: Divya Gehlot <divya.htco...@gmail.com> Cc: "user @spark" <user@spa

Re: Spark GraphFrames

2016-08-02 Thread Kazuaki Ishizaki
Hi, Kay wrote a procedure to use GraphFrames with Spark. https://gist.github.com/kayousterhout/7008a8ebf2babeedc7ce6f8723fd1bf4 Kazuaki Ishizaki From: Divya Gehlot <divya.htco...@gmail.com> To: "user @spark" <user@spark.apache.org> Date: 2016/08/02 14:52

[jira] [Commented] (SPARK-16807) Optimize some ABS() statements

2016-07-30 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400532#comment-15400532 ] Kazuaki Ishizaki commented on SPARK-16807: -- Interesting if we can ensure {{x - y

[jira] [Updated] (SPARK-16223) Codegen failure with a Dataframe program using an array

2016-07-14 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-16223: - Description: When we compile a Dataframe program with an operation to large array

[jira] [Comment Edited] (SPARK-15687) Columnar execution engine

2016-07-14 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376999#comment-15376999 ] Kazuaki Ishizaki edited comment on SPARK-15687 at 7/14/16 2:25 PM

[jira] [Commented] (SPARK-15687) Columnar execution engine

2016-07-14 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376999#comment-15376999 ] Kazuaki Ishizaki commented on SPARK-15687: -- It would be good to introduce trait for {{iterator[T

[jira] [Commented] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

2016-07-10 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369703#comment-15369703 ] Kazuaki Ishizaki commented on SPARK-15467: -- [Janino 3.0.0|https://mvnrepository.com/artifact

[jira] [Updated] (SPARK-16412) Generate Java code that gets an array in each column of CachedBatch when DataFrame.cache() is called

2016-07-07 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-16412: - Issue Type: Improvement (was: Bug) > Generate Java code that gets an array in e

[jira] [Created] (SPARK-16412) Generate Java code that gets an array in each column of CachedBatch when DataFrame.cache() is called

2016-07-07 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-16412: Summary: Generate Java code that gets an array in each column of CachedBatch when DataFrame.cache() is called Key: SPARK-16412 URL: https://issues.apache.org/jira/browse

[jira] [Commented] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

2016-07-03 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360752#comment-15360752 ] Kazuaki Ishizaki commented on SPARK-15467: -- My [PR|https://github.com/janino-compiler/janino

[jira] [Created] (SPARK-16223) Codegen failure with a Dataframe program using an array

2016-06-27 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-16223: Summary: Codegen failure with a Dataframe program using an array Key: SPARK-16223 URL: https://issues.apache.org/jira/browse/SPARK-16223 Project: Spark

[jira] [Commented] (SPARK-16070) DataFrame/Parquet issues with primitive arrays

2016-06-26 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15350005#comment-15350005 ] Kazuaki Ishizaki commented on SPARK-16070: -- I added two JIRA entries, which address DataFrame

[jira] [Created] (SPARK-16215) Reduce runtime overhead of a program that writes an primitive array in Dataframe/Dataset

2016-06-25 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-16215: Summary: Reduce runtime overhead of a program that writes an primitive array in Dataframe/Dataset Key: SPARK-16215 URL: https://issues.apache.org/jira/browse/SPARK-16215

[jira] [Created] (SPARK-16213) Reduce runtime overhead of a program that creates an primitive array in DataFrame

2016-06-25 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-16213: Summary: Reduce runtime overhead of a program that creates an primitive array in DataFrame Key: SPARK-16213 URL: https://issues.apache.org/jira/browse/SPARK-16213

[jira] [Commented] (SPARK-15899) file scheme should be used correctly

2016-06-22 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345620#comment-15345620 ] Kazuaki Ishizaki commented on SPARK-15899: -- I think so. As [~sowen] proposed, we may need

[jira] [Comment Edited] (SPARK-16070) DataFrame/Parquet issues with primitive arrays

2016-06-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341427#comment-15341427 ] Kazuaki Ishizaki edited comment on SPARK-16070 at 6/21/16 9:22 AM

[jira] [Comment Edited] (SPARK-16070) DataFrame/Parquet issues with primitive arrays

2016-06-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341408#comment-15341408 ] Kazuaki Ishizaki edited comment on SPARK-16070 at 6/21/16 9:22 AM

[jira] [Commented] (SPARK-16070) DataFrame/Parquet issues with primitive arrays

2016-06-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341427#comment-15341427 ] Kazuaki Ishizaki commented on SPARK-16070: -- Other JIRAs for DataFrame issues with primitive

[jira] [Commented] (SPARK-16070) DataFrame/Parquet issues with primitive arrays

2016-06-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341408#comment-15341408 ] Kazuaki Ishizaki commented on SPARK-16070: -- [~mengxr], thank you for creating an umbrella. I

[jira] [Commented] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

2016-06-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341376#comment-15341376 ] Kazuaki Ishizaki commented on SPARK-15467: -- Thank you for letting me know it. Now, it looks

[jira] [Created] (SPARK-16043) Prepare GenericArrayData implementation specialized for a primitive array

2016-06-18 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-16043: Summary: Prepare GenericArrayData implementation specialized for a primitive array Key: SPARK-16043 URL: https://issues.apache.org/jira/browse/SPARK-16043

[jira] [Created] (SPARK-16042) Eliminate nullcheck code at projection for an array type

2016-06-18 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-16042: Summary: Eliminate nullcheck code at projection for an array type Key: SPARK-16042 URL: https://issues.apache.org/jira/browse/SPARK-16042 Project: Spark

Question about equality of o.a.s.sql.Row

2016-06-17 Thread Kazuaki Ishizaki
equality of two Rows that have an Array or complex types (e.g. Map)? Best Regards, Kazuaki Ishizaki, @kiszk

[jira] [Commented] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

2016-06-16 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15334443#comment-15334443 ] Kazuaki Ishizaki commented on SPARK-15467: -- We are waiting for author's review at https

[jira] [Created] (SPARK-15985) Reduce runtime overhead of a program that reads an primitive array in Dataset

2016-06-16 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-15985: Summary: Reduce runtime overhead of a program that reads an primitive array in Dataset Key: SPARK-15985 URL: https://issues.apache.org/jira/browse/SPARK-15985

[jira] [Created] (SPARK-15962) Introduce additonal implementation with a dense format for UnsafeArrayData

2016-06-15 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-15962: Summary: Introduce additonal implementation with a dense format for UnsafeArrayData Key: SPARK-15962 URL: https://issues.apache.org/jira/browse/SPARK-15962

[jira] [Created] (SPARK-15950) Eliminate unreachable code at projection for complex types

2016-06-14 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-15950: Summary: Eliminate unreachable code at projection for complex types Key: SPARK-15950 URL: https://issues.apache.org/jira/browse/SPARK-15950 Project: Spark

[jira] [Issue Comment Deleted] (SPARK-15899) file scheme should be used correctly

2016-06-13 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-15899: - Comment: was deleted (was: When I added the two extra slashes, it works on Linux

[jira] [Commented] (SPARK-15899) file scheme should be used correctly

2016-06-13 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327825#comment-15327825 ] Kazuaki Ishizaki commented on SPARK-15899: -- When I added the two extra slashes, it works

[jira] [Commented] (SPARK-15899) file scheme should be used correctly

2016-06-13 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327824#comment-15327824 ] Kazuaki Ishizaki commented on SPARK-15899: -- When I added the two extra slashes, it works

[jira] [Commented] (SPARK-15899) file scheme should be used correctly

2016-06-12 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15326537#comment-15326537 ] Kazuaki Ishizaki commented on SPARK-15899: -- Thank you for your comments. It is a little bit

[jira] [Created] (SPARK-15899) file scheme should be used correctly

2016-06-11 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-15899: Summary: file scheme should be used correctly Key: SPARK-15899 URL: https://issues.apache.org/jira/browse/SPARK-15899 Project: Spark Issue Type

Re: Catalyst optimizer cpu/Io cost

2016-06-10 Thread Kazuaki Ishizaki
Hi Yin Huai's slide is avaiable at http://www.slideshare.net/databricks/deep-dive-into-catalyst-apache-spark-20s-optimizer Kazuaki Ishizaki From: Takeshi Yamamuro <linguin@gmail.com> To: Srinivasan Hariharan02 <srinivasan_...@infosys.com> Cc: "user@spark.

[jira] [Comment Edited] (SPARK-15687) Columnar execution engine

2016-06-06 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316858#comment-15316858 ] Kazuaki Ishizaki edited comment on SPARK-15687 at 6/6/16 5:49 PM: -- Thank

[jira] [Commented] (SPARK-15687) Columnar execution engine

2016-06-06 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316858#comment-15316858 ] Kazuaki Ishizaki commented on SPARK-15687: -- Thank you for your answers * How we pass columnar

[jira] [Commented] (SPARK-15687) Columnar execution engine

2016-06-01 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15310663#comment-15310663 ] Kazuaki Ishizaki commented on SPARK-15687: -- Thank you for creating interesting JIRA entry. Based

[jira] [Commented] (SPARK-15258) Nested/Chained case statements generate codegen over 64k exception

2016-05-24 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15297896#comment-15297896 ] Kazuaki Ishizaki commented on SPARK-15258: -- {{commit d642b273544bb77ef7f584326aa2d214649ac61b

[jira] [Commented] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

2016-05-24 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15297784#comment-15297784 ] Kazuaki Ishizaki commented on SPARK-15467: -- I submitted [a pull request|https://github.com

[jira] [Comment Edited] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

2016-05-23 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15296743#comment-15296743 ] Kazuaki Ishizaki edited comment on SPARK-15467 at 5/24/16 3:54 AM

[jira] [Commented] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

2016-05-23 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15296929#comment-15296929 ] Kazuaki Ishizaki commented on SPARK-15467: -- I created test cases to reproduce this problem

[jira] [Commented] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

2016-05-23 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15296743#comment-15296743 ] Kazuaki Ishizaki commented on SPARK-15467: -- The following code can reproduce the same issue

[jira] [Commented] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

2016-05-23 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15296102#comment-15296102 ] Kazuaki Ishizaki commented on SPARK-15467: -- I will try to submit a patch against Janino

[jira] [Commented] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

2016-05-22 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295499#comment-15295499 ] Kazuaki Ishizaki commented on SPARK-15467: -- IMHO, this issue is due to Janino's implementation

[jira] [Commented] (SPARK-15258) Nested/Chained case statements generate codegen over 64k exception

2016-05-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295402#comment-15295402 ] Kazuaki Ishizaki commented on SPARK-15258: -- This PR is not for SPARK-15258. This is for SPARK

[jira] [Comment Edited] (SPARK-15285) Generated SpecificSafeProjection.apply method grows beyond 64 KB

2016-05-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295299#comment-15295299 ] Kazuaki Ishizaki edited comment on SPARK-15285 at 5/21/16 11:33 PM: I

[jira] [Comment Edited] (SPARK-15285) Generated SpecificSafeProjection.apply method grows beyond 64 KB

2016-05-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295299#comment-15295299 ] Kazuaki Ishizaki edited comment on SPARK-15285 at 5/21/16 11:12 PM: I

[jira] [Commented] (SPARK-15285) Generated SpecificSafeProjection.apply method grows beyond 64 KB

2016-05-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295299#comment-15295299 ] Kazuaki Ishizaki commented on SPARK-15285: -- I created a (PR)[https://github.com/apache/spark

[jira] [Commented] (SPARK-15285) Generated SpecificSafeProjection.apply method grows beyond 64 KB

2016-05-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295014#comment-15295014 ] Kazuaki Ishizaki commented on SPARK-15285: -- I see, I started doing this. > Genera

[jira] [Commented] (SPARK-15285) Generated SpecificSafeProjection.apply method grows beyond 64 KB

2016-05-20 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294494#comment-15294494 ] Kazuaki Ishizaki commented on SPARK-15285: -- I can take it today if they are busy. > Genera

[jira] [Created] (SPARK-15380) Generate code that stores a float/double value in each column from ColumnarBatch when DataFrame.cache() is used

2016-05-18 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-15380: Summary: Generate code that stores a float/double value in each column from ColumnarBatch when DataFrame.cache() is used Key: SPARK-15380 URL: https://issues.apache.org

[jira] [Commented] (SPARK-14082) Add support for GPU resource when running on Mesos

2016-05-12 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281371#comment-15281371 ] Kazuaki Ishizaki commented on SPARK-14082: -- Sounds great. We expect that our [GPUEnabler|https

[jira] [Updated] (SPARK-15117) Generate code that get a value in each compressed column from CachedBatch when DataFrame.cache() is called

2016-05-04 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-15117: - Target Version/s: 2.1.0 > Generate code that get a value in each compressed column f

[jira] [Created] (SPARK-15117) Generate code that get a value in each compressed column from CachedBatch when DataFrame.cache() is called

2016-05-04 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-15117: Summary: Generate code that get a value in each compressed column from CachedBatch when DataFrame.cache() is called Key: SPARK-15117 URL: https://issues.apache.org/jira

[jira] [Updated] (SPARK-14098) Generate code that get a float/double value in each column from CachedBatch when DataFrame.cache() is called

2016-05-04 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-14098: - Description: When DataFrame.cache() is called, data is stored as column-oriented storage

[jira] [Updated] (SPARK-14098) Generate code that get a float/double value in each column from CachedBatch when DataFrame.cache() is called

2016-05-04 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-14098: - Summary: Generate code that get a float/double value in each column from CachedBatch

[jira] [Updated] (SPARK-14098) Generate code that get a value in each column from CachedBatch when DataFrame.cache() is called

2016-04-25 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-14098: - Target Version/s: 2.0.0 > Generate code that get a value in each column from CachedBa

[jira] [Commented] (SPARK-14138) Generated SpecificColumnarIterator code can exceed JVM size limit for cached DataFrames

2016-04-22 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254061#comment-15254061 ] Kazuaki Ishizaki commented on SPARK-14138: -- Yes, it will be included in 1.6.2 and 2.0.0

[jira] [Commented] (SPARK-13904) Add support for pluggable cluster manager

2016-04-19 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15247498#comment-15247498 ] Kazuaki Ishizaki commented on SPARK-13904: -- I agree with you since SPARK-14689 addresses. >

[jira] [Commented] (SPARK-13904) Add support for pluggable cluster manager

2016-04-17 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15244805#comment-15244805 ] Kazuaki Ishizaki commented on SPARK-13904: -- To merge this PR may have begun causing test

[jira] [Issue Comment Deleted] (SPARK-14690) [SQL] SPARK-8020 fails in Jenkins for master

2016-04-17 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-14690: - Comment: was deleted (was: Add a link to the original JIRA) > [SQL] SPARK-8020 fa

[jira] [Closed] (SPARK-14690) [SQL] SPARK-8020 fails in Jenkins for master

2016-04-17 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki closed SPARK-14690. Add a link to the original JIRA > [SQL] SPARK-8020 fails in Jenkins for mas

[jira] [Commented] (SPARK-14690) [SQL] SPARK-8020 fails in Jenkins for master

2016-04-17 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15244798#comment-15244798 ] Kazuaki Ishizaki commented on SPARK-14690: -- I see. I will reopen the original JIRA soon. > [

[jira] [Resolved] (SPARK-14690) [SQL] SPARK-8020 fails in Jenkins for master

2016-04-17 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki resolved SPARK-14690. -- Resolution: Duplicate > [SQL] SPARK-8020 fails in Jenkins for mas

Recent Jenkins always fails in specific two tests

2016-04-17 Thread Kazuaki Ishizaki
sues.apache.org/jira/browse/SPARK-14689 https://issues.apache.org/jira/browse/SPARK-14690 Best regards, Kazuaki Ishizaki

[jira] [Updated] (SPARK-14690) [SQL] SPARK-8020 fails in Jenkins for master

2016-04-17 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-14690: - Summary: [SQL] SPARK-8020 fails in Jenkins for master (was: [SQL] SPARK-9757 fails

[jira] [Created] (SPARK-14690) [SQL] SPARK-9757 fails in Jenkins for master

2016-04-17 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-14690: Summary: [SQL] SPARK-9757 fails in Jenkins for master Key: SPARK-14690 URL: https://issues.apache.org/jira/browse/SPARK-14690 Project: Spark Issue

[jira] [Created] (SPARK-14689) [SQL] SPARK-9757 fails in Jenkins for master

2016-04-17 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-14689: Summary: [SQL] SPARK-9757 fails in Jenkins for master Key: SPARK-14689 URL: https://issues.apache.org/jira/browse/SPARK-14689 Project: Spark Issue

[jira] [Commented] (SPARK-14656) Benchmark.getPorcessorName() always return "Unknown processor" on Linux

2016-04-15 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242688#comment-15242688 ] Kazuaki Ishizaki commented on SPARK-14656: -- The root cause is described at https://github.com

[jira] [Created] (SPARK-14656) Benchmark.getPorcessorName() always return "Unknown processor" on Linux

2016-04-15 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-14656: Summary: Benchmark.getPorcessorName() always return "Unknown processor" on Linux Key: SPARK-14656 URL: https://issues.apache.org/jira/browse/S

[jira] [Commented] (SPARK-14083) Analyze JVM bytecode and turn closures into Catalyst expressions

2016-04-01 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15222634#comment-15222634 ] Kazuaki Ishizaki commented on SPARK-14083: -- Good point. On the other hand, should we follow JVM

[jira] [Commented] (SPARK-14083) Analyze JVM bytecode and turn closures into Catalyst expressions

2016-03-31 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15220055#comment-15220055 ] Kazuaki Ishizaki commented on SPARK-14083: -- Thank you. Makes sense to me. It seems

[jira] [Commented] (SPARK-14083) Analyze JVM bytecode and turn closures into Catalyst expressions

2016-03-31 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219637#comment-15219637 ] Kazuaki Ishizaki commented on SPARK-14083: -- Looks interesting. I am also interested

[jira] [Commented] (SPARK-14083) Analyze JVM bytecode and turn closures into Catalyst expressions

2016-03-28 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15213959#comment-15213959 ] Kazuaki Ishizaki commented on SPARK-14083: -- Another possible bytecode analysis library is WALA

[jira] [Commented] (SPARK-14138) Generated SpecificColumnarIterator code can exceed JVM size limit for cached DataFrames

2016-03-26 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15213071#comment-15213071 ] Kazuaki Ishizaki commented on SPARK-14138: -- I will make a PR that includes two solutions: 1

[jira] [Comment Edited] (SPARK-14098) Generate code that get a value in each column from CachedBatch when DataFrame.cache() is called

2016-03-23 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209047#comment-15209047 ] Kazuaki Ishizaki edited comment on SPARK-14098 at 3/23/16 8:02 PM

[jira] [Commented] (SPARK-14098) Generate code that get a value in each column from CachedBatch when DataFrame.cache() is called

2016-03-23 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209047#comment-15209047 ] Kazuaki Ishizaki commented on SPARK-14098: -- After fixing some trivial compilation failures, I

[jira] [Created] (SPARK-14098) Generate code that get a value in each column from CachedBatch when DataFrame.cache() is called

2016-03-23 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-14098: Summary: Generate code that get a value in each column from CachedBatch when DataFrame.cache() is called Key: SPARK-14098 URL: https://issues.apache.org/jira/browse/SPARK

[jira] [Created] (SPARK-14072) Show JVM information when we run Benchmark

2016-03-22 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-14072: Summary: Show JVM information when we run Benchmark Key: SPARK-14072 URL: https://issues.apache.org/jira/browse/SPARK-14072 Project: Spark Issue

[jira] [Created] (SPARK-13844) Generate better code for filters with a non-nullable column

2016-03-12 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-13844: Summary: Generate better code for filters with a non-nullable column Key: SPARK-13844 URL: https://issues.apache.org/jira/browse/SPARK-13844 Project: Spark

[jira] [Created] (SPARK-13805) Direct consume ColumnVector in generated code when ColumnarBatch is used

2016-03-10 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-13805: Summary: Direct consume ColumnVector in generated code when ColumnarBatch is used Key: SPARK-13805 URL: https://issues.apache.org/jira/browse/SPARK-13805

[jira] [Updated] (SPARK-13644) Add the source file name and line into Logger when an exception occurs in the generated code

2016-03-03 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-13644: - Summary: Add the source file name and line into Logger when an exception occurs

[jira] [Updated] (SPARK-13644) add the source file name and line into Logger when an exception occurs in the generated code

2016-03-03 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-13644: - Issue Type: Improvement (was: Bug) Summary: add the source file name and line

[jira] [Created] (SPARK-13644) [SQL] add the source file name and line into Logger when an exception occurs in the generated code

2016-03-03 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-13644: Summary: [SQL] add the source file name and line into Logger when an exception occurs in the generated code Key: SPARK-13644 URL: https://issues.apache.org/jira/browse

[jira] [Commented] (SPARK-13530) Add ShortType support to UnsafeRowParquetRecordReader

2016-02-27 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15170636#comment-15170636 ] Kazuaki Ishizaki commented on SPARK-13530: -- I saw the same exception in my environment. Thank

[jira] [Closed] (SPARK-12907) Use BitSet to represent null fields in ColumnVector

2016-02-23 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki closed SPARK-12907. Resolution: Won't Fix > Use BitSet to represent null fields in ColumnVec

[jira] [Commented] (SPARK-13421) Make output of a SparkPlan configurable

2016-02-23 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15160157#comment-15160157 ] Kazuaki Ishizaki commented on SPARK-13421: -- I am willing to work for and contribute

[jira] [Commented] (SPARK-13431) Maven build fails due to: Method code too large! in Catalyst

2016-02-23 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159649#comment-15159649 ] Kazuaki Ishizaki commented on SPARK-13431: -- I identified why this problem occurs only in maven

[jira] [Created] (SPARK-13437) Add InternalColumn

2016-02-22 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-13437: Summary: Add InternalColumn Key: SPARK-13437 URL: https://issues.apache.org/jira/browse/SPARK-13437 Project: Spark Issue Type: Improvement

[jira] [Comment Edited] (SPARK-13431) Maven build fails due to: Method code too large! in Catalyst

2016-02-22 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15157297#comment-15157297 ] Kazuaki Ishizaki edited comment on SPARK-13431 at 2/22/16 5:17 PM

[jira] [Commented] (SPARK-13431) Maven build fails due to: Method code too large! in Catalyst

2016-02-22 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15157297#comment-15157297 ] Kazuaki Ishizaki commented on SPARK-13431: -- The size of a static initializer method

[jira] [Commented] (SPARK-13431) Maven build fails due to: Method code too large! in Catalyst

2016-02-22 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15157208#comment-15157208 ] Kazuaki Ishizaki commented on SPARK-13431: -- I am using mvn, and executed the following command

[jira] [Commented] (SPARK-13431) Maven build fails due to: Method code too large! in Catalyst

2016-02-22 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15157182#comment-15157182 ] Kazuaki Ishizaki commented on SPARK-13431: -- I succeeded to build Spark by commenting out lines

<    6   7   8   9   10   11   12   >