[jira] [Updated] (SPARK-23763) OffHeapColumnVector uses MemoryBlock

2018-04-05 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-23763: - Issue Type: Sub-task (was: Improvement) Parent: SPARK-23879

[jira] [Updated] (SPARK-23779) TaskMemoryManager and UnsafeSorter use MemoryBlock

2018-04-05 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-23779: - Issue Type: Sub-task (was: Improvement) Parent: SPARK-23879 > TaskMemoryMana

[jira] [Updated] (SPARK-23879) Introduce MemoryBlock API instead of Platform API with Object

2018-04-05 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-23879: - Component/s: Spark Core > Introduce MemoryBlock API instead of Platform API with Obj

[jira] [Created] (SPARK-23879) Introduce MemoryBlock API instead of Platform API with Object

2018-04-05 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-23879: Summary: Introduce MemoryBlock API instead of Platform API with Object Key: SPARK-23879 URL: https://issues.apache.org/jira/browse/SPARK-23879 Project

[jira] [Commented] (SPARK-23801) Consistent SIGSEGV after upgrading to Spark v2.3.0

2018-04-02 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16422955#comment-16422955 ] Kazuaki Ishizaki commented on SPARK-23801: -- If you use WebUI, you can easily see SQL plans

Re: Welcome Zhenhua Wang as a Spark committer

2018-04-01 Thread Kazuaki Ishizaki
Congratulations to Zhenhua! Kazuaki Ishizaki From: sujith chacko <sujithchacko.2...@gmail.com> To: Denny Lee <denny.g@gmail.com> Cc: Spark dev list <dev@spark.apache.org>, Wenchen Fan <cloud0...@gmail.com>, "叶先进" <advance...@gmail.com> Dat

[jira] [Commented] (SPARK-23801) Consistent SIGSEGV after upgrading to Spark v2.3.0

2018-03-27 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416835#comment-16416835 ] Kazuaki Ishizaki commented on SPARK-23801: -- While we know it is time-consuming work to narrow

[jira] [Created] (SPARK-23779) TaskMemoryManager and UnsafeSorter use MemoryBlock

2018-03-23 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-23779: Summary: TaskMemoryManager and UnsafeSorter use MemoryBlock Key: SPARK-23779 URL: https://issues.apache.org/jira/browse/SPARK-23779 Project: Spark

[jira] [Updated] (SPARK-23763) OffHeapColumnVector uses MemoryBlock

2018-03-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-23763: - Description: This JIRA entry tries to use {{MemoryBlock}} in {{OffHeapColumnVector

[jira] [Updated] (SPARK-23763) BufferHolder uses MemoryBlock

2018-03-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-23763: - Description: This JIRA entry tries to use {{MemoryBlock}} in {{BufferHolder

[jira] [Created] (SPARK-23763) UTF8StringBuilder uses MemoryBlock

2018-03-21 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-23763: Summary: UTF8StringBuilder uses MemoryBlock Key: SPARK-23763 URL: https://issues.apache.org/jira/browse/SPARK-23763 Project: Spark Issue Type

[jira] [Created] (SPARK-23762) UTF8StringBuilder uses MemoryBlock

2018-03-21 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-23762: Summary: UTF8StringBuilder uses MemoryBlock Key: SPARK-23762 URL: https://issues.apache.org/jira/browse/SPARK-23762 Project: Spark Issue Type

[jira] [Comment Edited] (SPARK-23713) Clean-up UnsafeWriter classes

2018-03-16 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403246#comment-16403246 ] Kazuaki Ishizaki edited comment on SPARK-23713 at 3/17/18 3:42 AM

[jira] [Commented] (SPARK-23713) Clean-up UnsafeWriter classes

2018-03-16 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403246#comment-16403246 ] Kazuaki Ishizaki commented on SPARK-23713: -- It will make sense. I will work for this. > Cl

[jira] [Commented] (SPARK-23673) PySpark dayofweek does not conform with ISO 8601

2018-03-15 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401392#comment-16401392 ] Kazuaki Ishizaki commented on SPARK-23673: -- In Spark, `dayofweek` comes from SQL. The result

[jira] [Updated] (SPARK-23656) Assertion in XXH64Suite is not performed on big endian platform

2018-03-12 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-23656: - Description: {{XXH64Suite.testKnownByteArrayInputs()}} performs assertions only

[jira] [Updated] (SPARK-23656) Assertion in XXH64Suite.testKnownByteArrayInputs() is not performed on big endian platform

2018-03-12 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-23656: - Summary: Assertion in XXH64Suite.testKnownByteArrayInputs() is not performed on big

[jira] [Created] (SPARK-23656) Assertion in XXH64Suite is not performed on big endian platform

2018-03-12 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-23656: Summary: Assertion in XXH64Suite is not performed on big endian platform Key: SPARK-23656 URL: https://issues.apache.org/jira/browse/SPARK-23656 Project

[jira] [Commented] (SPARK-23605) Conflicting dependencies for janino in 2.3.0

2018-03-09 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16394035#comment-16394035 ] Kazuaki Ishizaki commented on SPARK-23605: -- Thank you for providing information. If I am correct

[jira] [Comment Edited] (SPARK-23605) Conflicting dependencies for janino in 2.3.0

2018-03-09 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393793#comment-16393793 ] Kazuaki Ishizaki edited comment on SPARK-23605 at 3/9/18 11:30 PM

[jira] [Commented] (SPARK-23605) Conflicting dependencies for janino in 2.3.0

2018-03-09 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393793#comment-16393793 ] Kazuaki Ishizaki commented on SPARK-23605: -- Would it be possible to post a program that can

[jira] [Commented] (SPARK-23610) Cast of ArrayType of NullType to ArrayType of nullable material type does not work

2018-03-09 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393173#comment-16393173 ] Kazuaki Ishizaki commented on SPARK-23610: -- The following code works without an exception using

[jira] [Updated] (SPARK-22246) UnsafeRow, UnsafeArrayData, and UnsafeMapData use MemoryBlock

2018-03-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22246: - Issue Type: Sub-task (was: Improvement) Parent: SPARK-10399 > Unsafe

[jira] [Comment Edited] (SPARK-23598) WholeStageCodegen can lead to IllegalAccessError calling append for HashAggregateExec

2018-03-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391834#comment-16391834 ] Kazuaki Ishizaki edited comment on SPARK-23598 at 3/9/18 3:20 AM

[jira] [Commented] (SPARK-23598) WholeStageCodegen can lead to IllegalAccessError calling append for HashAggregateExec

2018-03-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391834#comment-16391834 ] Kazuaki Ishizaki commented on SPARK-23598: -- Thanks, I confirmed that I can reproduce this issue

[jira] [Comment Edited] (SPARK-23549) Spark SQL unexpected behavior when comparing timestamp to date

2018-03-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391386#comment-16391386 ] Kazuaki Ishizaki edited comment on SPARK-23549 at 3/8/18 5:33 PM: -- I see

[jira] [Commented] (SPARK-23549) Spark SQL unexpected behavior when comparing timestamp to date

2018-03-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391386#comment-16391386 ] Kazuaki Ishizaki commented on SPARK-23549: -- I see. Make sense. It would be good to cast

[jira] [Commented] (SPARK-23549) Spark SQL unexpected behavior when comparing timestamp to date

2018-03-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391344#comment-16391344 ] Kazuaki Ishizaki commented on SPARK-23549: -- I think that this is a problem in Spark. My question

[jira] [Comment Edited] (SPARK-23598) WholeStageCodegen can lead to IllegalAccessError calling append for HashAggregateExec

2018-03-07 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389766#comment-16389766 ] Kazuaki Ishizaki edited comment on SPARK-23598 at 3/7/18 4:39 PM: -- I

[jira] [Commented] (SPARK-23598) WholeStageCodegen can lead to IllegalAccessError calling append for HashAggregateExec

2018-03-07 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389766#comment-16389766 ] Kazuaki Ishizaki commented on SPARK-23598: -- I also think that the easiest way is to make `append

[jira] [Commented] (SPARK-18492) GeneratedIterator grows beyond 64 KB

2018-03-07 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389761#comment-16389761 ] Kazuaki Ishizaki commented on SPARK-18492: -- [~imranshaik] I ran the following program

[jira] [Commented] (SPARK-23582) Add interpreted execution to StaticInvoke expression

2018-03-06 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388729#comment-16388729 ] Kazuaki Ishizaki commented on SPARK-23582: -- I see. Now, I have a prototype using old-school

[jira] [Commented] (SPARK-23582) Add interpreted execution to StaticInvoke expression

2018-03-05 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387230#comment-16387230 ] Kazuaki Ishizaki commented on SPARK-23582: -- I am working for this > Add interpreted execut

[jira] [Commented] (SPARK-23583) Add interpreted execution to Invoke expression

2018-03-05 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387231#comment-16387231 ] Kazuaki Ishizaki commented on SPARK-23583: -- I am working for this > Add interpreted execut

[jira] [Commented] (SPARK-23580) Interpreted mode fallback should be implemented for all expressions & projections

2018-03-05 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386991#comment-16386991 ] Kazuaki Ishizaki commented on SPARK-23580: -- Sure, I will work for them > Interpreted m

[jira] [Commented] (SPARK-23573) Create linter rule to prevent misuse of SparkContext.hadoopConfiguration in SQL modules

2018-03-04 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16385026#comment-16385026 ] Kazuaki Ishizaki commented on SPARK-23573: -- Sorry for my unintentional changes

[jira] [Updated] (SPARK-23573) Create linter rule to prevent misuse of SparkContext.hadoopConfiguration in SQL modules

2018-03-04 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-23573: - Description: SparkContext.hadoopConfiguration should not be used directly in SQL module

Re: Welcoming some new committers

2018-03-02 Thread Kazuaki Ishizaki
Congratulations to everyone! Kazuaki Ishizaki From: Takeshi Yamamuro <linguin@gmail.com> To: Spark dev list <dev@spark.apache.org> Date: 2018/03/03 10:45 Subject:Re: Welcoming some new committers Congrats, all! On Sat, Mar 3, 2018 at 10:34 AM, Takuya

[jira] [Comment Edited] (SPARK-18492) GeneratedIterator grows beyond 64 KB

2018-03-02 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16384428#comment-16384428 ] Kazuaki Ishizaki edited comment on SPARK-18492 at 3/3/18 2:08 AM

[jira] [Commented] (SPARK-18492) GeneratedIterator grows beyond 64 KB

2018-03-02 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16384428#comment-16384428 ] Kazuaki Ishizaki commented on SPARK-18492: -- Thanks, what is the definition of {{truetradescast

[jira] [Commented] (SPARK-18492) GeneratedIterator grows beyond 64 KB

2018-03-02 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383936#comment-16383936 ] Kazuaki Ishizaki commented on SPARK-18492: -- According to the screenshot, the number

[jira] [Commented] (SPARK-18492) GeneratedIterator grows beyond 64 KB

2018-03-02 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383933#comment-16383933 ] Kazuaki Ishizaki commented on SPARK-18492: -- Thank you for reporting the issue. We would

[jira] [Created] (SPARK-23546) Refactor non-stateful methods/values in CodegenContext

2018-03-01 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-23546: Summary: Refactor non-stateful methods/values in CodegenContext Key: SPARK-23546 URL: https://issues.apache.org/jira/browse/SPARK-23546 Project: Spark

[jira] [Commented] (SPARK-23427) spark.sql.autoBroadcastJoinThreshold causing OOM exception in the driver

2018-03-01 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381815#comment-16381815 ] Kazuaki Ishizaki commented on SPARK-23427: -- I am working for this (now trying to collect heap

[jira] [Commented] (SPARK-23498) Accuracy problem in comparison with string and integer

2018-02-28 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16380722#comment-16380722 ] Kazuaki Ishizaki commented on SPARK-23498: -- Is SPARK-21774 related to this issue, too

[jira] [Commented] (SPARK-23509) Upgrade commons-net from 2.2 to 3.1

2018-02-25 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376198#comment-16376198 ] Kazuaki Ishizaki commented on SPARK-23509: -- During the build using sbt, we are seeing

[jira] [Commented] (SPARK-23512) Complex operations on Dataframe corrupts data

2018-02-25 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376171#comment-16376171 ] Kazuaki Ishizaki commented on SPARK-23512: -- This problem can be reproduced with master using

[jira] [Commented] (SPARK-23498) Accuracy problem in comparison with string and integer

2018-02-25 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376137#comment-16376137 ] Kazuaki Ishizaki commented on SPARK-23498: -- SPARK implicitly casts {{String}} to type of another

[jira] [Commented] (SPARK-23459) Improve the error message when unknown column is specified in partition columns

2018-02-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16371474#comment-16371474 ] Kazuaki Ishizaki commented on SPARK-23459: -- I am working for this. > Improve the error mess

[jira] [Created] (SPARK-23466) Remove redundant null checks in generated Java code by GenerateUnsafeProjection

2018-02-19 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-23466: Summary: Remove redundant null checks in generated Java code by GenerateUnsafeProjection Key: SPARK-23466 URL: https://issues.apache.org/jira/browse/SPARK-23466

[jira] [Commented] (SPARK-23459) Improve the error message when unknown column is specified in partition columns

2018-02-19 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369346#comment-16369346 ] Kazuaki Ishizaki commented on SPARK-23459: -- Have you started working for this? If not, I can

[jira] [Comment Edited] (SPARK-23427) spark.sql.autoBroadcastJoinThreshold causing OOM exception in the driver

2018-02-18 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368823#comment-16368823 ] Kazuaki Ishizaki edited comment on SPARK-23427 at 2/19/18 7:01 AM: --- I

[jira] [Commented] (SPARK-23427) spark.sql.autoBroadcastJoinThreshold causing OOM exception in the driver

2018-02-18 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368823#comment-16368823 ] Kazuaki Ishizaki commented on SPARK-23427: -- I got the OOM with the same stack trace for both

[jira] [Comment Edited] (SPARK-23427) spark.sql.autoBroadcastJoinThreshold causing OOM exception in the driver

2018-02-18 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368728#comment-16368728 ] Kazuaki Ishizaki edited comment on SPARK-23427 at 2/19/18 12:49 AM

[jira] [Commented] (SPARK-23427) spark.sql.autoBroadcastJoinThreshold causing OOM exception in the driver

2018-02-18 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368728#comment-16368728 ] Kazuaki Ishizaki commented on SPARK-23427: -- Thank you. I ran this program several times

[jira] [Commented] (SPARK-23415) BufferHolderSparkSubmitSuite is flaky

2018-02-15 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365931#comment-16365931 ] Kazuaki Ishizaki commented on SPARK-23415: -- I realized that an issue in this test case. I think

[jira] [Comment Edited] (SPARK-23415) BufferHolderSparkSubmitSuite is flaky

2018-02-15 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365931#comment-16365931 ] Kazuaki Ishizaki edited comment on SPARK-23415 at 2/15/18 5:02 PM: --- I

[jira] [Commented] (SPARK-23427) spark.sql.autoBroadcastJoinThreshold causing OOM in the driver

2018-02-14 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365065#comment-16365065 ] Kazuaki Ishizaki commented on SPARK-23427: -- We would appreciate it if you could post a program

[jira] [Commented] (SPARK-23415) BufferHolderSparkSubmitSuite is flaky

2018-02-14 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364682#comment-16364682 ] Kazuaki Ishizaki commented on SPARK-23415: -- I am working

[jira] [Created] (SPARK-23424) Add codegenStageId in comment

2018-02-14 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-23424: Summary: Add codegenStageId in comment Key: SPARK-23424 URL: https://issues.apache.org/jira/browse/SPARK-23424 Project: Spark Issue Type

[jira] [Commented] (SPARK-23310) Perf regression introduced by SPARK-21113

2018-02-10 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359429#comment-16359429 ] Kazuaki Ishizaki commented on SPARK-23310: -- got it, thanks > Perf regression introdu

[jira] [Commented] (SPARK-23309) Spark 2.3 cached query performance 20-30% worse then spark 2.2

2018-02-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357441#comment-16357441 ] Kazuaki Ishizaki commented on SPARK-23309: -- When there is a repro, I am happy to investigate

[jira] [Commented] (SPARK-23310) Perf regression introduced by SPARK-21113

2018-02-05 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352754#comment-16352754 ] Kazuaki Ishizaki commented on SPARK-23310: -- [~juliuszsompolski] We would appreciate it if you

[jira] [Commented] (SPARK-18016) Code Generation: Constant Pool Past Limit for Wide/Nested Dataset

2018-02-01 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348980#comment-16348980 ] Kazuaki Ishizaki commented on SPARK-18016: -- This issue has been fixed in Spark 2.3 or master

[jira] [Resolved] (SPARK-20826) Support compression/decompression of ColumnVector in generated code

2018-01-31 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki resolved SPARK-20826. -- Resolution: Duplicate > Support compression/decompression of ColumnVector in genera

[jira] [Resolved] (SPARK-20823) Generate code to build table cache using ColumnarBatch and to get value from ColumnVector for other types

2018-01-31 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki resolved SPARK-20823. -- Resolution: Won't Fix > Generate code to build table cache using ColumnarBa

[jira] [Resolved] (SPARK-20821) Add compression/decompression of data to ColumnVector for other data types

2018-01-31 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki resolved SPARK-20821. -- Resolution: Won't Fix > Add compression/decompression of data to ColumnVec

[jira] [Resolved] (SPARK-20820) Add compression/decompression of data to ColumnVector for other compression schemes

2018-01-31 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki resolved SPARK-20820. -- Resolution: Duplicate > Add compression/decompression of data to ColumnVec

[jira] [Resolved] (SPARK-20824) Generate code to get value from table cache with wider column in ColumnarBatch

2018-01-31 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki resolved SPARK-20824. -- Resolution: Won't Fix > Generate code to get value from table cache with wider col

[jira] [Resolved] (SPARK-20825) Generate code to get value from table cache with wider column in ColumnarBatch for other data types

2018-01-31 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki resolved SPARK-20825. -- Resolution: Not A Problem > Generate code to get value from table cache with wi

[jira] [Resolved] (SPARK-22747) Shorten lifetime of global variables used in HashAggregateExec

2018-01-31 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki resolved SPARK-22747. -- Resolution: Won't Fix > Shorten lifetime of global variables used in HashAggregateE

[jira] [Resolved] (SPARK-22935) Dataset with Java Beans for java.sql.Date throws CompileException

2018-01-31 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki resolved SPARK-22935. -- Resolution: Invalid > Dataset with Java Beans for java.sql.Date thr

[jira] [Commented] (SPARK-18016) Code Generation: Constant Pool Past Limit for Wide/Nested Dataset

2018-01-31 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347160#comment-16347160 ] Kazuaki Ishizaki commented on SPARK-18016: -- [~gaurav.garg] I found the message {{18/01/30 07:31

[jira] [Commented] (SPARK-18016) Code Generation: Constant Pool Past Limit for Wide/Nested Dataset

2018-01-31 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347092#comment-16347092 ] Kazuaki Ishizaki commented on SPARK-18016: -- [~gaurav.garg] Thank you for attaching the log file

[jira] [Comment Edited] (SPARK-18016) Code Generation: Constant Pool Past Limit for Wide/Nested Dataset

2018-01-30 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345551#comment-16345551 ] Kazuaki Ishizaki edited comment on SPARK-18016 at 1/30/18 6:27 PM

[jira] [Commented] (SPARK-18016) Code Generation: Constant Pool Past Limit for Wide/Nested Dataset

2018-01-30 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345551#comment-16345551 ] Kazuaki Ishizaki commented on SPARK-18016: -- [~gaurav.garg] Thank you for your confirmation. I am

[jira] [Comment Edited] (SPARK-18016) Code Generation: Constant Pool Past Limit for Wide/Nested Dataset

2018-01-23 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336957#comment-16336957 ] Kazuaki Ishizaki edited comment on SPARK-18016 at 1/24/18 5:49 AM: --- I

[jira] [Commented] (SPARK-18016) Code Generation: Constant Pool Past Limit for Wide/Nested Dataset

2018-01-23 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336957#comment-16336957 ] Kazuaki Ishizaki commented on SPARK-18016: -- I cannot reproduce this issue using the following

[jira] [Commented] (SPARK-18016) Code Generation: Constant Pool Past Limit for Wide/Nested Dataset

2018-01-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333897#comment-16333897 ] Kazuaki Ishizaki commented on SPARK-18016: -- Thanks, I will look at this. > Code Generat

[jira] [Commented] (SPARK-18016) Code Generation: Constant Pool Past Limit for Wide/Nested Dataset

2018-01-20 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333298#comment-16333298 ] Kazuaki Ishizaki commented on SPARK-18016: -- Thank you for reporting an issue. Could you post

[jira] [Commented] (SPARK-22935) Dataset with Java Beans for java.sql.Date throws CompileException

2018-01-02 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308194#comment-16308194 ] Kazuaki Ishizaki commented on SPARK-22935: -- [~jlaskowski] When you see the scheme

Re: Whole-stage codegen and SparkPlan.newPredicate

2018-01-01 Thread Kazuaki Ishizaki
updated the JIRA ticket. I am still working for this. Kazuaki Ishizaki From: Herman van Hövell tot Westerflier <hvanhov...@databricks.com> To: Kazuaki Ishizaki <ishiz...@jp.ibm.com> Cc: Jacek Laskowski <ja...@japila.pl>, dev <dev@spark.apache.org> Date:

[jira] [Updated] (SPARK-22935) Dataset with Java Beans for java.sql.Date throws CompileException

2018-01-01 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22935: - Description: The following code can throw an exception with or without whole-stage

[jira] [Resolved] (SPARK-21413) Multiple projections with CASE WHEN fails to run generated codes

2018-01-01 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki resolved SPARK-21413. -- Resolution: Duplicate > Multiple projections with CASE WHEN fails to run genera

[jira] [Commented] (SPARK-21413) Multiple projections with CASE WHEN fails to run generated codes

2018-01-01 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307531#comment-16307531 ] Kazuaki Ishizaki commented on SPARK-21413: -- SPARK-22520 solved this issue. > Multi

[jira] [Commented] (SPARK-22935) Dataset with Java Beans for java.sql.Date produces incorrect result

2018-01-01 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307520#comment-16307520 ] Kazuaki Ishizaki commented on SPARK-22935: -- I am working for this. > Dataset with Java Be

[jira] [Created] (SPARK-22935) Dataset with Java Beans for java.sql.Date produces incorrect result

2018-01-01 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-22935: Summary: Dataset with Java Beans for java.sql.Date produces incorrect result Key: SPARK-22935 URL: https://issues.apache.org/jira/browse/SPARK-22935 Project

Re: Whole-stage codegen and SparkPlan.newPredicate

2018-01-01 Thread Kazuaki Ishizaki
produces incorrect results. I created a JIRA entry https://issues.apache.org/jira/browse/SPARK-22934. Best Regards, Kazuaki Ishizaki From: Herman van Hövell tot Westerflier <hvanhov...@databricks.com> To: Jacek Laskowski <ja...@japila.pl> Cc: dev <dev@spark.apache.or

[jira] [Updated] (SPARK-22935) Dataset with Java Beans for java.sql.Date produces incorrect result

2018-01-01 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22935: - Description: The following code prints {{c=0}}. The value of {{c}} must be 2. {{ds

[jira] [Commented] (SPARK-22500) 64KB JVM bytecode limit problem with cast

2017-12-29 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16306244#comment-16306244 ] Kazuaki Ishizaki commented on SPARK-22500: -- Thank you for your report. I think

[jira] [Updated] (SPARK-22868) 64KB JVM bytecode limit problem with aggregation

2017-12-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22868: - Issue Type: Sub-task (was: Bug) Parent: SPARK-22510 > 64KB JVM bytecode li

[jira] [Updated] (SPARK-22869) 64KB JVM bytecode limit problem with filter

2017-12-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22869: - Issue Type: Sub-task (was: Bug) Parent: SPARK-22510 > 64KB JVM bytecode li

Re: Timeline for Spark 2.3

2017-12-21 Thread Kazuaki Ishizaki
+1 for cutting a branch earlier. In some Asian countries, 1st, 2nd, and 3rd January are off. https://www.timeanddate.com/holidays/ How about 4th or 5th? Regards, Kazuaki Ishizaki From: Felix Cheung <felixcheun...@hotmail.com> To: Michael Armbrust <mich...@databricks.com>,

[jira] [Commented] (SPARK-22869) 64KB JVM bytecode limit problem with filter

2017-12-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16300946#comment-16300946 ] Kazuaki Ishizaki commented on SPARK-22869: -- I am working for this and waiting for merging SPARK

[jira] [Commented] (SPARK-22868) 64KB JVM bytecode limit problem with aggregation

2017-12-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16300945#comment-16300945 ] Kazuaki Ishizaki commented on SPARK-22868: -- I am working for this and waiting for merging SPARK

[jira] [Created] (SPARK-22869) 64KB JVM bytecode limit problem with filter

2017-12-21 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-22869: Summary: 64KB JVM bytecode limit problem with filter Key: SPARK-22869 URL: https://issues.apache.org/jira/browse/SPARK-22869 Project: Spark Issue

[jira] [Created] (SPARK-22868) 64KB JVM bytecode limit problem with aggregation

2017-12-21 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-22868: Summary: 64KB JVM bytecode limit problem with aggregation Key: SPARK-22868 URL: https://issues.apache.org/jira/browse/SPARK-22868 Project: Spark

[jira] [Updated] (SPARK-22848) Avoid the generation of useless mutable states by Stack function

2017-12-20 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22848: - Description: Current Spark generates two mutable states in generated code for {{{Stack

[jira] [Updated] (SPARK-22848) Avoid the generation of useless mutable states by Stack function

2017-12-20 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22848: - Description: Current Spark generates two mutable states in generated code for {Stack

[jira] [Updated] (SPARK-22848) Avoid the generation of useless mutable states by Stack function

2017-12-20 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22848: - Description: Current Spark generates two mutable states in generated code for {{Stack

<    1   2   3   4   5   6   7   8   9   10   >