[jira] [Updated] (SPARK-22848) Avoid the generation of useless mutable states by Stack function

2017-12-20 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22848: - Summary: Avoid the generation of useless mutable states by Stack function (was: Do

[jira] [Updated] (SPARK-22848) Do not execute generated code of functions in Generator

2017-12-20 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22848: - Issue Type: Sub-task (was: Improvement) Parent: SPARK-22692 > Do not exec

[jira] [Created] (SPARK-22848) Do not execute generated code of functions in Generator

2017-12-20 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-22848: Summary: Do not execute generated code of functions in Generator Key: SPARK-22848 URL: https://issues.apache.org/jira/browse/SPARK-22848 Project: Spark

[jira] [Commented] (SPARK-22813) run-tests.py fails when /usr/sbin/lsof does not exist

2017-12-15 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293570#comment-16293570 ] Kazuaki Ishizaki commented on SPARK-22813: -- Thank you for pointing the PR that we worked. I

[jira] [Updated] (SPARK-22813) run-tests.py fails when /usr/sbin/lsof does not exist

2017-12-15 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22813: - Description: Running ./dev/run-tests.py for mvn on OS that does not have /usr/sbin/lsof

[jira] [Created] (SPARK-22813) run-tests.py fails when /usr/sbin/lsof does not exist

2017-12-15 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-22813: Summary: run-tests.py fails when /usr/sbin/lsof does not exist Key: SPARK-22813 URL: https://issues.apache.org/jira/browse/SPARK-22813 Project: Spark

[jira] [Commented] (SPARK-22776) Increase default value of spark.sql.codegen.maxFields

2017-12-13 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290351#comment-16290351 ] Kazuaki Ishizaki commented on SPARK-22776: -- Wait for re-merging SPARK-22600 > Increase defa

[jira] [Created] (SPARK-22776) Increase default value of spark.sql.codegen.maxFields

2017-12-13 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-22776: Summary: Increase default value of spark.sql.codegen.maxFields Key: SPARK-22776 URL: https://issues.apache.org/jira/browse/SPARK-22776 Project: Spark

[jira] [Commented] (SPARK-22774) Add compilation check for generated code in TPCDSQuerySuite

2017-12-13 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289471#comment-16289471 ] Kazuaki Ishizaki commented on SPARK-22774: -- I will commit a PR for this soon. > Add compilat

[jira] [Comment Edited] (SPARK-22774) Add compilation check for generated code in TPCDSQuerySuite

2017-12-13 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289471#comment-16289471 ] Kazuaki Ishizaki edited comment on SPARK-22774 at 12/13/17 4:24 PM: I

[jira] [Created] (SPARK-22774) Add compilation check for generated code in TPCDSQuerySuite

2017-12-13 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-22774: Summary: Add compilation check for generated code in TPCDSQuerySuite Key: SPARK-22774 URL: https://issues.apache.org/jira/browse/SPARK-22774 Project: Spark

[jira] [Created] (SPARK-22747) Shorten lifetime of global variables used in HashAggregateExec

2017-12-09 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-22747: Summary: Shorten lifetime of global variables used in HashAggregateExec Key: SPARK-22747 URL: https://issues.apache.org/jira/browse/SPARK-22747 Project

[jira] [Updated] (SPARK-22746) Avoid the generation of useless mutable states by SortMergeJoin

2017-12-09 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22746: - Issue Type: Sub-task (was: Bug) Parent: SPARK-22692 > Avoid the generat

[jira] [Created] (SPARK-22746) Avoid the generation of useless mutable states by SortMergeJoin

2017-12-09 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-22746: Summary: Avoid the generation of useless mutable states by SortMergeJoin Key: SPARK-22746 URL: https://issues.apache.org/jira/browse/SPARK-22746 Project

[jira] [Created] (SPARK-22705) Reduce # of mutable variables in Case, Coalesce, and In

2017-12-05 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-22705: Summary: Reduce # of mutable variables in Case, Coalesce, and In Key: SPARK-22705 URL: https://issues.apache.org/jira/browse/SPARK-22705 Project: Spark

[jira] [Created] (SPARK-22704) Reduce # of mutable variables in Least and greatest

2017-12-05 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-22704: Summary: Reduce # of mutable variables in Least and greatest Key: SPARK-22704 URL: https://issues.apache.org/jira/browse/SPARK-22704 Project: Spark

[jira] [Updated] (SPARK-22688) Upgrade Janino version to 3.0.8

2017-12-05 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22688: - Summary: Upgrade Janino version to 3.0.8 (was: Upgrade Janino version 3.0.8) > Upgr

[jira] [Updated] (SPARK-22688) Upgrade Janino version 3.0.8

2017-12-04 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22688: - Description: [Janino 3.0.8|https://janino-compiler.github.io/janino/changelog.html

[jira] [Updated] (SPARK-22688) Upgrade Janino version 3.0.8

2017-12-04 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22688: - Summary: Upgrade Janino version 3.0.8 (was: Upgrade Janino version 0.3.8) > Upgr

[jira] [Created] (SPARK-22688) Upgrade Janino version 0.3.8

2017-12-04 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-22688: Summary: Upgrade Janino version 0.3.8 Key: SPARK-22688 URL: https://issues.apache.org/jira/browse/SPARK-22688 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-22668) CodegenContext.splitExpressions() creates incorrect results with global variable arguments

2017-12-01 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22668: - Summary: CodegenContext.splitExpressions() creates incorrect results with global

[jira] [Updated] (SPARK-22668) CodegenContext.splitExpressions() creates incorrect results with global variable arguments

2017-12-01 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22668: - Description: {{CodegenContext.splitExpressions()}} creates incorrect results

[jira] [Commented] (SPARK-22668) CodegenContext.splitExpression() creates incorrect results with global variable arguments

2017-12-01 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16274128#comment-16274128 ] Kazuaki Ishizaki commented on SPARK-22668: -- I am working

[jira] [Created] (SPARK-22668) CodegenContext.splitExpression() creates incorrect results with global variable arguments

2017-12-01 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-22668: Summary: CodegenContext.splitExpression() creates incorrect results with global variable arguments Key: SPARK-22668 URL: https://issues.apache.org/jira/browse/SPARK

Re: [VOTE] Spark 2.2.1 (RC2)

2017-11-29 Thread Kazuaki Ishizaki
04 h [INFO] Finished at: 2017-11-30T01:48:15+09:00 [INFO] Final Memory: 128M/329M [INFO] [WARNING] The requested profile "hive" could not be activated because it does not exist. Kazuaki Ishizaki From: Dongj

[jira] [Commented] (SPARK-22373) Intermittent NullPointerException in org.codehaus.janino.IClass.isAssignableFrom

2017-11-27 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267901#comment-16267901 ] Kazuaki Ishizaki commented on SPARK-22373: -- [~mshen] Thank you for reporting this issue. We

[jira] [Updated] (SPARK-22608) Avoid code duplication regarding CodeGeneration.splitExpressions()

2017-11-26 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22608: - Issue Type: Improvement (was: Bug) > Avoid code duplication regard

[jira] [Updated] (SPARK-22608) Avoid code duplication regarding CodeGeneration.splitExpressions()

2017-11-26 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22608: - Priority: Minor (was: Major) > Avoid code duplication regard

[jira] [Updated] (SPARK-22608) Avoid code duplication regarding CodeGeneration.splitExpressions()

2017-11-26 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22608: - Summary: Avoid code duplication regarding CodeGeneration.splitExpressions() (was: Add

[jira] [Created] (SPARK-22608) Add new API to CodeGeneration.splitExpressions()

2017-11-26 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-22608: Summary: Add new API to CodeGeneration.splitExpressions() Key: SPARK-22608 URL: https://issues.apache.org/jira/browse/SPARK-22608 Project: Spark

[jira] [Updated] (SPARK-22603) 64KB JVM bytecode limit problem with FormatString

2017-11-24 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22603: - Issue Type: Sub-task (was: Bug) Parent: SPARK-22510 > 64KB JVM bytecode li

[jira] [Updated] (SPARK-22603) 64KB JVM bytecode limit problem with FormatString

2017-11-24 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22603: - Description: {{FormatString}} can throw an exception due to the 64KB JVM bytecode limit

[jira] [Created] (SPARK-22603) 64KB JVM bytecode limit problem with FormatString

2017-11-24 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-22603: Summary: 64KB JVM bytecode limit problem with FormatString Key: SPARK-22603 URL: https://issues.apache.org/jira/browse/SPARK-22603 Project: Spark

[jira] [Commented] (SPARK-22595) flaky test: CastSuite.SPARK-22500: cast for struct should not generate codes beyond 64KB

2017-11-23 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264663#comment-16264663 ] Kazuaki Ishizaki commented on SPARK-22595: -- I see. I will look at this. > flaky t

[jira] [Updated] (SPARK-22570) Create a lot of global variables to reuse an object in generated code

2017-11-22 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22570: - Description: Generated code for several operations such as {{Cast}}, {{RegExpReplace

[jira] [Updated] (SPARK-22570) Create a lot of global variables to reuse an object

2017-11-22 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22570: - Summary: Create a lot of global variables to reuse an object (was: Cast may create

[jira] [Updated] (SPARK-22570) Create a lot of global variables to reuse an object in generated code

2017-11-22 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22570: - Summary: Create a lot of global variables to reuse an object in generated code

[jira] [Updated] (SPARK-22570) Cast may create a lot of UTF8String.IntWrapper or UTF8String.longWrapper instances

2017-11-22 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22570: - Issue Type: Sub-task (was: Bug) Parent: SPARK-22510 > Cast may create a

[jira] [Updated] (SPARK-22510) Exceptions caused by 64KB JVM bytecode or constant pool entry limits

2017-11-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22510: - Description: Codegen can throw an exception due to the 64KB JVM bytecode or 64K constant

[jira] [Updated] (SPARK-22510) Exceptions caused by 64KB JVM bytecode or 64K constant pool entry limit

2017-11-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22510: - Summary: Exceptions caused by 64KB JVM bytecode or 64K constant pool entry limit

[jira] [Updated] (SPARK-22510) Exceptions caused by 64KB JVM bytecode or constant pool entry limits

2017-11-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22510: - Summary: Exceptions caused by 64KB JVM bytecode or constant pool entry limits

[jira] [Commented] (SPARK-22570) Cast may create a lot of UTF8String.IntWrapper or UTF8String.longWrapper instances

2017-11-20 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16260363#comment-16260363 ] Kazuaki Ishizaki commented on SPARK-22570: -- I am working on this with SPARK-22500 > Cast

[jira] [Created] (SPARK-22570) Cast may create a lot of UTF8String.IntWrapper or UTF8String.longWrapper instances

2017-11-20 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-22570: Summary: Cast may create a lot of UTF8String.IntWrapper or UTF8String.longWrapper instances Key: SPARK-22570 URL: https://issues.apache.org/jira/browse/SPARK-22570

[jira] [Updated] (SPARK-22550) 64KB JVM bytecode limit problem with elt

2017-11-17 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22550: - Issue Type: Sub-task (was: Bug) Parent: SPARK-22510 > 64KB JVM bytecode li

[jira] [Created] (SPARK-22550) 64KB JVM bytecode limit problem with elt

2017-11-17 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-22550: Summary: 64KB JVM bytecode limit problem with elt Key: SPARK-22550 URL: https://issues.apache.org/jira/browse/SPARK-22550 Project: Spark Issue Type

[jira] [Updated] (SPARK-22549) 64KB JVM bytecode limit problem with concat_ws

2017-11-17 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22549: - Issue Type: Sub-task (was: Bug) Parent: SPARK-22510 > 64KB JVM bytecode li

[jira] [Created] (SPARK-22549) 64KB JVM bytecode limit problem with concat_ws

2017-11-17 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-22549: Summary: 64KB JVM bytecode limit problem with concat_ws Key: SPARK-22549 URL: https://issues.apache.org/jira/browse/SPARK-22549 Project: Spark Issue

[jira] [Updated] (SPARK-22498) 64KB JVM bytecode limit problem with concat

2017-11-17 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22498: - Summary: 64KB JVM bytecode limit problem with concat (was: 64KB JVM bytecode limit

[jira] [Updated] (SPARK-22498) 64KB JVM bytecode limit problem with concat

2017-11-17 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22498: - Description: {{concat}} can throw an exception due to the 64KB JVM bytecode limit when

[jira] [Commented] (SPARK-22517) NullPointerException in ShuffleExternalSorter.spill()

2017-11-15 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254706#comment-16254706 ] Kazuaki Ishizaki commented on SPARK-22517: -- Could you please post a program that can reproduce

[jira] [Updated] (SPARK-21337) SQL which has large ‘case when’ expressions may cause code generation beyond 64KB

2017-11-14 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-21337: - Issue Type: Sub-task (was: Bug) Parent: SPARK-22510 > SQL which has large ‘c

[jira] [Commented] (SPARK-22510) Exceptions caused by 64KB JVM bytecode limit

2017-11-13 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250839#comment-16250839 ] Kazuaki Ishizaki commented on SPARK-22510: -- [~smilegator] Thanks, good idea. I will add other

[jira] [Created] (SPARK-22508) 64KB JVM bytecode limit problem with GenerateUnsafeRowJoiner.create()

2017-11-13 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-22508: Summary: 64KB JVM bytecode limit problem with GenerateUnsafeRowJoiner.create() Key: SPARK-22508 URL: https://issues.apache.org/jira/browse/SPARK-22508

[jira] [Commented] (SPARK-22501) 64KB JVM bytecode limit problem with in

2017-11-12 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249029#comment-16249029 ] Kazuaki Ishizaki commented on SPARK-22501: -- [~mgaido] I am working on this. > 64KB JVM bytec

[jira] [Created] (SPARK-22501) 64KB JVM bytecode limit problem with in

2017-11-12 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-22501: Summary: 64KB JVM bytecode limit problem with in Key: SPARK-22501 URL: https://issues.apache.org/jira/browse/SPARK-22501 Project: Spark Issue Type

[jira] [Updated] (SPARK-22499) 64KB JVM bytecode limit problem with least and greatest

2017-11-12 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22499: - Description: Both {{least}} and {{greatest}} can throw an exception due to the 64KB JVM

[jira] [Created] (SPARK-22500) 64KB JVM bytecode limit problem with cast

2017-11-12 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-22500: Summary: 64KB JVM bytecode limit problem with cast Key: SPARK-22500 URL: https://issues.apache.org/jira/browse/SPARK-22500 Project: Spark Issue Type

[jira] [Updated] (SPARK-22498) 64KB JVM bytecode limit problem with concat and concat_ws

2017-11-12 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22498: - Description: Both {{concat}} and {{concat_ws}} can throw an exception due to the 64KB

[jira] [Updated] (SPARK-22499) 64KB JVM bytecode limit problem with least and greatest

2017-11-12 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22499: - Description: Both {{least}} and {{greatest}} can thow an exception due to the 64KB JVM

[jira] [Updated] (SPARK-22499) 64KB JVM bytecode limit problem with least and greatest

2017-11-12 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22499: - Summary: 64KB JVM bytecode limit problem with least and greatest (was: 64KB JVM

[jira] [Created] (SPARK-22499) 64KB JVM bytecode limit problem with leastest and greatest

2017-11-12 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-22499: Summary: 64KB JVM bytecode limit problem with leastest and greatest Key: SPARK-22499 URL: https://issues.apache.org/jira/browse/SPARK-22499 Project: Spark

[jira] [Created] (SPARK-22498) 64KB JVM bytecode limit problem with concat and concat_ws

2017-11-12 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-22498: Summary: 64KB JVM bytecode limit problem with concat and concat_ws Key: SPARK-22498 URL: https://issues.apache.org/jira/browse/SPARK-22498 Project: Spark

[jira] [Commented] (SPARK-22481) CatalogImpl.refreshTable is slow

2017-11-10 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247873#comment-16247873 ] Kazuaki Ishizaki commented on SPARK-22481: -- Thanks. Is that dataframe cached

[jira] [Commented] (SPARK-22481) CatalogImpl.refreshTable is slow

2017-11-10 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247845#comment-16247845 ] Kazuaki Ishizaki commented on SPARK-22481: -- Sorry, I overlooked "where it used to take 2 se

[jira] [Commented] (SPARK-22481) CatalogImpl.refreshTable is slow

2017-11-10 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247826#comment-16247826 ] Kazuaki Ishizaki commented on SPARK-22481: -- How long does it take in the old code? Can you put

[jira] [Commented] (SPARK-22481) CatalogImpl.refreshTable is slow

2017-11-09 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246041#comment-16246041 ] Kazuaki Ishizaki commented on SPARK-22481: -- This change was introduce by [this PR|https

[jira] [Commented] (SPARK-22474) cannot read a parquet file containing a Seq[Map[MyCaseClass, String]]

2017-11-09 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245915#comment-16245915 ] Kazuaki Ishizaki commented on SPARK-22474: -- This check was introduced by [this PR|https

[jira] [Commented] (SPARK-22472) Datasets generate random values for null primitive types

2017-11-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244438#comment-16244438 ] Kazuaki Ishizaki commented on SPARK-22472: -- >From [the initial >version|https://gith

[jira] [Comment Edited] (SPARK-22472) Datasets generate random values for null primitive types

2017-11-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244327#comment-16244327 ] Kazuaki Ishizaki edited comment on SPARK-22472 at 11/8/17 5:30 PM

[jira] [Comment Edited] (SPARK-22472) Datasets generate random values for null primitive types

2017-11-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244327#comment-16244327 ] Kazuaki Ishizaki edited comment on SPARK-22472 at 11/8/17 5:22 PM

[jira] [Comment Edited] (SPARK-22472) Datasets generate random values for null primitive types

2017-11-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244327#comment-16244327 ] Kazuaki Ishizaki edited comment on SPARK-22472 at 11/8/17 5:11 PM

[jira] [Commented] (SPARK-22472) Datasets generate random values for null primitive types

2017-11-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244327#comment-16244327 ] Kazuaki Ishizaki commented on SPARK-22472: -- Thank you for reporting this behavior. When I

[jira] [Comment Edited] (SPARK-22472) Datasets generate random values for null primitive types

2017-11-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244327#comment-16244327 ] Kazuaki Ishizaki edited comment on SPARK-22472 at 11/8/17 5:10 PM

[jira] [Commented] (SPARK-22460) Spark De-serialization of Timestamp field is Incorrect

2017-11-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243913#comment-16243913 ] Kazuaki Ishizaki commented on SPARK-22460: -- I run the similar code with json. It can correctly

[jira] [Commented] (SPARK-22373) Intermittent NullPointerException in org.codehaus.janino.IClass.isAssignableFrom

2017-11-06 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241469#comment-16241469 ] Kazuaki Ishizaki commented on SPARK-22373: -- If you can submit a program that can reproduce

[jira] [Commented] (SPARK-22427) StackOverFlowError when using FPGrowth

2017-11-02 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16237112#comment-16237112 ] Kazuaki Ishizaki commented on SPARK-22427: -- Thank you for reporting an issue. Could you please

[jira] [Resolved] (SPARK-20807) Add compression/decompression of data to ColumnVector

2017-11-02 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki resolved SPARK-20807. -- Resolution: Won't Fix > Add compression/decompression of data to ColumnVec

[jira] [Commented] (SPARK-22254) clean up the implementation of `growToSize` in CompactBuffer

2017-11-02 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236105#comment-16236105 ] Kazuaki Ishizaki commented on SPARK-22254: -- I started working for this, and will submit a PR

[jira] [Created] (SPARK-22383) Generate code to directly get value of primitive type array from ColumnVector for table cache

2017-10-28 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-22383: Summary: Generate code to directly get value of primitive type array from ColumnVector for table cache Key: SPARK-22383 URL: https://issues.apache.org/jira/browse/SPARK

[jira] [Updated] (SPARK-22378) Redundant nullcheck is generated for extracting value in complex types

2017-10-28 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22378: - Description: Redundant null check is generated in the code for extracting an element

[jira] [Created] (SPARK-22378) Eliminate redundant nullcheck in generated code for extracting value in complex type

2017-10-28 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-22378: Summary: Eliminate redundant nullcheck in generated code for extracting value in complex type Key: SPARK-22378 URL: https://issues.apache.org/jira/browse/SPARK-22378

[jira] [Commented] (SPARK-22284) Code of class \"org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection\" grows beyond 64 KB

2017-10-22 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214332#comment-16214332 ] Kazuaki Ishizaki commented on SPARK-22284: -- Yes, there was a similar ticket that was solved

[jira] [Commented] (SPARK-21750) Use arrow 0.6.0

2017-10-20 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213702#comment-16213702 ] Kazuaki Ishizaki commented on SPARK-21750: -- Thank you. Good to hear that. > Use arrow 0.

[jira] [Commented] (SPARK-22284) Code of class \"org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection\" grows beyond 64 KB

2017-10-19 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211353#comment-16211353 ] Kazuaki Ishizaki commented on SPARK-22284: -- I found [this JIRA|https://issues.apache.org/jira

[jira] [Commented] (SPARK-22284) Code of class \"org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection\" grows beyond 64 KB

2017-10-19 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211315#comment-16211315 ] Kazuaki Ishizaki commented on SPARK-22284: -- Thank you for uploading the generated code

[jira] [Commented] (SPARK-22254) clean up the implementation of `growToSize` in CompactBuffer

2017-10-19 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210964#comment-16210964 ] Kazuaki Ishizaki commented on SPARK-22254: -- If noone has started working for this, I will work

[jira] [Commented] (SPARK-22284) Code of class \"org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection\" grows beyond 64 KB

2017-10-19 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210943#comment-16210943 ] Kazuaki Ishizaki commented on SPARK-22284: -- Can you attach the generated code? It may help which

[jira] [Commented] (SPARK-16845) org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering" grows beyond 64 KB

2017-10-12 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202330#comment-16202330 ] Kazuaki Ishizaki commented on SPARK-16845: -- [This PR|https://github.com/apache/spark/pull/18972

[jira] [Commented] (SPARK-18492) GeneratedIterator grows beyond 64 KB

2017-10-11 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200682#comment-16200682 ] Kazuaki Ishizaki commented on SPARK-18492: -- [~ramzanfarooq] I see. We do not need your original

[jira] [Commented] (SPARK-22226) splitExpression can create too many method calls (generating a Constant Pool limit error)

2017-10-11 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200629#comment-16200629 ] Kazuaki Ishizaki commented on SPARK-6: -- [~mgaido] I think that you can reopen this JIRA

[jira] [Commented] (SPARK-18492) GeneratedIterator grows beyond 64 KB

2017-10-11 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200606#comment-16200606 ] Kazuaki Ishizaki commented on SPARK-18492: -- [~cenyuhai][~ramzanfarooq] Thank you for reporting

[jira] [Created] (SPARK-22246) UnsafeRow, UnsafeArrayData, and UnsafeMapData use MemoryBlock

2017-10-11 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-22246: Summary: UnsafeRow, UnsafeArrayData, and UnsafeMapData use MemoryBlock Key: SPARK-22246 URL: https://issues.apache.org/jira/browse/SPARK-22246 Project: Spark

[jira] [Commented] (SPARK-22226) splitExpression can create too many method calls (generating a Constant Pool limit error)

2017-10-09 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197272#comment-16197272 ] Kazuaki Ishizaki commented on SPARK-6: -- You are right. [This PR|https://github.com/apache

[jira] [Commented] (SPARK-22226) Code generation fails for dataframes with 10000 columns

2017-10-09 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197164#comment-16197164 ] Kazuaki Ishizaki commented on SPARK-6: -- [This PR|https://github.com/apache/spark/pull/16648

[jira] [Created] (SPARK-22219) Refector "spark.sql.codegen.comments"

2017-10-06 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-22219: Summary: Refector "spark.sql.codegen.comments" Key: SPARK-22219 URL: https://issues.apache.org/jira/browse/SPARK-22219 Project: Spark

[jira] [Commented] (SPARK-19984) ERROR codegen.CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java'

2017-10-06 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194937#comment-16194937 ] Kazuaki Ishizaki commented on SPARK-19984: -- [~JohnSteidley] Here is my program that produced

[jira] [Commented] (SPARK-19984) ERROR codegen.CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java'

2017-10-05 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193201#comment-16193201 ] Kazuaki Ishizaki commented on SPARK-19984: -- [~JohnSteidley] Thank you for your comment. Now, I

[jira] [Commented] (SPARK-19984) ERROR codegen.CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java'

2017-10-03 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190050#comment-16190050 ] Kazuaki Ishizaki commented on SPARK-19984: -- [~JohnSteidley] Thank you for providing valuable

Re: [VOTE] Spark 2.1.2 (RC4)

2017-10-03 Thread Kazuaki Ishizaki
min [INFO] Finished at: 2017-10-03T23:53:33+09:00 [INFO] Final Memory: 112M/322M [INFO] [WARNING] The requested profile "hive" could not be activated because it does not exist. Kazuaki Ishizaki From: Dongj

[jira] [Commented] (SPARK-19984) ERROR codegen.CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java'

2017-10-02 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188320#comment-16188320 ] Kazuaki Ishizaki commented on SPARK-19984: -- [~JohnSteidley] Thank you for your report. I also

<    2   3   4   5   6   7   8   9   10   11   >