Re: Welcoming Tejas Patil as a Spark committer

Congratulation Tejas! Kazuaki Ishizaki From: Matei Zaharia <matei.zaha...@gmail.com> To: "dev@spark.apache.org" <dev@spark.apache.org> Date: 2017/09/30 04:58 Subject:Welcoming Tejas Patil as a Spark committer Hi all, The Spark PMC recently added Teja

[jira] [Commented] (SPARK-18016) Code Generation: Constant Pool Past Limit for Wide/Nested Dataset

[ https://issues.apache.org/jira/browse/SPARK-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184206#comment-16184206 ] Kazuaki Ishizaki commented on SPARK-18016: -- Thank you for reporting this again. While I pinged

Re: [VOTE] Spark 2.1.2 (RC2)

min [INFO] Finished at: 2017-09-28T12:16:46+09:00 [INFO] Final Memory: 103M/309M [INFO] [WARNING] The requested profile "hive" could not be activated because it does not exist. Kazuaki Ishizaki From: Dongj

[jira] [Updated] (SPARK-22130) UTF8String.trim() inefficiently scans all white-space string twice.

[ https://issues.apache.org/jira/browse/SPARK-22130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22130: - Issue Type: Improvement (was: Bug) > UTF8String.trim() inefficiently scans all wh

[jira] [Commented] (SPARK-16845) org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering" grows beyond 64 KB

[ https://issues.apache.org/jira/browse/SPARK-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181190#comment-16181190 ] Kazuaki Ishizaki commented on SPARK-16845: -- [~mvelusce] Thank you for reporting an issue

[jira] [Commented] (SPARK-22130) UTF8String.trim() inefficiently scans all white-space string twice.

[ https://issues.apache.org/jira/browse/SPARK-22130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181070#comment-16181070 ] Kazuaki Ishizaki commented on SPARK-22130: -- I will submit a PR soon. > UTF8String.t

[jira] [Created] (SPARK-22130) UTF8String.trim() inefficiently scans all white-space string twice.

Kazuaki Ishizaki created SPARK-22130: Summary: UTF8String.trim() inefficiently scans all white-space string twice. Key: SPARK-22130 URL: https://issues.apache.org/jira/browse/SPARK-22130 Project

[jira] [Comment Edited] (SPARK-22105) Dataframe has poor performance when computing on many columns with codegen

[ https://issues.apache.org/jira/browse/SPARK-22105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176655#comment-16176655 ] Kazuaki Ishizaki edited comment on SPARK-22105 at 9/22/17 4:22 PM: --- Can

[jira] [Commented] (SPARK-22105) Dataframe has poor performance when computing on many columns with codegen

[ https://issues.apache.org/jira/browse/SPARK-22105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176655#comment-16176655 ] Kazuaki Ishizaki commented on SPARK-22105: -- Can this PR at https://issues.apache.org/jira/browse

[jira] [Commented] (SPARK-22000) org.codehaus.commons.compiler.CompileException: toString method is not declared

[ https://issues.apache.org/jira/browse/SPARK-22000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170319#comment-16170319 ] Kazuaki Ishizaki commented on SPARK-22000: -- If there is no sample code, it may take a long time

[jira] [Commented] (SPARK-22033) BufferHolder size checks should account for the specific VM array size limitations

[ https://issues.apache.org/jira/browse/SPARK-22033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169318#comment-16169318 ] Kazuaki Ishizaki commented on SPARK-22033: -- I think {{ColumnVector}} and {{HashMapGrowthStrategy

[jira] [Commented] (SPARK-22000) org.codehaus.commons.compiler.CompileException: toString method is not declared

[ https://issues.apache.org/jira/browse/SPARK-22000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165902#comment-16165902 ] Kazuaki Ishizaki commented on SPARK-22000: -- Thank you for good suggestion. I will try to use

[jira] [Commented] (SPARK-22000) org.codehaus.commons.compiler.CompileException: toString method is not declared

[ https://issues.apache.org/jira/browse/SPARK-22000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165883#comment-16165883 ] Kazuaki Ishizaki commented on SPARK-22000: -- It would be good to generate {{((Long)value13

[jira] [Commented] (SPARK-21907) NullPointerException in UnsafeExternalSorter.spill()

[ https://issues.apache.org/jira/browse/SPARK-21907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158995#comment-16158995 ] Kazuaki Ishizaki commented on SPARK-21907: -- If you cannot provide a repro, could you please run

[jira] [Commented] (SPARK-21905) ClassCastException when call sqlContext.sql on temp table

[ https://issues.apache.org/jira/browse/SPARK-21905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158496#comment-16158496 ] Kazuaki Ishizaki commented on SPARK-21905: -- While I ran the following code (I do not have

[jira] [Commented] (SPARK-21946) Flaky test: InMemoryCatalogedDDLSuite.`alter table: rename cached table`

[ https://issues.apache.org/jira/browse/SPARK-21946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158044#comment-16158044 ] Kazuaki Ishizaki commented on SPARK-21946: -- If someone has not worked for this, I will create

[jira] [Commented] (SPARK-21907) NullPointerException in UnsafeExternalSorter.spill()

[ https://issues.apache.org/jira/browse/SPARK-21907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156015#comment-16156015 ] Kazuaki Ishizaki commented on SPARK-21907: -- Thank you for your report. Could you please attach

[jira] [Commented] (SPARK-21894) Some Netty errors do not propagate to the top level driver

[ https://issues.apache.org/jira/browse/SPARK-21894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151930#comment-16151930 ] Kazuaki Ishizaki commented on SPARK-21894: -- Thank you for reporting this issue. Could you please

Re: Welcoming Saisai (Jerry) Shao as a committer

Congratulations, Jerry! Kazuaki Ishizaki From: Hyukjin Kwon <gurwls...@gmail.com> To: dev <dev@spark.apache.org> Date: 2017/08/29 12:24 Subject:Re: Welcoming Saisai (Jerry) Shao as a committer Congratulations! Very well deserved. 2017-08-29 11:41 GMT+09:00 Li

[jira] [Commented] (SPARK-18016) Code Generation: Constant Pool Past Limit for Wide/Nested Dataset

[ https://issues.apache.org/jira/browse/SPARK-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16141650#comment-16141650 ] Kazuaki Ishizaki commented on SPARK-18016: -- The issue {{Caused

[jira] [Commented] (SPARK-21828) org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering" grows beyond 64 KB...again

[ https://issues.apache.org/jira/browse/SPARK-21828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16141361#comment-16141361 ] Kazuaki Ishizaki commented on SPARK-21828: -- Thank you for your report. Some fixes solved

[jira] [Commented] (SPARK-21828) org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering" grows beyond 64 KB...again

[ https://issues.apache.org/jira/browse/SPARK-21828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140282#comment-16140282 ] Kazuaki Ishizaki commented on SPARK-21828: -- Thank you for reporting a problem. First, IIUC

[jira] [Commented] (SPARK-21750) Use arrow 0.6.0

[ https://issues.apache.org/jira/browse/SPARK-21750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136920#comment-16136920 ] Kazuaki Ishizaki commented on SPARK-21750: -- Closed this since to upgrade Arrow requires

[jira] [Closed] (SPARK-21750) Use arrow 0.6.0

[ https://issues.apache.org/jira/browse/SPARK-21750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki closed SPARK-21750. Resolution: Won't Fix > Use arrow 0.6.0 > --- > >

[jira] [Commented] (SPARK-21794) exception about reading task serial data(broadcast) value when the storage memory is not enough to unroll

[ https://issues.apache.org/jira/browse/SPARK-21794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134440#comment-16134440 ] Kazuaki Ishizaki commented on SPARK-21794: -- Thank you for reporting this issue. Could you please

[jira] [Commented] (SPARK-21776) How to use the memory-mapped file on Spark??

[ https://issues.apache.org/jira/browse/SPARK-21776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131681#comment-16131681 ] Kazuaki Ishizaki commented on SPARK-21776: -- Is this a question? It this is a kind of questions

[jira] [Commented] (SPARK-21720) Filter predicate with many conditions throw stackoverflow error

[ https://issues.apache.org/jira/browse/SPARK-21720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130231#comment-16130231 ] Kazuaki Ishizaki commented on SPARK-21720: -- I identified issues in {{predicates.scala}}. I am

[jira] [Created] (SPARK-21751) CodeGeneraor.splitExpressions counts code size more precisely

Kazuaki Ishizaki created SPARK-21751: Summary: CodeGeneraor.splitExpressions counts code size more precisely Key: SPARK-21751 URL: https://issues.apache.org/jira/browse/SPARK-21751 Project: Spark

[jira] [Commented] (SPARK-21750) Use arrow 0.6.0

[ https://issues.apache.org/jira/browse/SPARK-21750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129133#comment-16129133 ] Kazuaki Ishizaki commented on SPARK-21750: -- Waiting for it on mvnrepository > Use arrow 0.

[jira] [Created] (SPARK-21750) Use arrow 0.6.0

Kazuaki Ishizaki created SPARK-21750: Summary: Use arrow 0.6.0 Key: SPARK-21750 URL: https://issues.apache.org/jira/browse/SPARK-21750 Project: Spark Issue Type: Improvement

[jira] [Comment Edited] (SPARK-21720) Filter predicate with many conditions throw stackoverflow error

[ https://issues.apache.org/jira/browse/SPARK-21720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127477#comment-16127477 ] Kazuaki Ishizaki edited comment on SPARK-21720 at 8/15/17 4:26 PM

[jira] [Commented] (SPARK-21720) Filter predicate with many conditions throw stackoverflow error

[ https://issues.apache.org/jira/browse/SPARK-21720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127477#comment-16127477 ] Kazuaki Ishizaki commented on SPARK-21720: -- In this case, to add JVM option {{-Xss512m

[jira] [Commented] (SPARK-21720) Filter predicate with many conditions throw stackoverflow error

[ https://issues.apache.org/jira/browse/SPARK-21720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16125253#comment-16125253 ] Kazuaki Ishizaki commented on SPARK-21720: -- I confirmed that this occurs in the master branch. I

[jira] [Comment Edited] (SPARK-19372) Code generation for Filter predicate including many OR conditions exceeds JVM method size limit

[ https://issues.apache.org/jira/browse/SPARK-19372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124982#comment-16124982 ] Kazuaki Ishizaki edited comment on SPARK-19372 at 8/13/17 5:05 PM

[jira] [Commented] (SPARK-19372) Code generation for Filter predicate including many OR conditions exceeds JVM method size limit

[ https://issues.apache.org/jira/browse/SPARK-19372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124982#comment-16124982 ] Kazuaki Ishizaki commented on SPARK-19372: -- [~srinivasanm] I can reproduce this issue by using

[jira] [Commented] (SPARK-19372) Code generation for Filter predicate including many OR conditions exceeds JVM method size limit

[ https://issues.apache.org/jira/browse/SPARK-19372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124856#comment-16124856 ] Kazuaki Ishizaki commented on SPARK-19372: -- Thank you for letting us know the problem. I

[jira] [Commented] (SPARK-21276) Update lz4-java to remove custom LZ4BlockInputStream

[ https://issues.apache.org/jira/browse/SPARK-21276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118402#comment-16118402 ] Kazuaki Ishizaki commented on SPARK-21276: -- Is it better to update affected version? > Upd

Re: tuning - Spark data serialization for cache() ?

, Kazuaki Ishizaki From: Ofir Manor <ofir.ma...@equalum.io> To: Kazuaki Ishizaki <ishiz...@jp.ibm.com> Cc: user <user@spark.apache.org> Date: 2017/08/08 03:12 Subject:Re: tuning - Spark data serialization for cache() ? Thanks a lot for the quick pointer! S

Re: tuning - Spark data serialization for cache() ?

that these PRs will be integrated into Spark 2.3. Kazuaki Ishizaki From: Ofir Manor <ofir.ma...@equalum.io> To: user <user@spark.apache.org> Date: 2017/08/08 02:04 Subject:tuning - Spark data serialization for cache() ? Hi, I'm using Spark 2.2, and have a big batc

Re: Welcoming Hyukjin Kwon and Sameer Agarwal as committers

Congratulation, Hyukjin and Sameer, well deserved!! Kazuaki Ishizaki From: Matei Zaharia <matei.zaha...@gmail.com> To: dev <dev@spark.apache.org> Date: 2017/08/08 00:53 Subject:Welcoming Hyukjin Kwon and Sameer Agarwal as committers Hi everyone, The Spark PMC re

[jira] [Commented] (SPARK-21390) Dataset filter api inconsistency

[ https://issues.apache.org/jira/browse/SPARK-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110576#comment-16110576 ] Kazuaki Ishizaki commented on SPARK-21390: -- Thank you very much for pointing out the good JIRA

[jira] [Commented] (SPARK-21591) Implement treeAggregate on Dataset API

[ https://issues.apache.org/jira/browse/SPARK-21591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108644#comment-16108644 ] Kazuaki Ishizaki commented on SPARK-21591: -- I like this idea > Implement treeAggreg

[jira] [Commented] (SPARK-18016) Code Generation: Constant Pool Past Limit for Wide/Nested Dataset

[ https://issues.apache.org/jira/browse/SPARK-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104284#comment-16104284 ] Kazuaki Ishizaki commented on SPARK-18016: -- [~jamcon] Thank you reporting the problem. We fixed

[jira] [Commented] (SPARK-21496) Support codegen for TakeOrderedAndProjectExec

[ https://issues.apache.org/jira/browse/SPARK-21496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099840#comment-16099840 ] Kazuaki Ishizaki commented on SPARK-21496: -- Is there any good benchmark program

[jira] [Commented] (SPARK-21517) Fetch local data via block manager cause oom

[ https://issues.apache.org/jira/browse/SPARK-21517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099654#comment-16099654 ] Kazuaki Ishizaki commented on SPARK-21517: -- Does it occur in Spark 2.2? > Fetch local data

[jira] [Commented] (SPARK-21501) Spark shuffle index cache size should be memory based

[ https://issues.apache.org/jira/browse/SPARK-21501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099351#comment-16099351 ] Kazuaki Ishizaki commented on SPARK-21501: -- I see. I misunderstood the description. You expect

[jira] [Closed] (SPARK-21387) org.apache.spark.memory.TaskMemoryManager.allocatePage causes OOM

[ https://issues.apache.org/jira/browse/SPARK-21387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki closed SPARK-21387. Resolution: Cannot Reproduce > org.apache.spark.memory.TaskMemoryManager.allocateP

[jira] [Closed] (SPARK-21387) org.apache.spark.memory.TaskMemoryManager.allocatePage causes OOM

[ https://issues.apache.org/jira/browse/SPARK-21387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki closed SPARK-21387. Resolution: Fixed > org.apache.spark.memory.TaskMemoryManager.allocatePage causes

[jira] [Reopened] (SPARK-21387) org.apache.spark.memory.TaskMemoryManager.allocatePage causes OOM

[ https://issues.apache.org/jira/browse/SPARK-21387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki reopened SPARK-21387: -- > org.apache.spark.memory.TaskMemoryManager.allocatePage causes

[jira] [Commented] (SPARK-21387) org.apache.spark.memory.TaskMemoryManager.allocatePage causes OOM

[ https://issues.apache.org/jira/browse/SPARK-21387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098545#comment-16098545 ] Kazuaki Ishizaki commented on SPARK-21387: -- While I got OOM in my unit test, I have

[jira] [Commented] (SPARK-21501) Spark shuffle index cache size should be memory based

[ https://issues.apache.org/jira/browse/SPARK-21501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098531#comment-16098531 ] Kazuaki Ishizaki commented on SPARK-21501: -- I guess that to use Spark 2.1 or later version

[jira] [Created] (SPARK-21516) overriding afterEach() in DatasetCacheSuite must call super.afterEach()

Kazuaki Ishizaki created SPARK-21516: Summary: overriding afterEach() in DatasetCacheSuite must call super.afterEach() Key: SPARK-21516 URL: https://issues.apache.org/jira/browse/SPARK-21516

[jira] [Comment Edited] (SPARK-21512) DatasetCacheSuite needs to execute unpersistent after executing peristent

[ https://issues.apache.org/jira/browse/SPARK-21512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097565#comment-16097565 ] Kazuaki Ishizaki edited comment on SPARK-21512 at 7/24/17 4:53 AM

[jira] [Commented] (SPARK-21512) DatasetCacheSuite needs to execute unpersistent after executing peristent

[ https://issues.apache.org/jira/browse/SPARK-21512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097565#comment-16097565 ] Kazuaki Ishizaki commented on SPARK-21512: -- When {DatasetCacheSuite} is executed, the following

[jira] [Updated] (SPARK-21512) DatasetCacheSuite needs to execute unpersistent after executing peristent

[ https://issues.apache.org/jira/browse/SPARK-21512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-21512: - Summary: DatasetCacheSuite needs to execute unpersistent after executing peristent

[jira] [Updated] (SPARK-21512) DatasetCacheSuites need to execute unpersistent after executing peristent

[ https://issues.apache.org/jira/browse/SPARK-21512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-21512: - Summary: DatasetCacheSuites need to execute unpersistent after executing peristent

[jira] [Created] (SPARK-21512) DatasetCacheSuite need to execute unpersistent after executing peristent

Kazuaki Ishizaki created SPARK-21512: Summary: DatasetCacheSuite need to execute unpersistent after executing peristent Key: SPARK-21512 URL: https://issues.apache.org/jira/browse/SPARK-21512

[jira] [Updated] (SPARK-20822) Generate code to get value from ColumnVector in ColumnarBatch

[ https://issues.apache.org/jira/browse/SPARK-20822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-20822: - Summary: Generate code to get value from ColumnVector in ColumnarBatch (was: Generate

[jira] [Updated] (SPARK-20822) Generate code to get value from CachedBatchColumnVector in ColumnarBatch

[ https://issues.apache.org/jira/browse/SPARK-20822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-20822: - Summary: Generate code to get value from CachedBatchColumnVector in ColumnarBatch

[jira] [Commented] (SPARK-21443) Very long planning duration for queries with lots of operations

[ https://issues.apache.org/jira/browse/SPARK-21443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090242#comment-16090242 ] Kazuaki Ishizaki commented on SPARK-21443: -- These two optimizations

[jira] [Commented] (SPARK-21415) Triage scapegoat warnings, part 1

[ https://issues.apache.org/jira/browse/SPARK-21415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089915#comment-16089915 ] Kazuaki Ishizaki commented on SPARK-21415: -- I see. When another JIRA will happen

[jira] [Commented] (SPARK-21390) Dataset filter api inconsistency

[ https://issues.apache.org/jira/browse/SPARK-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089012#comment-16089012 ] Kazuaki Ishizaki commented on SPARK-21390: -- cc: [~ueshin] Is there any thought

[jira] [Commented] (SPARK-21390) Dataset filter api inconsistency

[ https://issues.apache.org/jira/browse/SPARK-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088977#comment-16088977 ] Kazuaki Ishizaki commented on SPARK-21390: -- When I ran the following test suite

[jira] [Comment Edited] (SPARK-21418) NoSuchElementException: None.get on DataFrame.rdd

[ https://issues.apache.org/jira/browse/SPARK-21418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088659#comment-16088659 ] Kazuaki Ishizaki edited comment on SPARK-21418 at 7/16/17 8:25 AM: --- I

[jira] [Comment Edited] (SPARK-21393) spark (pyspark) crashes unpredictably when using show() or toPandas()

[ https://issues.apache.org/jira/browse/SPARK-21393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088660#comment-16088660 ] Kazuaki Ishizaki edited comment on SPARK-21393 at 7/15/17 4:53 PM: --- I

[jira] [Commented] (SPARK-21393) spark (pyspark) crashes unpredictably when using show() or toPandas()

[ https://issues.apache.org/jira/browse/SPARK-21393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088660#comment-16088660 ] Kazuaki Ishizaki commented on SPARK-21393: -- I confirmed that this python program works well

[jira] [Comment Edited] (SPARK-21418) NoSuchElementException: None.get on DataFrame.rdd

[ https://issues.apache.org/jira/browse/SPARK-21418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088659#comment-16088659 ] Kazuaki Ishizaki edited comment on SPARK-21418 at 7/15/17 4:43 PM: --- I

[jira] [Commented] (SPARK-21418) NoSuchElementException: None.get on DataFrame.rdd

[ https://issues.apache.org/jira/browse/SPARK-21418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088659#comment-16088659 ] Kazuaki Ishizaki commented on SPARK-21418: -- I am curious why

[jira] [Commented] (SPARK-21393) spark (pyspark) crashes unpredictably when using show() or toPandas()

[ https://issues.apache.org/jira/browse/SPARK-21393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087391#comment-16087391 ] Kazuaki Ishizaki commented on SPARK-21393: -- Not yet, however I created a patch not to cause

[jira] [Commented] (SPARK-21415) Triage scapegoat warnings, part 1

[ https://issues.apache.org/jira/browse/SPARK-21415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087097#comment-16087097 ] Kazuaki Ishizaki commented on SPARK-21415: -- Thank you. Is it better to create an umbrella JIRA

[jira] [Commented] (SPARK-21413) Multiple projections with CASE WHEN fails to run generated codes

[ https://issues.apache.org/jira/browse/SPARK-21413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16086927#comment-16086927 ] Kazuaki Ishizaki commented on SPARK-21413: -- Thank you for preparing a good repro. I can

[jira] [Commented] (SPARK-21393) spark (pyspark) crashes unpredictably when using show() or toPandas()

[ https://issues.apache.org/jira/browse/SPARK-21393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16086254#comment-16086254 ] Kazuaki Ishizaki commented on SPARK-21393: -- This program can cause the same exception {code

[jira] [Comment Edited] (SPARK-21393) spark (pyspark) crashes unpredictably when using show() or toPandas()

[ https://issues.apache.org/jira/browse/SPARK-21393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16086189#comment-16086189 ] Kazuaki Ishizaki edited comment on SPARK-21393 at 7/13/17 6:39 PM

[jira] [Commented] (SPARK-21393) spark (pyspark) crashes unpredictably when using show() or toPandas()

[ https://issues.apache.org/jira/browse/SPARK-21393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16086189#comment-16086189 ] Kazuaki Ishizaki commented on SPARK-21393: -- Thank you for uploading files. When I insert

[jira] [Updated] (SPARK-21393) spark (pyspark) crashes unpredictably when using show() or toPandas()

[ https://issues.apache.org/jira/browse/SPARK-21393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-21393: - Affects Version/s: 2.2.0 > spark (pyspark) crashes unpredictably when using s

[jira] [Commented] (SPARK-21391) Cannot convert a Seq of Map whose value type is again a seq, into a dataset

[ https://issues.apache.org/jira/browse/SPARK-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085939#comment-16085939 ] Kazuaki Ishizaki commented on SPARK-21391: -- I created [a PR|https://github.com/apache/spark/pull

[jira] [Updated] (SPARK-21390) Dataset filter api inconsistency

[ https://issues.apache.org/jira/browse/SPARK-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-21390: - Affects Version/s: 2.1.0 > Dataset filter api inconsiste

[jira] [Updated] (SPARK-21390) Dataset filter api inconsistency

[ https://issues.apache.org/jira/browse/SPARK-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-21390: - Affects Version/s: 2.2.0 > Dataset filter api inconsiste

[jira] [Commented] (SPARK-21393) spark (pyspark) crashes unpredictably when using show() or toPandas()

[ https://issues.apache.org/jira/browse/SPARK-21393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085726#comment-16085726 ] Kazuaki Ishizaki commented on SPARK-21393: -- This program seems to require 7 csv files to execute

[jira] [Commented] (SPARK-21391) Cannot convert a Seq of Map whose value type is again a seq, into a dataset

[ https://issues.apache.org/jira/browse/SPARK-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085342#comment-16085342 ] Kazuaki Ishizaki commented on SPARK-21391: -- [~neelrr] Do you want to have fix in future release

[jira] [Comment Edited] (SPARK-21391) Cannot convert a Seq of Map whose value type is again a seq, into a dataset

[ https://issues.apache.org/jira/browse/SPARK-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085099#comment-16085099 ] Kazuaki Ishizaki edited comment on SPARK-21391 at 7/13/17 3:42 AM

[jira] [Commented] (SPARK-21391) Cannot convert a Seq of Map whose value type is again a seq, into a dataset

[ https://issues.apache.org/jira/browse/SPARK-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085099#comment-16085099 ] Kazuaki Ishizaki commented on SPARK-21391: -- [~hyukjin.kwon] I think that [SPARK-19254|https

[jira] [Comment Edited] (SPARK-21391) Cannot convert a Seq of Map whose value type is again a seq, into a dataset

[ https://issues.apache.org/jira/browse/SPARK-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084333#comment-16084333 ] Kazuaki Ishizaki edited comment on SPARK-21391 at 7/12/17 5:19 PM

[jira] [Comment Edited] (SPARK-21391) Cannot convert a Seq of Map whose value type is again a seq, into a dataset

[ https://issues.apache.org/jira/browse/SPARK-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084333#comment-16084333 ] Kazuaki Ishizaki edited comment on SPARK-21391 at 7/12/17 5:19 PM

[jira] [Commented] (SPARK-21391) Cannot convert a Seq of Map whose value type is again a seq, into a dataset

[ https://issues.apache.org/jira/browse/SPARK-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084333#comment-16084333 ] Kazuaki Ishizaki commented on SPARK-21391: -- This program works with the master. {code

[jira] [Commented] (SPARK-21390) Dataset filter api inconsistency

[ https://issues.apache.org/jira/browse/SPARK-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084306#comment-16084306 ] Kazuaki Ishizaki commented on SPARK-21390: -- Another interesting results with Spark-2.2: On IDE

[jira] [Comment Edited] (SPARK-21390) Dataset filter api inconsistency

[ https://issues.apache.org/jira/browse/SPARK-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084306#comment-16084306 ] Kazuaki Ishizaki edited comment on SPARK-21390 at 7/12/17 5:09 PM

[jira] [Commented] (SPARK-21390) Dataset filter api inconsistency

[ https://issues.apache.org/jira/browse/SPARK-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084266#comment-16084266 ] Kazuaki Ishizaki commented on SPARK-21390: -- Thank you for reporting this. I can reproduce

[jira] [Created] (SPARK-21387) org.apache.spark.memory.TaskMemoryManager.allocatePage causes OOM

Kazuaki Ishizaki created SPARK-21387: Summary: org.apache.spark.memory.TaskMemoryManager.allocatePage causes OOM Key: SPARK-21387 URL: https://issues.apache.org/jira/browse/SPARK-21387 Project

[jira] [Comment Edited] (SPARK-21373) Update Jetty to 9.3.20.v20170531

[ https://issues.apache.org/jira/browse/SPARK-21373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082426#comment-16082426 ] Kazuaki Ishizaki edited comment on SPARK-21373 at 7/11/17 3:56 PM

[jira] [Updated] (SPARK-21373) Update Jetty to 9.3.20.v20170531

[ https://issues.apache.org/jira/browse/SPARK-21373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-21373: - Summary: Update Jetty to 9.3.20.v20170531 (was: Update Jetty to 9.3.20.v20170531 to fix

[jira] [Commented] (SPARK-21373) Update Jetty to 9.3.20.v20170531

[ https://issues.apache.org/jira/browse/SPARK-21373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082426#comment-16082426 ] Kazuaki Ishizaki commented on SPARK-21373: -- Since I have not clarified, I changed the title. I

[jira] [Created] (SPARK-21373) Update Jetty to 9.3.20.v20170531 to fix CVE-2017-9735

Kazuaki Ishizaki created SPARK-21373: Summary: Update Jetty to 9.3.20.v20170531 to fix CVE-2017-9735 Key: SPARK-21373 URL: https://issues.apache.org/jira/browse/SPARK-21373 Project: Spark

[jira] [Updated] (SPARK-21373) Update Jetty to 9.3.20.v20170531 to fix CVE-2017-9735

[ https://issues.apache.org/jira/browse/SPARK-21373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-21373: - Description: This is derived from https://issues.apache.org/jira/browse/FELIX-5664

[jira] [Commented] (SPARK-21364) IndexOutOfBoundsException on equality check of two complex array elements

[ https://issues.apache.org/jira/browse/SPARK-21364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16080963#comment-16080963 ] Kazuaki Ishizaki commented on SPARK-21364: -- When I ran the following test case that is derived

[jira] [Commented] (SPARK-21337) SQL which has large ‘case when’ expressions may cause code generation beyond 64KB

[ https://issues.apache.org/jira/browse/SPARK-21337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079032#comment-16079032 ] Kazuaki Ishizaki commented on SPARK-21337: -- I cannot reproduce this by using the latest or v2.1

[jira] [Commented] (SPARK-21344) BinaryType comparison does signed byte array comparison

[ https://issues.apache.org/jira/browse/SPARK-21344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079010#comment-16079010 ] Kazuaki Ishizaki commented on SPARK-21344: -- I will work for this if anyone has finished a PR

[jira] [Commented] (SPARK-21337) SQL which has large ‘case when’ expressions may cause code generation beyond 64KB

[ https://issues.apache.org/jira/browse/SPARK-21337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077952#comment-16077952 ] Kazuaki Ishizaki commented on SPARK-21337: -- In the master branch, I cannot see a huge dump

Re: [VOTE] Apache Spark 2.2.0 (RC6)

06 h [INFO] Finished at: 2017-07-01T15:20:04+09:00 [INFO] Final Memory: 56M/591M [INFO] [WARNING] The requested profile "hive" could not be activated because it does not exist. Kazuaki Ishizaki From: Mi

[jira] [Comment Edited] (SPARK-21271) UnsafeRow.hashCode assertion when sizeInBytes not multiple of 8

[ https://issues.apache.org/jira/browse/SPARK-21271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16070973#comment-16070973 ] Kazuaki Ishizaki edited comment on SPARK-21271 at 7/1/17 3:18 AM: -- I see

<    3   4   5   6   7   8   9   10   11   12   >