[jira] [Comment Edited] (SPARK-21476) RandomForest classification model not using broadcast in transform

2017-07-25 Thread Saurabh Agrawal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100151#comment-16100151 ] Saurabh Agrawal edited comment on SPARK-21476 at 7/26/17 6:56 AM: -

[jira] [Commented] (SPARK-21476) RandomForest classification model not using broadcast in transform

2017-07-25 Thread Saurabh Agrawal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101241#comment-16101241 ] Saurabh Agrawal commented on SPARK-21476: - [~srowen] Can a fix for this go in the

[jira] [Commented] (SPARK-21445) NotSerializableException thrown by UTF8String.IntWrapper

2017-07-25 Thread jin xing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101224#comment-16101224 ] jin xing commented on SPARK-21445: -- Sorry, I report the exception by mistake. With the c

[jira] [Created] (SPARK-21536) Remove the workaroud to allow dots in field names in R's createDataFame

2017-07-25 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-21536: Summary: Remove the workaroud to allow dots in field names in R's createDataFame Key: SPARK-21536 URL: https://issues.apache.org/jira/browse/SPARK-21536 Project: Spar

[jira] [Updated] (SPARK-21440) Refactor ArrowConverters and add ArrayType and StructType support.

2017-07-25 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-21440: -- Description: This is a refactoring of {{ArrowConverters}} and related classes. # Refactor {{Co

[jira] [Updated] (SPARK-21440) Refactor ArrowConverters and add ArrayType and StructType support.

2017-07-25 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-21440: -- Summary: Refactor ArrowConverters and add ArrayType and StructType support. (was: Refactor Arr

[jira] [Commented] (SPARK-19720) Redact sensitive information from SparkSubmit console output

2017-07-25 Thread Diogo Munaro Vieira (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101096#comment-16101096 ] Diogo Munaro Vieira commented on SPARK-19720: - Do you have plans to apply thi

[jira] [Closed] (SPARK-21527) Use buffer limit in order to take advantage of JAVA NIO Util's buffercache

2017-07-25 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang closed SPARK-21527. Resolution: Invalid > Use buffer limit in order to take advantage of JAVA NIO Util's buffercache > ---

[jira] [Resolved] (SPARK-13786) Pyspark ml.tuning support export/import

2017-07-25 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-13786. --- Resolution: Duplicate > Pyspark ml.tuning support export/import > ---

[jira] [Commented] (SPARK-13786) Pyspark ml.tuning support export/import

2017-07-25 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101057#comment-16101057 ] Joseph K. Bradley commented on SPARK-13786: --- This has been resolved now via [SP

[jira] [Assigned] (SPARK-21517) Fetch local data via block manager cause oom

2017-07-25 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-21517: Assignee: zhoukang > Fetch local data via block manager cause oom > --

[jira] [Resolved] (SPARK-21517) Fetch local data via block manager cause oom

2017-07-25 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-21517. -- Resolution: Fixed Fix Version/s: 2.3.0 > Fetch local data via block manager cause oom >

[jira] [Resolved] (SPARK-21494) Spark 2.2.0 AES encryption not working with External shuffle

2017-07-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-21494. Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 2.3.0

[jira] [Comment Edited] (SPARK-21530) Update description of spark.shuffle.maxChunksBeingTransferred

2017-07-25 Thread jin xing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100236#comment-16100236 ] jin xing edited comment on SPARK-21530 at 7/26/17 12:56 AM: I

[jira] [Commented] (SPARK-21190) SPIP: Vectorized UDFs in Python

2017-07-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101014#comment-16101014 ] Wenchen Fan commented on SPARK-21190: - I think (2) is already done by {{ArrowColumnVe

[jira] [Commented] (SPARK-21534) PickleException when creating dataframe from python row with empty bytearray

2017-07-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101009#comment-16101009 ] Hyukjin Kwon commented on SPARK-21534: -- cc [~zasdfgbnm] and [~ueshin], this one look

[jira] [Resolved] (SPARK-20586) Add deterministic to ScalaUDF

2017-07-25 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-20586. - Resolution: Fixed Fix Version/s: 2.3.0 > Add deterministic to ScalaUDF > -

[jira] [Updated] (SPARK-20586) Add deterministic to ScalaUDF

2017-07-25 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-20586: Summary: Add deterministic to ScalaUDF (was: Add deterministic and distinctLike to ScalaUDF) > Add determ

[jira] [Closed] (SPARK-21231) Conda install of packages during Jenkins testing is causing intermittent failure

2017-07-25 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler closed SPARK-21231. Resolved by https://github.com/apache/spark/pull/18459 > Conda install of packages during Jenkins test

[jira] [Resolved] (SPARK-21231) Conda install of packages during Jenkins testing is causing intermittent failure

2017-07-25 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler resolved SPARK-21231. -- Resolution: Resolved > Conda install of packages during Jenkins testing is causing intermittent

[jira] [Updated] (SPARK-20586) Add deterministic and distinctLike to ScalaUDF

2017-07-25 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-20586: Description: https://hive.apache.org/javadocs/r2.0.1/api/org/apache/hadoop/hive/ql/udf/UDFType.html Like H

[jira] [Commented] (SPARK-4820) Spark build encounters "File name too long" on some encrypted filesystems

2017-07-25 Thread antonkulaga (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100966#comment-16100966 ] antonkulaga commented on SPARK-4820: This issue is valid for Spark 2.2.0 on Ubuntu 16.

[jira] [Commented] (SPARK-18935) Use Mesos "Dynamic Reservation" resource for Spark

2017-07-25 Thread Ameen Akel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100863#comment-16100863 ] Ameen Akel commented on SPARK-18935: Although I'm only one data point: I'm interested

[jira] [Commented] (SPARK-21535) Reduce memory requirement for CrossValidator and TrainValidationSplit

2017-07-25 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100860#comment-16100860 ] yuhao yang commented on SPARK-21535: https://github.com/apache/spark/pulls > Reduce

[jira] [Updated] (SPARK-21535) Reduce memory requirement for CrossValidator and TrainValidationSplit

2017-07-25 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yuhao yang updated SPARK-21535: --- Description: CrossValidator and TrainValidationSplit both use {code}models = est.fit(trainingDataset

[jira] [Created] (SPARK-21535) Reduce memory requirement for CrossValidator and TrainValidationSplit

2017-07-25 Thread yuhao yang (JIRA)
yuhao yang created SPARK-21535: -- Summary: Reduce memory requirement for CrossValidator and TrainValidationSplit Key: SPARK-21535 URL: https://issues.apache.org/jira/browse/SPARK-21535 Project: Spark

[jira] [Updated] (SPARK-19526) Spark should raise an exception when it tries to read a Hive view but it doesn't have read access on the corresponding table(s)

2017-07-25 Thread Mark Grover (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Grover updated SPARK-19526: Summary: Spark should raise an exception when it tries to read a Hive view but it doesn't have read

[jira] [Updated] (SPARK-21534) PickleException when creating dataframe from python row with empty bytearray

2017-07-25 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-21534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Bryński updated SPARK-21534: --- Summary: PickleException when creating dataframe from python row with empty bytearray (was:

[jira] [Created] (SPARK-21534) Exception when creating dataframe from python row with empty bytearray

2017-07-25 Thread JIRA
Maciej Bryński created SPARK-21534: -- Summary: Exception when creating dataframe from python row with empty bytearray Key: SPARK-21534 URL: https://issues.apache.org/jira/browse/SPARK-21534 Project: S

[jira] [Resolved] (SPARK-21532) Improve console progress bar in RStudio

2017-07-25 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21532. --- Resolution: Not A Problem I don't think this is a Spark-related issue, so shouldn't be a JIRA here.

[jira] [Commented] (SPARK-21532) Improve console progress bar in RStudio

2017-07-25 Thread Brendan Dwyer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100806#comment-16100806 ] Brendan Dwyer commented on SPARK-21532: --- I've opened [an issue with RStudio|https:

[jira] [Resolved] (SPARK-21491) Performance enhancement: eliminate creation of intermediate collections

2017-07-25 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21491. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18693 [https://github.co

[jira] [Assigned] (SPARK-21491) Performance enhancement: eliminate creation of intermediate collections

2017-07-25 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-21491: - Assignee: Iurii Antykhovych > Performance enhancement: eliminate creation of intermediate collec

[jira] [Commented] (SPARK-21532) Improve console progress bar in RStudio

2017-07-25 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100717#comment-16100717 ] Sean Owen commented on SPARK-21532: --- Sure, but that's not a Spark issue. > Improve con

[jira] [Commented] (SPARK-21190) SPIP: Vectorized UDFs in Python

2017-07-25 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100712#comment-16100712 ] Li Jin commented on SPARK-21190: [~bryanc], I have looked at your PR at https://github.c

[jira] [Comment Edited] (SPARK-21190) SPIP: Vectorized UDFs in Python

2017-07-25 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100582#comment-16100582 ] Li Jin edited comment on SPARK-21190 at 7/25/17 8:17 PM: - I have

[jira] [Commented] (SPARK-21532) Improve console progress bar in RStudio

2017-07-25 Thread Brendan Dwyer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100695#comment-16100695 ] Brendan Dwyer commented on SPARK-21532: --- https://support.rstudio.com/hc/en-us/commu

[jira] [Commented] (SPARK-21532) Improve console progress bar in RStudio

2017-07-25 Thread Brendan Dwyer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100693#comment-16100693 ] Brendan Dwyer commented on SPARK-21532: --- [~srowen] I assume so. I think in it's cu

[jira] [Commented] (SPARK-21479) Outer join filter pushdown in null supplying table when condition is on one of the joined columns

2017-07-25 Thread Anton Okolnychyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100691#comment-16100691 ] Anton Okolnychyi commented on SPARK-21479: -- I used the following code to investi

[jira] [Commented] (SPARK-21501) Spark shuffle index cache size should be memory based

2017-07-25 Thread Sanket Reddy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100658#comment-16100658 ] Sanket Reddy commented on SPARK-21501: -- Hi I am working on this issue just to avoid

[jira] [Comment Edited] (SPARK-21190) SPIP: Vectorized UDFs in Python

2017-07-25 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100582#comment-16100582 ] Li Jin edited comment on SPARK-21190 at 7/25/17 7:12 PM: - I have

[jira] [Comment Edited] (SPARK-21190) SPIP: Vectorized UDFs in Python

2017-07-25 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100582#comment-16100582 ] Li Jin edited comment on SPARK-21190 at 7/25/17 7:12 PM: - I have

[jira] [Commented] (SPARK-21190) SPIP: Vectorized UDFs in Python

2017-07-25 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100582#comment-16100582 ] Li Jin commented on SPARK-21190: I have created this PR for the groupby().apply() use cas

[jira] [Commented] (SPARK-21532) Improve console progress bar in RStudio

2017-07-25 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100573#comment-16100573 ] Sean Owen commented on SPARK-21532: --- I think that's a function of RStudio and carriage

[jira] [Commented] (SPARK-21375) Add date and timestamp support to ArrowConverters for toPandas() collection

2017-07-25 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100572#comment-16100572 ] Bryan Cutler commented on SPARK-21375: -- Also, there has been some discussion about T

[jira] [Commented] (SPARK-20396) groupBy().apply() with pandas udf in pyspark

2017-07-25 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100562#comment-16100562 ] Li Jin commented on SPARK-20396: PR: https://github.com/apache/spark/pull/18732 > groupB

[jira] [Created] (SPARK-21533) "configure(...)" method not called when using Hive Generic UDFs

2017-07-25 Thread Dean Gurvitz (JIRA)
Dean Gurvitz created SPARK-21533: Summary: "configure(...)" method not called when using Hive Generic UDFs Key: SPARK-21533 URL: https://issues.apache.org/jira/browse/SPARK-21533 Project: Spark

[jira] [Resolved] (SPARK-11170) ​ EOFException on History server reading in progress lz4

2017-07-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-11170. Resolution: Duplicate > ​ EOFException on History server reading in progress lz4 >

[jira] [Resolved] (SPARK-21447) Spark history server fails to render compressed inprogress history file in some cases.

2017-07-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-21447. Resolution: Fixed Assignee: Eric Vandenberg Fix Version/s: 2.3.0

[jira] [Commented] (SPARK-11170) ​ EOFException on History server reading in progress lz4

2017-07-25 Thread Eric Vandenberg (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100537#comment-16100537 ] Eric Vandenberg commented on SPARK-11170: - There's a fix for this, see https://is

[jira] [Created] (SPARK-21532) Improve console progress bar in RStudio

2017-07-25 Thread Brendan Dwyer (JIRA)
Brendan Dwyer created SPARK-21532: - Summary: Improve console progress bar in RStudio Key: SPARK-21532 URL: https://issues.apache.org/jira/browse/SPARK-21532 Project: Spark Issue Type: Improve

[jira] [Resolved] (SPARK-21531) CLONE - Spark build encounters "File name too long" on some encrypted filesystems

2017-07-25 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21531. --- Resolution: Duplicate Fix Version/s: (was: 1.4.0) This should not be copied > CLONE - Spa

[jira] [Commented] (SPARK-21087) CrossValidator, TrainValidationSplit should preserve all models after fitting: Scala

2017-07-25 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100447#comment-16100447 ] yuhao yang commented on SPARK-21087: Withdrawing my PR, anyone with interests please

[jira] [Updated] (SPARK-20396) groupBy().apply() with pandas udf in pyspark

2017-07-25 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Jin updated SPARK-20396: --- Summary: groupBy().apply() with pandas udf in pyspark (was: Add support for pandas udf in pyspark) > groupB

[jira] [Comment Edited] (SPARK-21375) Add date and timestamp support to ArrowConverters for toPandas() collection

2017-07-25 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100434#comment-16100434 ] Bryan Cutler edited comment on SPARK-21375 at 7/25/17 5:50 PM:

[jira] [Comment Edited] (SPARK-21375) Add date and timestamp support to ArrowConverters for toPandas() collection

2017-07-25 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100434#comment-16100434 ] Bryan Cutler edited comment on SPARK-21375 at 7/25/17 5:49 PM:

[jira] [Commented] (SPARK-21375) Add date and timestamp support to ArrowConverters for toPandas() collection

2017-07-25 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100434#comment-16100434 ] Bryan Cutler commented on SPARK-21375: -- Thanks for the details [~wesmckinn]. The ap

[jira] [Updated] (SPARK-21175) shuffle service should reject fetch requests if there are already many requests in progress

2017-07-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-21175: Summary: shuffle service should reject fetch requests if there are already many requests in progres

[jira] [Updated] (SPARK-21531) CLONE - Spark build encounters "File name too long" on some encrypted filesystems

2017-07-25 Thread antonkulaga (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] antonkulaga updated SPARK-21531: Description: Originally this issue was discovere din Spark 1.x, then fixed in 1.x but it is still

[jira] [Updated] (SPARK-21531) CLONE - Spark build encounters "File name too long" on some encrypted filesystems

2017-07-25 Thread antonkulaga (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] antonkulaga updated SPARK-21531: Description: Originally this issue was discovere din Spark 1.x, then fixed in 1.x but it is still

[jira] [Updated] (SPARK-21531) CLONE - Spark build encounters "File name too long" on some encrypted filesystems

2017-07-25 Thread antonkulaga (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] antonkulaga updated SPARK-21531: Priority: Major (was: Minor) > CLONE - Spark build encounters "File name too long" on some encrypt

[jira] [Created] (SPARK-21531) CLONE - Spark build encounters "File name too long" on some encrypted filesystems

2017-07-25 Thread antonkulaga (JIRA)
antonkulaga created SPARK-21531: --- Summary: CLONE - Spark build encounters "File name too long" on some encrypted filesystems Key: SPARK-21531 URL: https://issues.apache.org/jira/browse/SPARK-21531 Proje

[jira] [Resolved] (SPARK-21383) YARN can allocate too many executors

2017-07-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-21383. Resolution: Fixed Assignee: DjvuLee Fix Version/s: 2.3.0

[jira] [Comment Edited] (SPARK-21375) Add date and timestamp support to ArrowConverters for toPandas() collection

2017-07-25 Thread Wes McKinney (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100371#comment-16100371 ] Wes McKinney edited comment on SPARK-21375 at 7/25/17 5:10 PM:

[jira] [Commented] (SPARK-21375) Add date and timestamp support to ArrowConverters for toPandas() collection

2017-07-25 Thread Wes McKinney (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100371#comment-16100371 ] Wes McKinney commented on SPARK-21375: -- What is the summary of how you're handling t

[jira] [Commented] (SPARK-19528) external shuffle service would close while still have request from executor when dynamic allocation is enabled

2017-07-25 Thread Rahij Ramsharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100357#comment-16100357 ] Rahij Ramsharan commented on SPARK-19528: - Hi, has this issue been resolved? Seei

[jira] [Commented] (SPARK-21521) History service requires user is in any group

2017-07-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100354#comment-16100354 ] Marcelo Vanzin commented on SPARK-21521: BTW I'd be ok with just properly documen

[jira] [Commented] (SPARK-21530) Update description of spark.shuffle.maxChunksBeingTransferred

2017-07-25 Thread jin xing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100236#comment-16100236 ] jin xing commented on SPARK-21530: -- I will send follow-up PR soon. Thanks [~tgraves] >

[jira] [Commented] (SPARK-21476) RandomForest classification model not using broadcast in transform

2017-07-25 Thread Saurabh Agrawal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100151#comment-16100151 ] Saurabh Agrawal commented on SPARK-21476: - I am using it in spark streaming where

[jira] [Commented] (SPARK-20307) SparkR: pass on setHandleInvalid to spark.mllib functions that use StringIndexer

2017-07-25 Thread Joseph Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100114#comment-16100114 ] Joseph Wang commented on SPARK-20307: - I am testing in pyspark now. However, handleIn

[jira] [Created] (SPARK-21530) Update description of spark.shuffle.maxChunksBeingTransferred

2017-07-25 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-21530: - Summary: Update description of spark.shuffle.maxChunksBeingTransferred Key: SPARK-21530 URL: https://issues.apache.org/jira/browse/SPARK-21530 Project: Spark

[jira] [Commented] (SPARK-20592) Alter table concatenate is not working as expected.

2017-07-25 Thread Saksham Srivastava (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100098#comment-16100098 ] Saksham Srivastava commented on SPARK-20592: Seeing the same error when using

[jira] [Commented] (SPARK-15142) Spark Mesos dispatcher becomes unusable when the Mesos master restarts

2017-07-25 Thread Devaraj K (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100079#comment-16100079 ] Devaraj K commented on SPARK-15142: --- bq. That means there is no way to detect the new m

[jira] [Commented] (SPARK-9776) Another instance of Derby may have already booted the database

2017-07-25 Thread Quincy HSIEH (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100019#comment-16100019 ] Quincy HSIEH commented on SPARK-9776: - Hi, I have this problem when I try to run Spar

[jira] [Commented] (SPARK-18935) Use Mesos "Dynamic Reservation" resource for Spark

2017-07-25 Thread Arthur Rand (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100016#comment-16100016 ] Arthur Rand commented on SPARK-18935: - Are people still interested in this being fixe

[jira] [Commented] (SPARK-21501) Spark shuffle index cache size should be memory based

2017-07-25 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1616#comment-1616 ] Thomas Graves commented on SPARK-21501: --- We want to change it from a # of entries t

[jira] [Assigned] (SPARK-21175) Slow down "open blocks" on shuffle service when memory shortage to avoid OOM.

2017-07-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-21175: --- Assignee: jin xing > Slow down "open blocks" on shuffle service when memory shortage to avoi

[jira] [Resolved] (SPARK-21175) Slow down "open blocks" on shuffle service when memory shortage to avoid OOM.

2017-07-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21175. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18388 [https://githu

[jira] [Created] (SPARK-21529) Uniontype not supported when reading from Hive tables.

2017-07-25 Thread Elliot West (JIRA)
Elliot West created SPARK-21529: --- Summary: Uniontype not supported when reading from Hive tables. Key: SPARK-21529 URL: https://issues.apache.org/jira/browse/SPARK-21529 Project: Spark Issue Ty

[jira] [Updated] (SPARK-21529) Uniontype not supported when reading from Hive tables.

2017-07-25 Thread Elliot West (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliot West updated SPARK-21529: Description: We encounter errors when attempting to read Hive tables whose schema contains the {{u

[jira] [Commented] (SPARK-21498) quick start -> one py demo have some bug in code

2017-07-25 Thread LiZhaochuan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099947#comment-16099947 ] LiZhaochuan commented on SPARK-21498: - I am reading your book , very good > quick

[jira] [Comment Edited] (SPARK-15142) Spark Mesos dispatcher becomes unusable when the Mesos master restarts

2017-07-25 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099923#comment-16099923 ] Stavros Kontopoulos edited comment on SPARK-15142 at 7/25/17 12:10 PM:

[jira] [Comment Edited] (SPARK-15142) Spark Mesos dispatcher becomes unusable when the Mesos master restarts

2017-07-25 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099923#comment-16099923 ] Stavros Kontopoulos edited comment on SPARK-15142 at 7/25/17 12:10 PM:

[jira] [Comment Edited] (SPARK-15142) Spark Mesos dispatcher becomes unusable when the Mesos master restarts

2017-07-25 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099923#comment-16099923 ] Stavros Kontopoulos edited comment on SPARK-15142 at 7/25/17 12:07 PM:

[jira] [Comment Edited] (SPARK-15142) Spark Mesos dispatcher becomes unusable when the Mesos master restarts

2017-07-25 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099923#comment-16099923 ] Stavros Kontopoulos edited comment on SPARK-15142 at 7/25/17 12:05 PM:

[jira] [Comment Edited] (SPARK-15142) Spark Mesos dispatcher becomes unusable when the Mesos master restarts

2017-07-25 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099923#comment-16099923 ] Stavros Kontopoulos edited comment on SPARK-15142 at 7/25/17 12:04 PM:

[jira] [Comment Edited] (SPARK-15142) Spark Mesos dispatcher becomes unusable when the Mesos master restarts

2017-07-25 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099923#comment-16099923 ] Stavros Kontopoulos edited comment on SPARK-15142 at 7/25/17 11:59 AM:

[jira] [Commented] (SPARK-15142) Spark Mesos dispatcher becomes unusable when the Mesos master restarts

2017-07-25 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099923#comment-16099923 ] Stavros Kontopoulos commented on SPARK-15142: - [~devaraj.k] Great, so one que

[jira] [Commented] (SPARK-21445) NotSerializableException thrown by UTF8String.IntWrapper

2017-07-25 Thread jin xing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099916#comment-16099916 ] jin xing commented on SPARK-21445: -- I'm not sure how to reproduce, I will try. > NotSer

[jira] [Resolved] (SPARK-21528) spark failed with native memory exhausted , need immediate attention

2017-07-25 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21528. --- Resolution: Invalid Please direct questions like this to StackOverflow > spark failed with native me

[jira] [Updated] (SPARK-21528) spark failed with native memory exhausted , need immediate attention

2017-07-25 Thread Mansr (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mansr updated SPARK-21528: -- Attachment: hs_err_pid1004.log > spark failed with native memory exhausted , need immediate attention > ---

[jira] [Created] (SPARK-21528) spark failed with native memory exhausted , need immediate attention

2017-07-25 Thread Mansr (JIRA)
Mansr created SPARK-21528: - Summary: spark failed with native memory exhausted , need immediate attention Key: SPARK-21528 URL: https://issues.apache.org/jira/browse/SPARK-21528 Project: Spark Issue

[jira] [Commented] (SPARK-21496) Support codegen for TakeOrderedAndProjectExec

2017-07-25 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099840#comment-16099840 ] Kazuaki Ishizaki commented on SPARK-21496: -- Is there any good benchmark program

[jira] [Updated] (SPARK-21527) Use buffer limit in order to take advantage of JAVA NIO Util's buffercache

2017-07-25 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated SPARK-21527: - Summary: Use buffer limit in order to take advantage of JAVA NIO Util's buffercache (was: Use buffer li

[jira] [Created] (SPARK-21527) Use buffer limit in order to use JAVA NIO Util's buffercache

2017-07-25 Thread zhoukang (JIRA)
zhoukang created SPARK-21527: Summary: Use buffer limit in order to use JAVA NIO Util's buffercache Key: SPARK-21527 URL: https://issues.apache.org/jira/browse/SPARK-21527 Project: Spark Issue T

[jira] [Commented] (SPARK-21498) quick start -> one py demo have some bug in code

2017-07-25 Thread LiZhaochuan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099811#comment-16099811 ] LiZhaochuan commented on SPARK-21498: - When can I see the changes submitted? :D:D:D h

[jira] [Assigned] (SPARK-21498) quick start -> one py demo have some bug in code

2017-07-25 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-21498: - Assignee: LiZhaochuan > quick start -> one py demo have some bug in code > --

[jira] [Resolved] (SPARK-21498) quick start -> one py demo have some bug in code

2017-07-25 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21498. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18722 [https://github.co

[jira] [Commented] (SPARK-21445) NotSerializableException thrown by UTF8String.IntWrapper

2017-07-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099796#comment-16099796 ] Wenchen Fan commented on SPARK-21445: - [~jinxing6...@126.com] can you post the code s

[jira] [Updated] (SPARK-21402) Java encoders - switch fields on collectAsList

2017-07-25 Thread Tom (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom updated SPARK-21402: Priority: Major (was: Minor) > Java encoders - switch fields on collectAsList > --

  1   2   >