[jira] [Updated] (SPARK-21440) Refactor ArrowConverters and add ArrayType and StructType support.

[ https://issues.apache.org/jira/browse/SPARK-21440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-21440: -- Description: This is a refactoring of {{ArrowConverters}} and related classes. # Refactor

[jira] [Updated] (SPARK-21440) Refactor ArrowConverters and add ArrayType and StructType support.

[ https://issues.apache.org/jira/browse/SPARK-21440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-21440: -- Summary: Refactor ArrowConverters and add ArrayType and StructType support. (was: Refactor

[jira] [Commented] (SPARK-19720) Redact sensitive information from SparkSubmit console output

[ https://issues.apache.org/jira/browse/SPARK-19720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16101096#comment-16101096 ] Diogo Munaro Vieira commented on SPARK-19720: - Do you have plans to apply this fix in a

[jira] [Closed] (SPARK-21527) Use buffer limit in order to take advantage of JAVA NIO Util's buffercache

[ https://issues.apache.org/jira/browse/SPARK-21527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang closed SPARK-21527. Resolution: Invalid > Use buffer limit in order to take advantage of JAVA NIO Util's buffercache >

[jira] [Resolved] (SPARK-13786) Pyspark ml.tuning support export/import

[ https://issues.apache.org/jira/browse/SPARK-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-13786. --- Resolution: Duplicate > Pyspark ml.tuning support export/import >

[jira] [Commented] (SPARK-13786) Pyspark ml.tuning support export/import

[ https://issues.apache.org/jira/browse/SPARK-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16101057#comment-16101057 ] Joseph K. Bradley commented on SPARK-13786: --- This has been resolved now via [SPARK-11893],

[jira] [Assigned] (SPARK-21517) Fetch local data via block manager cause oom

[ https://issues.apache.org/jira/browse/SPARK-21517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-21517: Assignee: zhoukang > Fetch local data via block manager cause oom >

[jira] [Resolved] (SPARK-21517) Fetch local data via block manager cause oom

[ https://issues.apache.org/jira/browse/SPARK-21517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-21517. -- Resolution: Fixed Fix Version/s: 2.3.0 > Fetch local data via block manager cause oom >

[jira] [Resolved] (SPARK-21494) Spark 2.2.0 AES encryption not working with External shuffle

[ https://issues.apache.org/jira/browse/SPARK-21494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-21494. Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 2.3.0

[jira] [Comment Edited] (SPARK-21530) Update description of spark.shuffle.maxChunksBeingTransferred

[ https://issues.apache.org/jira/browse/SPARK-21530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100236#comment-16100236 ] jin xing edited comment on SPARK-21530 at 7/26/17 12:56 AM: I will send

[jira] [Commented] (SPARK-21190) SPIP: Vectorized UDFs in Python

[ https://issues.apache.org/jira/browse/SPARK-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16101014#comment-16101014 ] Wenchen Fan commented on SPARK-21190: - I think (2) is already done by {{ArrowColumnVector}} (written

[jira] [Commented] (SPARK-21534) PickleException when creating dataframe from python row with empty bytearray

[ https://issues.apache.org/jira/browse/SPARK-21534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16101009#comment-16101009 ] Hyukjin Kwon commented on SPARK-21534: -- cc [~zasdfgbnm] and [~ueshin], this one looks related with

[jira] [Resolved] (SPARK-20586) Add deterministic to ScalaUDF

[ https://issues.apache.org/jira/browse/SPARK-20586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-20586. - Resolution: Fixed Fix Version/s: 2.3.0 > Add deterministic to ScalaUDF >

[jira] [Updated] (SPARK-20586) Add deterministic to ScalaUDF

[ https://issues.apache.org/jira/browse/SPARK-20586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-20586: Summary: Add deterministic to ScalaUDF (was: Add deterministic and distinctLike to ScalaUDF) > Add

[jira] [Closed] (SPARK-21231) Conda install of packages during Jenkins testing is causing intermittent failure

[ https://issues.apache.org/jira/browse/SPARK-21231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler closed SPARK-21231. Resolved by https://github.com/apache/spark/pull/18459 > Conda install of packages during Jenkins

[jira] [Resolved] (SPARK-21231) Conda install of packages during Jenkins testing is causing intermittent failure

[ https://issues.apache.org/jira/browse/SPARK-21231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler resolved SPARK-21231. -- Resolution: Resolved > Conda install of packages during Jenkins testing is causing

[jira] [Updated] (SPARK-20586) Add deterministic and distinctLike to ScalaUDF

[ https://issues.apache.org/jira/browse/SPARK-20586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-20586: Description: https://hive.apache.org/javadocs/r2.0.1/api/org/apache/hadoop/hive/ql/udf/UDFType.html Like

[jira] [Commented] (SPARK-4820) Spark build encounters "File name too long" on some encrypted filesystems

[ https://issues.apache.org/jira/browse/SPARK-4820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100966#comment-16100966 ] antonkulaga commented on SPARK-4820: This issue is valid for Spark 2.2.0 on Ubuntu 16.04 and it is a

[jira] [Commented] (SPARK-18935) Use Mesos "Dynamic Reservation" resource for Spark

[ https://issues.apache.org/jira/browse/SPARK-18935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100863#comment-16100863 ] Ameen Akel commented on SPARK-18935: Although I'm only one data point: I'm interested in this being

[jira] [Commented] (SPARK-21535) Reduce memory requirement for CrossValidator and TrainValidationSplit

[ https://issues.apache.org/jira/browse/SPARK-21535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100860#comment-16100860 ] yuhao yang commented on SPARK-21535: https://github.com/apache/spark/pulls > Reduce memory

[jira] [Updated] (SPARK-21535) Reduce memory requirement for CrossValidator and TrainValidationSplit

[ https://issues.apache.org/jira/browse/SPARK-21535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yuhao yang updated SPARK-21535: --- Description: CrossValidator and TrainValidationSplit both use {code}models =

[jira] [Created] (SPARK-21535) Reduce memory requirement for CrossValidator and TrainValidationSplit

yuhao yang created SPARK-21535: -- Summary: Reduce memory requirement for CrossValidator and TrainValidationSplit Key: SPARK-21535 URL: https://issues.apache.org/jira/browse/SPARK-21535 Project: Spark

[jira] [Updated] (SPARK-19526) Spark should raise an exception when it tries to read a Hive view but it doesn't have read access on the corresponding table(s)

[ https://issues.apache.org/jira/browse/SPARK-19526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Grover updated SPARK-19526: Summary: Spark should raise an exception when it tries to read a Hive view but it doesn't have

[jira] [Updated] (SPARK-21534) PickleException when creating dataframe from python row with empty bytearray

[ https://issues.apache.org/jira/browse/SPARK-21534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Bryński updated SPARK-21534: --- Summary: PickleException when creating dataframe from python row with empty bytearray (was:

[jira] [Created] (SPARK-21534) Exception when creating dataframe from python row with empty bytearray

Maciej Bryński created SPARK-21534: -- Summary: Exception when creating dataframe from python row with empty bytearray Key: SPARK-21534 URL: https://issues.apache.org/jira/browse/SPARK-21534 Project:

[jira] [Resolved] (SPARK-21532) Improve console progress bar in RStudio

[ https://issues.apache.org/jira/browse/SPARK-21532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21532. --- Resolution: Not A Problem I don't think this is a Spark-related issue, so shouldn't be a JIRA here.

[jira] [Commented] (SPARK-21532) Improve console progress bar in RStudio

[ https://issues.apache.org/jira/browse/SPARK-21532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100806#comment-16100806 ] Brendan Dwyer commented on SPARK-21532: --- I've opened [an issue with

[jira] [Resolved] (SPARK-21491) Performance enhancement: eliminate creation of intermediate collections

[ https://issues.apache.org/jira/browse/SPARK-21491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21491. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18693

[jira] [Assigned] (SPARK-21491) Performance enhancement: eliminate creation of intermediate collections

[ https://issues.apache.org/jira/browse/SPARK-21491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-21491: - Assignee: Iurii Antykhovych > Performance enhancement: eliminate creation of intermediate

[jira] [Commented] (SPARK-21532) Improve console progress bar in RStudio

[ https://issues.apache.org/jira/browse/SPARK-21532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100717#comment-16100717 ] Sean Owen commented on SPARK-21532: --- Sure, but that's not a Spark issue. > Improve console progress

[jira] [Commented] (SPARK-21190) SPIP: Vectorized UDFs in Python

[ https://issues.apache.org/jira/browse/SPARK-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100712#comment-16100712 ] Li Jin commented on SPARK-21190: [~bryanc], I have looked at your PR at

[jira] [Comment Edited] (SPARK-21190) SPIP: Vectorized UDFs in Python

[ https://issues.apache.org/jira/browse/SPARK-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100582#comment-16100582 ] Li Jin edited comment on SPARK-21190 at 7/25/17 8:17 PM: - I have created this PR

[jira] [Commented] (SPARK-21532) Improve console progress bar in RStudio

[ https://issues.apache.org/jira/browse/SPARK-21532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100695#comment-16100695 ] Brendan Dwyer commented on SPARK-21532: ---

[jira] [Commented] (SPARK-21532) Improve console progress bar in RStudio

[ https://issues.apache.org/jira/browse/SPARK-21532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100693#comment-16100693 ] Brendan Dwyer commented on SPARK-21532: --- [~srowen] I assume so. I think in it's current state the

[jira] [Commented] (SPARK-21479) Outer join filter pushdown in null supplying table when condition is on one of the joined columns

[ https://issues.apache.org/jira/browse/SPARK-21479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100691#comment-16100691 ] Anton Okolnychyi commented on SPARK-21479: -- I used the following code to investigate: {code}

[jira] [Commented] (SPARK-21501) Spark shuffle index cache size should be memory based

[ https://issues.apache.org/jira/browse/SPARK-21501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100658#comment-16100658 ] Sanket Reddy commented on SPARK-21501: -- Hi I am working on this issue just to avoid any redundancies

[jira] [Comment Edited] (SPARK-21190) SPIP: Vectorized UDFs in Python

[ https://issues.apache.org/jira/browse/SPARK-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100582#comment-16100582 ] Li Jin edited comment on SPARK-21190 at 7/25/17 7:12 PM: - I have created this PR

[jira] [Comment Edited] (SPARK-21190) SPIP: Vectorized UDFs in Python

[ https://issues.apache.org/jira/browse/SPARK-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100582#comment-16100582 ] Li Jin edited comment on SPARK-21190 at 7/25/17 7:12 PM: - I have created this PR

[jira] [Commented] (SPARK-21190) SPIP: Vectorized UDFs in Python

[ https://issues.apache.org/jira/browse/SPARK-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100582#comment-16100582 ] Li Jin commented on SPARK-21190: I have created this PR for the groupby().apply() use case with pandas

[jira] [Commented] (SPARK-21532) Improve console progress bar in RStudio

[ https://issues.apache.org/jira/browse/SPARK-21532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100573#comment-16100573 ] Sean Owen commented on SPARK-21532: --- I think that's a function of RStudio and carriage returns right?

[jira] [Commented] (SPARK-21375) Add date and timestamp support to ArrowConverters for toPandas() collection

[ https://issues.apache.org/jira/browse/SPARK-21375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100572#comment-16100572 ] Bryan Cutler commented on SPARK-21375: -- Also, there has been some discussion about Timestamps on the

[jira] [Commented] (SPARK-20396) groupBy().apply() with pandas udf in pyspark

[ https://issues.apache.org/jira/browse/SPARK-20396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100562#comment-16100562 ] Li Jin commented on SPARK-20396: PR: https://github.com/apache/spark/pull/18732 > groupBy().apply() with

[jira] [Created] (SPARK-21533) "configure(...)" method not called when using Hive Generic UDFs

Dean Gurvitz created SPARK-21533: Summary: "configure(...)" method not called when using Hive Generic UDFs Key: SPARK-21533 URL: https://issues.apache.org/jira/browse/SPARK-21533 Project: Spark

[jira] [Resolved] (SPARK-11170) ​ EOFException on History server reading in progress lz4

[ https://issues.apache.org/jira/browse/SPARK-11170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-11170. Resolution: Duplicate > ​ EOFException on History server reading in progress lz4 >

[jira] [Resolved] (SPARK-21447) Spark history server fails to render compressed inprogress history file in some cases.

[ https://issues.apache.org/jira/browse/SPARK-21447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-21447. Resolution: Fixed Assignee: Eric Vandenberg Fix Version/s: 2.3.0

[jira] [Commented] (SPARK-11170) ​ EOFException on History server reading in progress lz4

[ https://issues.apache.org/jira/browse/SPARK-11170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100537#comment-16100537 ] Eric Vandenberg commented on SPARK-11170: - There's a fix for this, see

[jira] [Created] (SPARK-21532) Improve console progress bar in RStudio

Brendan Dwyer created SPARK-21532: - Summary: Improve console progress bar in RStudio Key: SPARK-21532 URL: https://issues.apache.org/jira/browse/SPARK-21532 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-21531) CLONE - Spark build encounters "File name too long" on some encrypted filesystems

[ https://issues.apache.org/jira/browse/SPARK-21531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21531. --- Resolution: Duplicate Fix Version/s: (was: 1.4.0) This should not be copied > CLONE -

[jira] [Commented] (SPARK-21087) CrossValidator, TrainValidationSplit should preserve all models after fitting: Scala

[ https://issues.apache.org/jira/browse/SPARK-21087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100447#comment-16100447 ] yuhao yang commented on SPARK-21087: Withdrawing my PR, anyone with interests please go ahead and

[jira] [Updated] (SPARK-20396) groupBy().apply() with pandas udf in pyspark

[ https://issues.apache.org/jira/browse/SPARK-20396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Jin updated SPARK-20396: --- Summary: groupBy().apply() with pandas udf in pyspark (was: Add support for pandas udf in pyspark) >

[jira] [Comment Edited] (SPARK-21375) Add date and timestamp support to ArrowConverters for toPandas() collection

[ https://issues.apache.org/jira/browse/SPARK-21375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100434#comment-16100434 ] Bryan Cutler edited comment on SPARK-21375 at 7/25/17 5:50 PM: --- Thanks for

[jira] [Comment Edited] (SPARK-21375) Add date and timestamp support to ArrowConverters for toPandas() collection

[ https://issues.apache.org/jira/browse/SPARK-21375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100434#comment-16100434 ] Bryan Cutler edited comment on SPARK-21375 at 7/25/17 5:49 PM: --- Thanks for

[jira] [Commented] (SPARK-21375) Add date and timestamp support to ArrowConverters for toPandas() collection

[ https://issues.apache.org/jira/browse/SPARK-21375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100434#comment-16100434 ] Bryan Cutler commented on SPARK-21375: -- Thanks for the details [~wesmckinn]. The approach that

[jira] [Updated] (SPARK-21175) shuffle service should reject fetch requests if there are already many requests in progress

[ https://issues.apache.org/jira/browse/SPARK-21175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-21175: Summary: shuffle service should reject fetch requests if there are already many requests in

[jira] [Updated] (SPARK-21531) CLONE - Spark build encounters "File name too long" on some encrypted filesystems

[ https://issues.apache.org/jira/browse/SPARK-21531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] antonkulaga updated SPARK-21531: Description: Originally this issue was discovere din Spark 1.x, then fixed in 1.x but it is still

[jira] [Updated] (SPARK-21531) CLONE - Spark build encounters "File name too long" on some encrypted filesystems

[ https://issues.apache.org/jira/browse/SPARK-21531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] antonkulaga updated SPARK-21531: Description: Originally this issue was discovere din Spark 1.x, then fixed in 1.x but it is still

[jira] [Updated] (SPARK-21531) CLONE - Spark build encounters "File name too long" on some encrypted filesystems

[ https://issues.apache.org/jira/browse/SPARK-21531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] antonkulaga updated SPARK-21531: Priority: Major (was: Minor) > CLONE - Spark build encounters "File name too long" on some

[jira] [Created] (SPARK-21531) CLONE - Spark build encounters "File name too long" on some encrypted filesystems

antonkulaga created SPARK-21531: --- Summary: CLONE - Spark build encounters "File name too long" on some encrypted filesystems Key: SPARK-21531 URL: https://issues.apache.org/jira/browse/SPARK-21531

[jira] [Resolved] (SPARK-21383) YARN can allocate too many executors

[ https://issues.apache.org/jira/browse/SPARK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-21383. Resolution: Fixed Assignee: DjvuLee Fix Version/s: 2.3.0

[jira] [Comment Edited] (SPARK-21375) Add date and timestamp support to ArrowConverters for toPandas() collection

[ https://issues.apache.org/jira/browse/SPARK-21375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100371#comment-16100371 ] Wes McKinney edited comment on SPARK-21375 at 7/25/17 5:10 PM: --- What is the

[jira] [Commented] (SPARK-21375) Add date and timestamp support to ArrowConverters for toPandas() collection

[ https://issues.apache.org/jira/browse/SPARK-21375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100371#comment-16100371 ] Wes McKinney commented on SPARK-21375: -- What is the summary of how you're handling the time zone

[jira] [Commented] (SPARK-19528) external shuffle service would close while still have request from executor when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100357#comment-16100357 ] Rahij Ramsharan commented on SPARK-19528: - Hi, has this issue been resolved? Seeing something

[jira] [Commented] (SPARK-21521) History service requires user is in any group

[ https://issues.apache.org/jira/browse/SPARK-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100354#comment-16100354 ] Marcelo Vanzin commented on SPARK-21521: BTW I'd be ok with just properly documenting the

[jira] [Commented] (SPARK-21530) Update description of spark.shuffle.maxChunksBeingTransferred

[ https://issues.apache.org/jira/browse/SPARK-21530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100236#comment-16100236 ] jin xing commented on SPARK-21530: -- I will send follow-up PR soon. Thanks [~tgraves] > Update

[jira] [Commented] (SPARK-21476) RandomForest classification model not using broadcast in transform

[ https://issues.apache.org/jira/browse/SPARK-21476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100151#comment-16100151 ] Saurabh Agrawal commented on SPARK-21476: - I am using it in spark streaming where I give 16 cores

[jira] [Commented] (SPARK-20307) SparkR: pass on setHandleInvalid to spark.mllib functions that use StringIndexer

[ https://issues.apache.org/jira/browse/SPARK-20307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100114#comment-16100114 ] Joseph Wang commented on SPARK-20307: - I am testing in pyspark now. However, handleInvalid= ‘skip’

[jira] [Created] (SPARK-21530) Update description of spark.shuffle.maxChunksBeingTransferred

Thomas Graves created SPARK-21530: - Summary: Update description of spark.shuffle.maxChunksBeingTransferred Key: SPARK-21530 URL: https://issues.apache.org/jira/browse/SPARK-21530 Project: Spark

[jira] [Commented] (SPARK-20592) Alter table concatenate is not working as expected.

[ https://issues.apache.org/jira/browse/SPARK-20592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100098#comment-16100098 ] Saksham Srivastava commented on SPARK-20592: Seeing the same error when using alter-table in

[jira] [Commented] (SPARK-15142) Spark Mesos dispatcher becomes unusable when the Mesos master restarts

[ https://issues.apache.org/jira/browse/SPARK-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100079#comment-16100079 ] Devaraj K commented on SPARK-15142: --- bq. That means there is no way to detect the new master while the

[jira] [Commented] (SPARK-9776) Another instance of Derby may have already booted the database

[ https://issues.apache.org/jira/browse/SPARK-9776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100019#comment-16100019 ] Quincy HSIEH commented on SPARK-9776: - Hi, I have this problem when I try to run SparkR shell and

[jira] [Commented] (SPARK-18935) Use Mesos "Dynamic Reservation" resource for Spark

[ https://issues.apache.org/jira/browse/SPARK-18935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100016#comment-16100016 ] Arthur Rand commented on SPARK-18935: - Are people still interested in this being fixed? I'm going to

[jira] [Commented] (SPARK-21501) Spark shuffle index cache size should be memory based

[ https://issues.apache.org/jira/browse/SPARK-21501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1616#comment-1616 ] Thomas Graves commented on SPARK-21501: --- We want to change it from a # of entries to a size of

[jira] [Assigned] (SPARK-21175) Slow down "open blocks" on shuffle service when memory shortage to avoid OOM.

[ https://issues.apache.org/jira/browse/SPARK-21175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-21175: --- Assignee: jin xing > Slow down "open blocks" on shuffle service when memory shortage to

[jira] [Resolved] (SPARK-21175) Slow down "open blocks" on shuffle service when memory shortage to avoid OOM.

[ https://issues.apache.org/jira/browse/SPARK-21175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21175. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18388

[jira] [Created] (SPARK-21529) Uniontype not supported when reading from Hive tables.

Elliot West created SPARK-21529: --- Summary: Uniontype not supported when reading from Hive tables. Key: SPARK-21529 URL: https://issues.apache.org/jira/browse/SPARK-21529 Project: Spark Issue

[jira] [Updated] (SPARK-21529) Uniontype not supported when reading from Hive tables.

[ https://issues.apache.org/jira/browse/SPARK-21529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliot West updated SPARK-21529: Description: We encounter errors when attempting to read Hive tables whose schema contains the

[jira] [Commented] (SPARK-21498) quick start -> one py demo have some bug in code

[ https://issues.apache.org/jira/browse/SPARK-21498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099947#comment-16099947 ] LiZhaochuan commented on SPARK-21498: - I am reading your book , very good > quick start -> one

[jira] [Comment Edited] (SPARK-15142) Spark Mesos dispatcher becomes unusable when the Mesos master restarts

[ https://issues.apache.org/jira/browse/SPARK-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099923#comment-16099923 ] Stavros Kontopoulos edited comment on SPARK-15142 at 7/25/17 12:10 PM:

[jira] [Comment Edited] (SPARK-15142) Spark Mesos dispatcher becomes unusable when the Mesos master restarts

[ https://issues.apache.org/jira/browse/SPARK-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099923#comment-16099923 ] Stavros Kontopoulos edited comment on SPARK-15142 at 7/25/17 12:10 PM:

[jira] [Comment Edited] (SPARK-15142) Spark Mesos dispatcher becomes unusable when the Mesos master restarts

[ https://issues.apache.org/jira/browse/SPARK-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099923#comment-16099923 ] Stavros Kontopoulos edited comment on SPARK-15142 at 7/25/17 12:07 PM:

[jira] [Comment Edited] (SPARK-15142) Spark Mesos dispatcher becomes unusable when the Mesos master restarts

[ https://issues.apache.org/jira/browse/SPARK-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099923#comment-16099923 ] Stavros Kontopoulos edited comment on SPARK-15142 at 7/25/17 12:05 PM:

[jira] [Comment Edited] (SPARK-15142) Spark Mesos dispatcher becomes unusable when the Mesos master restarts

[ https://issues.apache.org/jira/browse/SPARK-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099923#comment-16099923 ] Stavros Kontopoulos edited comment on SPARK-15142 at 7/25/17 12:04 PM:

[jira] [Comment Edited] (SPARK-15142) Spark Mesos dispatcher becomes unusable when the Mesos master restarts

[ https://issues.apache.org/jira/browse/SPARK-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099923#comment-16099923 ] Stavros Kontopoulos edited comment on SPARK-15142 at 7/25/17 11:59 AM:

[jira] [Commented] (SPARK-15142) Spark Mesos dispatcher becomes unusable when the Mesos master restarts

[ https://issues.apache.org/jira/browse/SPARK-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099923#comment-16099923 ] Stavros Kontopoulos commented on SPARK-15142: - [~devaraj.k] Great, so one question to clarify

[jira] [Commented] (SPARK-21445) NotSerializableException thrown by UTF8String.IntWrapper

[ https://issues.apache.org/jira/browse/SPARK-21445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099916#comment-16099916 ] jin xing commented on SPARK-21445: -- I'm not sure how to reproduce, I will try. >

[jira] [Resolved] (SPARK-21528) spark failed with native memory exhausted , need immediate attention

[ https://issues.apache.org/jira/browse/SPARK-21528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21528. --- Resolution: Invalid Please direct questions like this to StackOverflow > spark failed with native

[jira] [Updated] (SPARK-21528) spark failed with native memory exhausted , need immediate attention

[ https://issues.apache.org/jira/browse/SPARK-21528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mansr updated SPARK-21528: -- Attachment: hs_err_pid1004.log > spark failed with native memory exhausted , need immediate attention >

[jira] [Created] (SPARK-21528) spark failed with native memory exhausted , need immediate attention

Mansr created SPARK-21528: - Summary: spark failed with native memory exhausted , need immediate attention Key: SPARK-21528 URL: https://issues.apache.org/jira/browse/SPARK-21528 Project: Spark

[jira] [Commented] (SPARK-21496) Support codegen for TakeOrderedAndProjectExec

[ https://issues.apache.org/jira/browse/SPARK-21496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099840#comment-16099840 ] Kazuaki Ishizaki commented on SPARK-21496: -- Is there any good benchmark program for this? >

[jira] [Updated] (SPARK-21527) Use buffer limit in order to take advantage of JAVA NIO Util's buffercache

[ https://issues.apache.org/jira/browse/SPARK-21527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated SPARK-21527: - Summary: Use buffer limit in order to take advantage of JAVA NIO Util's buffercache (was: Use buffer

[jira] [Created] (SPARK-21527) Use buffer limit in order to use JAVA NIO Util's buffercache

zhoukang created SPARK-21527: Summary: Use buffer limit in order to use JAVA NIO Util's buffercache Key: SPARK-21527 URL: https://issues.apache.org/jira/browse/SPARK-21527 Project: Spark Issue

[jira] [Commented] (SPARK-21498) quick start -> one py demo have some bug in code

[ https://issues.apache.org/jira/browse/SPARK-21498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099811#comment-16099811 ] LiZhaochuan commented on SPARK-21498: - When can I see the changes submitted? :D:D:D

[jira] [Assigned] (SPARK-21498) quick start -> one py demo have some bug in code

[ https://issues.apache.org/jira/browse/SPARK-21498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-21498: - Assignee: LiZhaochuan > quick start -> one py demo have some bug in code >

[jira] [Resolved] (SPARK-21498) quick start -> one py demo have some bug in code

[ https://issues.apache.org/jira/browse/SPARK-21498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21498. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18722

[jira] [Commented] (SPARK-21445) NotSerializableException thrown by UTF8String.IntWrapper

[ https://issues.apache.org/jira/browse/SPARK-21445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099796#comment-16099796 ] Wenchen Fan commented on SPARK-21445: - [~jinxing6...@126.com] can you post the code snippet to

[jira] [Updated] (SPARK-21402) Java encoders - switch fields on collectAsList

[ https://issues.apache.org/jira/browse/SPARK-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom updated SPARK-21402: Priority: Major (was: Minor) > Java encoders - switch fields on collectAsList >

[jira] [Commented] (SPARK-21445) NotSerializableException thrown by UTF8String.IntWrapper

[ https://issues.apache.org/jira/browse/SPARK-21445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099722#comment-16099722 ] jin xing commented on SPARK-21445: -- With this change, I'm still seeing exception below:

[jira] [Comment Edited] (SPARK-12261) pyspark crash for large dataset

[ https://issues.apache.org/jira/browse/SPARK-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099686#comment-16099686 ] Paul Magnus Sørensen-Clark edited comment on SPARK-12261 at 7/25/17 8:10 AM:

[jira] [Commented] (SPARK-21517) Fetch local data via block manager cause oom

[ https://issues.apache.org/jira/browse/SPARK-21517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099685#comment-16099685 ] zhoukang commented on SPARK-21517: -- [~kiszk] In our production cluster we use 1.6.1 and 2.1.0 which all

[jira] [Commented] (SPARK-12261) pyspark crash for large dataset

[ https://issues.apache.org/jira/browse/SPARK-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099686#comment-16099686 ] Paul Magnus Sørensen-Clark commented on SPARK-12261: I have a similar problem, so I

  1   2   >