[jira] [Commented] (SPARK-21902) BlockManager.doPut will hide actually exception when exception thrown in finally block

2017-09-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16159623#comment-16159623 ] Apache Spark commented on SPARK-21902: -- User 'caneGuy' has created a pull request for this issue:

[jira] [Created] (SPARK-21962) Distributed Tracing in Spark

2017-09-08 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-21962: -- Summary: Distributed Tracing in Spark Key: SPARK-21962 URL: https://issues.apache.org/jira/browse/SPARK-21962 Project: Spark Issue Type: New Feature

[jira] [Assigned] (SPARK-21961) Filter out BlockStatuses Accumulators during replaying history logs in Spark History Server

2017-09-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21961: Assignee: Apache Spark > Filter out BlockStatuses Accumulators during replaying history

[jira] [Assigned] (SPARK-21961) Filter out BlockStatuses Accumulators during replaying history logs in Spark History Server

2017-09-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21961: Assignee: (was: Apache Spark) > Filter out BlockStatuses Accumulators during

[jira] [Commented] (SPARK-21961) Filter out BlockStatuses Accumulators during replaying history logs in Spark History Server

2017-09-08 Thread Ye Zhou (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16159514#comment-16159514 ] Ye Zhou commented on SPARK-21961: - Pull Request Added: https://github.com/apache/spark/pull/19170 >

[jira] [Updated] (SPARK-21961) Filter out BlockStatuses Accumulators during replaying history logs in Spark History Server

2017-09-08 Thread Ye Zhou (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ye Zhou updated SPARK-21961: Description: As described in SPARK-20923, TaskMetrics._updatedBlockStatuses uses a lot of memory in

[jira] [Updated] (SPARK-21961) Filter out BlockStatuses Accumulators during replaying history logs in Spark History Server

2017-09-08 Thread Ye Zhou (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ye Zhou updated SPARK-21961: Description: As described in SPARK-20923, TaskMetrics._updatedBlockStatuses uses a lot of memory in

[jira] [Updated] (SPARK-21961) Filter out BlockStatuses Accumulators during replaying history logs in Spark History Server

2017-09-08 Thread Ye Zhou (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ye Zhou updated SPARK-21961: Description: As described in SPARK-20923, TaskMetrics._updatedBlockStatuses uses a lot of memory in

[jira] [Updated] (SPARK-21961) Filter out BlockStatuses Accumulators during replaying history logs in Spark History Server

2017-09-08 Thread Ye Zhou (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ye Zhou updated SPARK-21961: Description: As described in SPARK-20923, TaskMetrics._updatedBlockStatuses uses a lot of memory in

[jira] [Updated] (SPARK-21961) Filter out BlockStatuses Accumulators during replaying history logs in Spark History Server

2017-09-08 Thread Ye Zhou (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ye Zhou updated SPARK-21961: Description: As described in SPARK-20923, TaskMetrics._updatedBlockStatuses uses a lot of memory in

[jira] [Updated] (SPARK-21961) Filter out BlockStatuses Accumulators during replaying history logs in Spark History Server

2017-09-08 Thread Ye Zhou (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ye Zhou updated SPARK-21961: Attachment: Objects_Count_in_Heap.png One_Thread_Took_24GB.png > Filter out BlockStatuses

[jira] [Created] (SPARK-21961) Filter out BlockStatuses Accumulators during replaying history logs in Spark History Server

2017-09-08 Thread Ye Zhou (JIRA)
Ye Zhou created SPARK-21961: --- Summary: Filter out BlockStatuses Accumulators during replaying history logs in Spark History Server Key: SPARK-21961 URL: https://issues.apache.org/jira/browse/SPARK-21961

[jira] [Created] (SPARK-21960) Spark Streaming Dynamic Allocation should respect spark.executor.instances

2017-09-08 Thread Karthik Palaniappan (JIRA)
Karthik Palaniappan created SPARK-21960: --- Summary: Spark Streaming Dynamic Allocation should respect spark.executor.instances Key: SPARK-21960 URL: https://issues.apache.org/jira/browse/SPARK-21960

[jira] [Resolved] (SPARK-19866) Add local version of Word2Vec findSynonyms for spark.ml: Python API

2017-09-08 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-19866. - Resolution: Fixed Fix Version/s: 2.3.0 > Add local version of Word2Vec findSynonyms for spark.ml:

[jira] [Resolved] (SPARK-15243) Binarizer.explainParam(u"...") raises ValueError

2017-09-08 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-15243. - Resolution: Fixed > Binarizer.explainParam(u"...") raises ValueError >

[jira] [Updated] (SPARK-15243) Binarizer.explainParam(u"...") raises ValueError

2017-09-08 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk updated SPARK-15243: Fix Version/s: 2.3.0 > Binarizer.explainParam(u"...") raises ValueError >

[jira] [Assigned] (SPARK-15243) Binarizer.explainParam(u"...") raises ValueError

2017-09-08 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk reassigned SPARK-15243: --- Assignee: Hyukjin Kwon (was: Seth Hendrickson) > Binarizer.explainParam(u"...") raises ValueError

[jira] [Commented] (SPARK-18085) SPIP: Better History Server scalability for many / large applications

2017-09-08 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158997#comment-16158997 ] Marcelo Vanzin commented on SPARK-18085: [~jincheng] that is caused by SPARK-17701. The bug is

[jira] [Commented] (SPARK-21907) NullPointerException in UnsafeExternalSorter.spill()

2017-09-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158995#comment-16158995 ] Kazuaki Ishizaki commented on SPARK-21907: -- If you cannot provide a repro, could you please run

[jira] [Resolved] (SPARK-21959) Python RDD goes into never ending garbage collection service when spark submit is triggered in oozie

2017-09-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21959. --- Resolution: Invalid There's no detail on the job, and no indication that this is a problem in

[jira] [Closed] (SPARK-21959) Python RDD goes into never ending garbage collection service when spark submit is triggered in oozie

2017-09-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen closed SPARK-21959. - > Python RDD goes into never ending garbage collection service when spark > submit is triggered in oozie >

[jira] [Created] (SPARK-21959) Python RDD goes into never ending garbage collection service when spark submit is triggered in oozie

2017-09-08 Thread Vega Paleri (JIRA)
Vega Paleri created SPARK-21959: --- Summary: Python RDD goes into never ending garbage collection service when spark submit is triggered in oozie Key: SPARK-21959 URL:

[jira] [Updated] (SPARK-21893) Put Kafka 0.8 behind a profile

2017-09-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21893: -- Description: Kafka does not support 0.8.x for Scala 2.12. This code will have to, at least, be

[jira] [Commented] (SPARK-21944) Watermark on window column is wrong

2017-09-08 Thread Kevin Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158948#comment-16158948 ] Kevin Zhang commented on SPARK-21944: - [~mgaido] Do you mean the following way by saying "define the

[jira] [Commented] (SPARK-20958) Roll back parquet-mr 1.8.2 to parquet-1.8.1

2017-09-08 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158924#comment-16158924 ] Ryan Blue commented on SPARK-20958: --- [~spiricalsalsaz], you need to only pin parquet-avro, not the

[jira] [Updated] (SPARK-21128) Running R tests multiple times failed due to pre-exiting "spark-warehouse" / "metastore_db"

2017-09-08 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-21128: - Target Version/s: 2.2.1, 2.3.0 (was: 2.3.0) Fix Version/s: 2.2.1 > Running R tests

[jira] [Resolved] (SPARK-21946) Flaky test: InMemoryCatalogedDDLSuite.`alter table: rename cached table`

2017-09-08 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-21946. - Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 > Flaky test:

[jira] [Assigned] (SPARK-21946) Flaky test: InMemoryCatalogedDDLSuite.`alter table: rename cached table`

2017-09-08 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-21946: --- Assignee: Kazuaki Ishizaki > Flaky test: InMemoryCatalogedDDLSuite.`alter table: rename cached

[jira] [Updated] (SPARK-21936) backward compatibility test framework for HiveExternalCatalog

2017-09-08 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21936: Fix Version/s: 2.2.1 > backward compatibility test framework for HiveExternalCatalog >

[jira] [Commented] (SPARK-18085) SPIP: Better History Server scalability for many / large applications

2017-09-08 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158865#comment-16158865 ] Marcelo Vanzin commented on SPARK-18085: Do you really mean fixed, as in you're not seeing it

[jira] [Commented] (SPARK-21190) SPIP: Vectorized UDFs in Python

2017-09-08 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158845#comment-16158845 ] Li Jin commented on SPARK-21190: To [~bryanc]'s point, PR

[jira] [Created] (SPARK-21958) Attempting to save large Word2Vec model hangs driver in constant GC.

2017-09-08 Thread Travis Hegner (JIRA)
Travis Hegner created SPARK-21958: - Summary: Attempting to save large Word2Vec model hangs driver in constant GC. Key: SPARK-21958 URL: https://issues.apache.org/jira/browse/SPARK-21958 Project:

[jira] [Commented] (SPARK-21067) Thrift Server - CTAS fail with Unable to move source

2017-09-08 Thread Dominic Ricard (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158802#comment-16158802 ] Dominic Ricard commented on SPARK-21067: [~zhangxin0112zx] Our solution was to migrate the CTAS

[jira] [Assigned] (SPARK-21957) Add current_user function

2017-09-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21957: Assignee: Apache Spark > Add current_user function > - > >

[jira] [Commented] (SPARK-21957) Add current_user function

2017-09-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158745#comment-16158745 ] Apache Spark commented on SPARK-21957: -- User 'mgaido91' has created a pull request for this issue:

[jira] [Assigned] (SPARK-21957) Add current_user function

2017-09-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21957: Assignee: (was: Apache Spark) > Add current_user function > -

[jira] [Created] (SPARK-21957) Add current_user function

2017-09-08 Thread Marco Gaido (JIRA)
Marco Gaido created SPARK-21957: --- Summary: Add current_user function Key: SPARK-21957 URL: https://issues.apache.org/jira/browse/SPARK-21957 Project: Spark Issue Type: New Feature

[jira] [Comment Edited] (SPARK-20958) Roll back parquet-mr 1.8.2 to parquet-1.8.1

2017-09-08 Thread Anthony Dotterer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155387#comment-16155387 ] Anthony Dotterer edited comment on SPARK-20958 at 9/8/17 2:27 PM: -- As a

[jira] [Assigned] (SPARK-21956) Fetch up to max bytes when buf really released

2017-09-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21956: Assignee: Apache Spark > Fetch up to max bytes when buf really released >

[jira] [Assigned] (SPARK-21956) Fetch up to max bytes when buf really released

2017-09-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21956: Assignee: (was: Apache Spark) > Fetch up to max bytes when buf really released >

[jira] [Commented] (SPARK-21956) Fetch up to max bytes when buf really released

2017-09-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158548#comment-16158548 ] Apache Spark commented on SPARK-21956: -- User 'caneGuy' has created a pull request for this issue:

[jira] [Created] (SPARK-21956) Fetch up to max bytes when buf really released

2017-09-08 Thread zhoukang (JIRA)
zhoukang created SPARK-21956: Summary: Fetch up to max bytes when buf really released Key: SPARK-21956 URL: https://issues.apache.org/jira/browse/SPARK-21956 Project: Spark Issue Type:

[jira] [Updated] (SPARK-21942) DiskBlockManager crashing when a root local folder has been externally deleted by OS

2017-09-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21942: -- Affects Version/s: (was: 2.2.1) (was: 2.3.0)

[jira] [Commented] (SPARK-21905) ClassCastException when call sqlContext.sql on temp table

2017-09-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158496#comment-16158496 ] Kazuaki Ishizaki commented on SPARK-21905: -- While I ran the following code (I do not have

[jira] [Reopened] (SPARK-21951) Unable to add the new column and writing into the Hive using spark

2017-09-08 Thread jalendhar Baddam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jalendhar Baddam reopened SPARK-21951: -- This is still exist. And it's getting the AnalysisException. > Unable to add the new

[jira] [Reopened] (SPARK-21952) Unable to load the csv file into Dataset using Spark with java

2017-09-08 Thread jalendhar Baddam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jalendhar Baddam reopened SPARK-21952: -- This is still exist ,Please re check > Unable to load the csv file into Dataset using

[jira] [Commented] (SPARK-4502) Spark SQL reads unneccesary nested fields from Parquet

2017-09-08 Thread Andriy Kushnir (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158444#comment-16158444 ] Andriy Kushnir commented on SPARK-4502: --- Just tried this patch on Spark 2.2.0 There are *really

[jira] [Comment Edited] (SPARK-21944) Watermark on window column is wrong

2017-09-08 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158406#comment-16158406 ] Marco Gaido edited comment on SPARK-21944 at 9/8/17 10:31 AM: -- [~KevinZwx]

[jira] [Comment Edited] (SPARK-19090) Dynamic Resource Allocation not respecting spark.executor.cores

2017-09-08 Thread Carlos Vicenti (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16157932#comment-16157932 ] Carlos Vicenti edited comment on SPARK-19090 at 9/8/17 10:19 AM: - I have

[jira] [Comment Edited] (SPARK-21944) Watermark on window column is wrong

2017-09-08 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158406#comment-16158406 ] Marco Gaido edited comment on SPARK-21944 at 9/8/17 9:57 AM: - [~KevinZwx] you

[jira] [Commented] (SPARK-21944) Watermark on window column is wrong

2017-09-08 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158406#comment-16158406 ] Marco Gaido commented on SPARK-21944: - [~kevinzhang] you should define the watermark on the column

[jira] [Commented] (SPARK-21955) OneForOneStreamManager may leak memory when network is poor

2017-09-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158402#comment-16158402 ] Sean Owen commented on SPARK-21955: --- You might be on to something, but this is poorly described. Can

[jira] [Created] (SPARK-21955) OneForOneStreamManager may leak memory when network is poor

2017-09-08 Thread poseidon (JIRA)
poseidon created SPARK-21955: Summary: OneForOneStreamManager may leak memory when network is poor Key: SPARK-21955 URL: https://issues.apache.org/jira/browse/SPARK-21955 Project: Spark Issue

[jira] [Commented] (SPARK-21942) DiskBlockManager crashing when a root local folder has been externally deleted by OS

2017-09-08 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158320#comment-16158320 ] Saisai Shao commented on SPARK-21942: - Personally I would like to fail fast if such things happened,

[jira] [Commented] (SPARK-21942) DiskBlockManager crashing when a root local folder has been externally deleted by OS

2017-09-08 Thread Ruslan Shestopalyuk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158304#comment-16158304 ] Ruslan Shestopalyuk commented on SPARK-21942: - [~jerryshao] I believe the only objective

[jira] [Commented] (SPARK-21954) JacksonUtils should verify MapType's value type instead of key type

2017-09-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158300#comment-16158300 ] Apache Spark commented on SPARK-21954: -- User 'viirya' has created a pull request for this issue:

[jira] [Assigned] (SPARK-21954) JacksonUtils should verify MapType's value type instead of key type

2017-09-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21954: Assignee: (was: Apache Spark) > JacksonUtils should verify MapType's value type

[jira] [Assigned] (SPARK-21954) JacksonUtils should verify MapType's value type instead of key type

2017-09-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21954: Assignee: Apache Spark > JacksonUtils should verify MapType's value type instead of key

[jira] [Created] (SPARK-21954) JacksonUtils should verify MapType's value type instead of key type

2017-09-08 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-21954: --- Summary: JacksonUtils should verify MapType's value type instead of key type Key: SPARK-21954 URL: https://issues.apache.org/jira/browse/SPARK-21954 Project:

[jira] [Commented] (SPARK-21128) Running R tests multiple times failed due to pre-exiting "spark-warehouse" / "metastore_db"

2017-09-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158276#comment-16158276 ] Apache Spark commented on SPARK-21128: -- User 'felixcheung' has created a pull request for this

[jira] [Commented] (SPARK-21942) DiskBlockManager crashing when a root local folder has been externally deleted by OS

2017-09-08 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158271#comment-16158271 ] Saisai Shao commented on SPARK-21942: - {quote}

[jira] [Commented] (SPARK-21953) Show both memory and disk bytes spilled if either is present

2017-09-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158263#comment-16158263 ] Apache Spark commented on SPARK-21953: -- User 'ash211' has created a pull request for this issue:

[jira] [Assigned] (SPARK-21953) Show both memory and disk bytes spilled if either is present

2017-09-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21953: Assignee: Apache Spark > Show both memory and disk bytes spilled if either is present >

[jira] [Assigned] (SPARK-21953) Show both memory and disk bytes spilled if either is present

2017-09-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21953: Assignee: (was: Apache Spark) > Show both memory and disk bytes spilled if either is

[jira] [Created] (SPARK-21953) Show both memory and disk bytes spilled if either is present

2017-09-08 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-21953: -- Summary: Show both memory and disk bytes spilled if either is present Key: SPARK-21953 URL: https://issues.apache.org/jira/browse/SPARK-21953 Project: Spark

[jira] [Commented] (SPARK-18085) SPIP: Better History Server scalability for many / large applications

2017-09-08 Thread jincheng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158247#comment-16158247 ] jincheng commented on SPARK-18085: -- {code:java}

[jira] [Commented] (SPARK-21936) backward compatibility test framework for HiveExternalCatalog

2017-09-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158241#comment-16158241 ] Apache Spark commented on SPARK-21936: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Commented] (SPARK-21726) Check for structural integrity of the plan in QO in test mode

2017-09-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158224#comment-16158224 ] Apache Spark commented on SPARK-21726: -- User 'viirya' has created a pull request for this issue:

[jira] [Resolved] (SPARK-21931) add LNNVL function

2017-09-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21931. --- Resolution: Won't Fix > add LNNVL function > -- > > Key: SPARK-21931

[jira] [Resolved] (SPARK-21915) Model 1 and Model 2 ParamMaps Missing

2017-09-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21915. --- Resolution: Fixed Fix Version/s: 2.2.1 Issue resolved by pull request 19152

[jira] [Resolved] (SPARK-21951) Unable to add the new column and writing into the Hive using spark

2017-09-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21951. --- Resolution: Invalid This doesn't express a problem, and questions should go to the mailing list >

[jira] [Resolved] (SPARK-21952) Unable to load the csv file into Dataset using Spark with java

2017-09-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21952. --- Resolution: Invalid Spam > Unable to load the csv file into Dataset using Spark with java >

[jira] [Created] (SPARK-21952) Unable to load the csv file into Dataset using Spark with java

2017-09-08 Thread jalendhar Baddam (JIRA)
jalendhar Baddam created SPARK-21952: Summary: Unable to load the csv file into Dataset using Spark with java Key: SPARK-21952 URL: https://issues.apache.org/jira/browse/SPARK-21952 Project:

[jira] [Created] (SPARK-21951) Unable to add the new column and writing into the Hive using spark

2017-09-08 Thread jalendhar Baddam (JIRA)
jalendhar Baddam created SPARK-21951: Summary: Unable to add the new column and writing into the Hive using spark Key: SPARK-21951 URL: https://issues.apache.org/jira/browse/SPARK-21951 Project:

[jira] [Commented] (SPARK-650) Add a "setup hook" API for running initialization code on each executor

2017-09-08 Thread yiming.xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158202#comment-16158202 ] yiming.xu commented on SPARK-650: - I need a hook too. Some case, We need init something like spring

[jira] [Commented] (SPARK-21067) Thrift Server - CTAS fail with Unable to move source

2017-09-08 Thread xinzhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158192#comment-16158192 ] xinzhang commented on SPARK-21067: -- hi [~dricard] do u have any solutions now? any suggests will

[jira] [Commented] (SPARK-18492) GeneratedIterator grows beyond 64 KB

2017-09-08 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158189#comment-16158189 ] cen yuhai commented on SPARK-18492: --- spark 2.1.1 also have this problem > GeneratedIterator grows

[jira] [Resolved] (SPARK-21936) backward compatibility test framework for HiveExternalCatalog

2017-09-08 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-21936. - Resolution: Fixed Fix Version/s: 2.3.0 > backward compatibility test framework for

[jira] [Assigned] (SPARK-21934) Expose Netty memory usage via Metrics System

2017-09-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21934: Assignee: (was: Apache Spark) > Expose Netty memory usage via Metrics System >

[jira] [Commented] (SPARK-21934) Expose Netty memory usage via Metrics System

2017-09-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158184#comment-16158184 ] Apache Spark commented on SPARK-21934: -- User 'jerryshao' has created a pull request for this issue:

[jira] [Assigned] (SPARK-21934) Expose Netty memory usage via Metrics System

2017-09-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21934: Assignee: Apache Spark > Expose Netty memory usage via Metrics System >

[jira] [Assigned] (SPARK-21946) Flaky test: InMemoryCatalogedDDLSuite.`alter table: rename cached table`

2017-09-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21946: Assignee: (was: Apache Spark) > Flaky test: InMemoryCatalogedDDLSuite.`alter table:

[jira] [Assigned] (SPARK-21946) Flaky test: InMemoryCatalogedDDLSuite.`alter table: rename cached table`

2017-09-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21946: Assignee: Apache Spark > Flaky test: InMemoryCatalogedDDLSuite.`alter table: rename

[jira] [Commented] (SPARK-21946) Flaky test: InMemoryCatalogedDDLSuite.`alter table: rename cached table`

2017-09-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158176#comment-16158176 ] Apache Spark commented on SPARK-21946: -- User 'kiszk' has created a pull request for this issue:

[jira] [Resolved] (SPARK-21726) Check for structural integrity of the plan in QO in test mode

2017-09-08 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-21726. - Resolution: Fixed Assignee: Liang-Chi Hsieh Fix Version/s: 2.3.0 > Check for structural

[jira] [Resolved] (SPARK-21949) Tables created in unit tests should be dropped after use

2017-09-08 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-21949. - Resolution: Fixed Assignee: liuxian Fix Version/s: 2.3.0 > Tables created in unit tests