[jira] [Commented] (SPARK-27722) Remove UnsafeKeyValueSorter

2019-05-15 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840971#comment-16840971 ] Xianyin Xin commented on SPARK-27722: - When doing the moving, I didn't find any reference of this

[jira] [Commented] (SPARK-27726) Performance of InMemoryStore suffers under load

2019-05-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840948#comment-16840948 ] Josh Rosen commented on SPARK-27726: Thanks for the detailed bug repot! I appreciate the performance

[jira] [Comment Edited] (SPARK-27718) incorrect result from pagerank

2019-05-15 Thread De-En Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840945#comment-16840945 ] De-En Lin edited comment on SPARK-27718 at 5/16/19 2:55 AM: In wiki, the

[jira] [Commented] (SPARK-27718) incorrect result from pagerank

2019-05-15 Thread De-En Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840945#comment-16840945 ] De-En Lin commented on SPARK-27718: --- In wiki, the equation of PageRank is as follows: !螢幕快照

[jira] [Commented] (SPARK-22128) Update paranamer to 2.8 to avoid BytecodeReadingParanamer ArrayIndexOutOfBoundsException with Scala 2.12 + Java 8 lambda

2019-05-15 Thread Michael Heuer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840940#comment-16840940 ] Michael Heuer commented on SPARK-22128: --- We've run into this again with the binary distribution

[jira] [Updated] (SPARK-27718) incorrect result from pagerank

2019-05-15 Thread De-En Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] De-En Lin updated SPARK-27718: -- Attachment: 螢幕快照 2019-05-16 上午10.09.45.png > incorrect result from pagerank >

[jira] [Commented] (SPARK-27688) Beeline should show database in the prompt

2019-05-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840916#comment-16840916 ] Hyukjin Kwon commented on SPARK-27688: -- ah, right. I rushed to read. thanks :D. > Beeline should

[jira] [Commented] (SPARK-15463) Support for creating a dataframe from CSV in Dataset[String]

2019-05-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840914#comment-16840914 ] Hyukjin Kwon commented on SPARK-15463: -- Please ask a question to the mailing list. > Support for

[jira] [Resolved] (SPARK-27740) JIRA status test

2019-05-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27740. -- Resolution: Invalid > JIRA status test > > > Key:

[jira] [Resolved] (SPARK-27740) JIRA status test

2019-05-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27740. -- Resolution: Unresolved > JIRA status test > > > Key:

[jira] [Reopened] (SPARK-27740) JIRA status test

2019-05-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-27740: -- > JIRA status test > > > Key: SPARK-27740 > URL:

[jira] [Reopened] (SPARK-27740) JIRA status test

2019-05-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-27740: -- > JIRA status test > > > Key: SPARK-27740 > URL:

[jira] [Resolved] (SPARK-27740) JIRA status test

2019-05-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27740. -- Resolution: Auto Closed > JIRA status test > > > Key:

[jira] [Assigned] (SPARK-27739) df.persist should save stats from optimized plan

2019-05-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27739: Assignee: (was: Apache Spark) > df.persist should save stats from optimized plan >

[jira] [Assigned] (SPARK-27739) df.persist should save stats from optimized plan

2019-05-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27739: Assignee: Apache Spark > df.persist should save stats from optimized plan >

[jira] [Created] (SPARK-27740) JIRA status test

2019-05-15 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-27740: Summary: JIRA status test Key: SPARK-27740 URL: https://issues.apache.org/jira/browse/SPARK-27740 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-27739) df.persist should save stats from optimized plan

2019-05-15 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27739: --- Summary: df.persist should save stats from optimized plan (was: Persist should use stats from

[jira] [Updated] (SPARK-27739) Persist should use stats from optimized plan

2019-05-15 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27739: --- Summary: Persist should use stats from optimized plan (was: CacheManager.cacheQuery should copy

[jira] [Updated] (SPARK-27739) Persist should use stats from optimized plan

2019-05-15 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27739: --- Description: CacheManager.cacheQuery passes the stats for `planToCache` to InMemoryRelation. Since

[jira] [Created] (SPARK-27739) CacheManager.cacheQuery should copy stats from optimized plan

2019-05-15 Thread John Zhuge (JIRA)
John Zhuge created SPARK-27739: -- Summary: CacheManager.cacheQuery should copy stats from optimized plan Key: SPARK-27739 URL: https://issues.apache.org/jira/browse/SPARK-27739 Project: Spark

[jira] [Commented] (SPARK-27722) Remove UnsafeKeyValueSorter

2019-05-15 Thread Shivu Sondur (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840881#comment-16840881 ] Shivu Sondur commented on SPARK-27722: -- [~viirya] I also verified it looks this class is not

[jira] [Assigned] (SPARK-27722) Remove UnsafeKeyValueSorter

2019-05-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27722: Assignee: (was: Apache Spark) > Remove UnsafeKeyValueSorter >

[jira] [Assigned] (SPARK-27722) Remove UnsafeKeyValueSorter

2019-05-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27722: Assignee: Apache Spark > Remove UnsafeKeyValueSorter > --- > >

[jira] [Assigned] (SPARK-27738) Upgrade the built-in Hive to 2.3.5 for hadoop-3.2

2019-05-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27738: Assignee: Apache Spark > Upgrade the built-in Hive to 2.3.5 for hadoop-3.2 >

[jira] [Assigned] (SPARK-27738) Upgrade the built-in Hive to 2.3.5 for hadoop-3.2

2019-05-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27738: Assignee: (was: Apache Spark) > Upgrade the built-in Hive to 2.3.5 for hadoop-3.2 >

[jira] [Commented] (SPARK-26192) MesosClusterScheduler reads options from dispatcher conf instead of submission conf

2019-05-15 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840867#comment-16840867 ] Dongjoon Hyun commented on SPARK-26192: --- Sorry for being late, [~mwlon]. Could you make a PR

[jira] [Created] (SPARK-27738) Upgrade the built-in Hive to 2.3.5 for hadoop-3.2

2019-05-15 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-27738: --- Summary: Upgrade the built-in Hive to 2.3.5 for hadoop-3.2 Key: SPARK-27738 URL: https://issues.apache.org/jira/browse/SPARK-27738 Project: Spark Issue Type:

[jira] [Updated] (SPARK-27736) Improve handling of FetchFailures caused by ExternalShuffleService losing track of executor registrations

2019-05-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27736: --- Description: This ticket describes a fault-tolerance edge-case which can cause Spark jobs to fail

[jira] [Updated] (SPARK-27736) Improve handling of FetchFailures caused by ExternalShuffleService losing track of executor registrations

2019-05-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27736: --- Description: I have discovered a fault-tolerance edge-case which can cause Spark jobs to fail if a

[jira] [Updated] (SPARK-27736) Improve handling of FetchFailures caused by ExternalShuffleService losing track of executor registrations

2019-05-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27736: --- Description: I have discovered a fault-tolerance edge-case which can cause Spark jobs to fail if a

[jira] [Assigned] (SPARK-27737) Upgrade to 2.3.5 for Hive Metastore Client 2.3

2019-05-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27737: Assignee: (was: Apache Spark) > Upgrade to 2.3.5 for Hive Metastore Client 2.3 >

[jira] [Assigned] (SPARK-27737) Upgrade to 2.3.5 for Hive Metastore Client 2.3

2019-05-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27737: Assignee: Apache Spark > Upgrade to 2.3.5 for Hive Metastore Client 2.3 >

[jira] [Assigned] (SPARK-27735) Interval string in upper case is not supported in Trigger

2019-05-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27735: Assignee: Apache Spark (was: Shixiong Zhu) > Interval string in upper case is not

[jira] [Assigned] (SPARK-27735) Interval string in upper case is not supported in Trigger

2019-05-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27735: Assignee: Shixiong Zhu (was: Apache Spark) > Interval string in upper case is not

[jira] [Updated] (SPARK-27735) Interval string in upper case is not supported in Trigger

2019-05-15 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-27735: - Description: Some APIs in Structured Streaming requires the user to specify an interval. Right

[jira] [Created] (SPARK-27737) Upgrade to 2.3.5 for Hive Metastore Client 2.3

2019-05-15 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-27737: --- Summary: Upgrade to 2.3.5 for Hive Metastore Client 2.3 Key: SPARK-27737 URL: https://issues.apache.org/jira/browse/SPARK-27737 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-27674) the hint should not be dropped after cache lookup

2019-05-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-27674. - Resolution: Fixed Fix Version/s: 3.0.0 > the hint should not be dropped after cache lookup >

[jira] [Updated] (SPARK-27736) Improve handling of FetchFailures caused by ExternalShuffleService losing track of executor registrations

2019-05-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27736: --- Description: I have discovered a fault-tolerance edge-case which can cause Spark jobs to fail if a

[jira] [Commented] (SPARK-27723) Unable to pull the oracle table data using patitionColumn date/timeStamp

2019-05-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840842#comment-16840842 ] Xiao Li commented on SPARK-27723: - [~Shyama] Could you submit a PR to improve our document?  > Unable

[jira] [Created] (SPARK-27736) Improve handling of FetchFailures caused by ExternalShuffleService losing track of executor registrations

2019-05-15 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-27736: -- Summary: Improve handling of FetchFailures caused by ExternalShuffleService losing track of executor registrations Key: SPARK-27736 URL:

[jira] [Created] (SPARK-27735) Interval string in upper case is not supported in Trigger

2019-05-15 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-27735: Summary: Interval string in upper case is not supported in Trigger Key: SPARK-27735 URL: https://issues.apache.org/jira/browse/SPARK-27735 Project: Spark

[jira] [Assigned] (SPARK-27734) Add memory based thresholds for shuffle spill

2019-05-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27734: Assignee: (was: Apache Spark) > Add memory based thresholds for shuffle spill >

[jira] [Assigned] (SPARK-27734) Add memory based thresholds for shuffle spill

2019-05-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27734: Assignee: Apache Spark > Add memory based thresholds for shuffle spill >

[jira] [Created] (SPARK-27734) Add memory based thresholds for shuffle spill

2019-05-15 Thread Adrian Muraru (JIRA)
Adrian Muraru created SPARK-27734: - Summary: Add memory based thresholds for shuffle spill Key: SPARK-27734 URL: https://issues.apache.org/jira/browse/SPARK-27734 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-27354) Move incompatible code from the hive-thriftserver module to sql/hive-thriftserver/v1.2.1

2019-05-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-27354. - Resolution: Fixed Assignee: Yuming Wang Fix Version/s: 3.0.0 > Move incompatible code

[jira] [Resolved] (SPARK-27036) Even Broadcast thread is timed out, BroadCast Job is not aborted.

2019-05-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-27036. - Resolution: Fixed Assignee: Xingbo Jiang Fix Version/s: 3.0.0 > Even Broadcast thread

[jira] [Resolved] (SPARK-20774) BroadcastExchangeExec doesn't cancel the Spark job if broadcasting a relation timeouts.

2019-05-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-20774. - Resolution: Fixed Assignee: Xingbo Jiang Fix Version/s: 3.0.0 > BroadcastExchangeExec

[jira] [Assigned] (SPARK-27732) DataSourceV2: Add CreateTable logical operation

2019-05-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27732: Assignee: (was: Apache Spark) > DataSourceV2: Add CreateTable logical operation >

[jira] [Assigned] (SPARK-27732) DataSourceV2: Add CreateTable logical operation

2019-05-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27732: Assignee: Apache Spark > DataSourceV2: Add CreateTable logical operation >

[jira] [Created] (SPARK-27733) Upgrade to Avro 1.9.x

2019-05-15 Thread JIRA
Ismaël Mejía created SPARK-27733: Summary: Upgrade to Avro 1.9.x Key: SPARK-27733 URL: https://issues.apache.org/jira/browse/SPARK-27733 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-27732) DataSourceV2: Add CreateTable logical operation

2019-05-15 Thread Ryan Blue (JIRA)
Ryan Blue created SPARK-27732: - Summary: DataSourceV2: Add CreateTable logical operation Key: SPARK-27732 URL: https://issues.apache.org/jira/browse/SPARK-27732 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-27726) Performance of InMemoryStore suffers under load

2019-05-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27726: Assignee: Apache Spark > Performance of InMemoryStore suffers under load >

[jira] [Assigned] (SPARK-27726) Performance of InMemoryStore suffers under load

2019-05-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27726: Assignee: (was: Apache Spark) > Performance of InMemoryStore suffers under load >

[jira] [Updated] (SPARK-27731) Cleanup some non-compile time type checking and exception handling

2019-05-15 Thread David C Navas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David C Navas updated SPARK-27731: -- Description: Previous checkins cleaned up some of the odd exception propagation choices

[jira] [Updated] (SPARK-27731) Cleanup some non-compile time type checking and exception handling

2019-05-15 Thread David C Navas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David C Navas updated SPARK-27731: -- Summary: Cleanup some non-compile time type checking and exception handling (was: Cleanup

[jira] [Created] (SPARK-27731) Cleanup some odd-looking typing choices and exception handling

2019-05-15 Thread David C Navas (JIRA)
David C Navas created SPARK-27731: - Summary: Cleanup some odd-looking typing choices and exception handling Key: SPARK-27731 URL: https://issues.apache.org/jira/browse/SPARK-27731 Project: Spark

[jira] [Updated] (SPARK-27730) Add support for removeAllKeys

2019-05-15 Thread David C Navas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David C Navas updated SPARK-27730: -- Attachment: RemoveAll.pdf > Add support for removeAllKeys > - > >

[jira] [Created] (SPARK-27730) Add support for removeAllKeys

2019-05-15 Thread David C Navas (JIRA)
David C Navas created SPARK-27730: - Summary: Add support for removeAllKeys Key: SPARK-27730 URL: https://issues.apache.org/jira/browse/SPARK-27730 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-27729) Extract deletion of the summaries from the stage deletion loop

2019-05-15 Thread David C Navas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David C Navas updated SPARK-27729: -- Attachment: ExtractDeletes.pdf > Extract deletion of the summaries from the stage deletion

[jira] [Created] (SPARK-27729) Extract deletion of the summaries from the stage deletion loop

2019-05-15 Thread David C Navas (JIRA)
David C Navas created SPARK-27729: - Summary: Extract deletion of the summaries from the stage deletion loop Key: SPARK-27729 URL: https://issues.apache.org/jira/browse/SPARK-27729 Project: Spark

[jira] [Updated] (SPARK-27728) Address thread-safety of InMemoryStore and ElementTrackingStores.

2019-05-15 Thread David C Navas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David C Navas updated SPARK-27728: -- Attachment: RaceConditions.pdf > Address thread-safety of InMemoryStore and

[jira] [Created] (SPARK-27728) Address thread-safety of InMemoryStore and ElementTrackingStores.

2019-05-15 Thread David C Navas (JIRA)
David C Navas created SPARK-27728: - Summary: Address thread-safety of InMemoryStore and ElementTrackingStores. Key: SPARK-27728 URL: https://issues.apache.org/jira/browse/SPARK-27728 Project: Spark

[jira] [Commented] (SPARK-27726) Performance of InMemoryStore suffers under load

2019-05-15 Thread Mark Hamstra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840636#comment-16840636 ] Mark Hamstra commented on SPARK-27726: -- [~vanzin] > Performance of InMemoryStore suffers under

[jira] [Created] (SPARK-27727) Asynchronous ElementStore cleanup should have only one pending cleanup per class

2019-05-15 Thread David C Navas (JIRA)
David C Navas created SPARK-27727: - Summary: Asynchronous ElementStore cleanup should have only one pending cleanup per class Key: SPARK-27727 URL: https://issues.apache.org/jira/browse/SPARK-27727

[jira] [Updated] (SPARK-27727) Asynchronous ElementStore cleanup should have only one pending cleanup per class

2019-05-15 Thread David C Navas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David C Navas updated SPARK-27727: -- Attachment: AsyncDefer.pdf > Asynchronous ElementStore cleanup should have only one pending

[jira] [Updated] (SPARK-27726) Performance of InMemoryStore suffers under load

2019-05-15 Thread David C Navas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David C Navas updated SPARK-27726: -- Attachment: GCRateIssues.pdf > Performance of InMemoryStore suffers under load >

[jira] [Updated] (SPARK-27726) Performance of InMemoryStore suffers under load

2019-05-15 Thread David C Navas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David C Navas updated SPARK-27726: -- Attachment: PerformanceBeforeAndAfter.pdf > Performance of InMemoryStore suffers under load >

[jira] [Created] (SPARK-27726) Performance of InMemoryStore suffers under load

2019-05-15 Thread David C Navas (JIRA)
David C Navas created SPARK-27726: - Summary: Performance of InMemoryStore suffers under load Key: SPARK-27726 URL: https://issues.apache.org/jira/browse/SPARK-27726 Project: Spark Issue

[jira] [Assigned] (SPARK-27488) Driver interface to support GPU resources

2019-05-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27488: Assignee: Thomas Graves (was: Apache Spark) > Driver interface to support GPU resources

[jira] [Assigned] (SPARK-27488) Driver interface to support GPU resources

2019-05-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27488: Assignee: Apache Spark (was: Thomas Graves) > Driver interface to support GPU resources

[jira] [Resolved] (SPARK-27687) Kafka consumer cache parameter rename and documentation

2019-05-15 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27687. --- Resolution: Fixed Assignee: Gabor Somogyi Fix Version/s: 3.0.0 This is

[jira] [Created] (SPARK-27725) GPU Scheduling - add an example discovery Script

2019-05-15 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-27725: - Summary: GPU Scheduling - add an example discovery Script Key: SPARK-27725 URL: https://issues.apache.org/jira/browse/SPARK-27725 Project: Spark Issue

[jira] [Created] (SPARK-27724) Add RTAS logical operation

2019-05-15 Thread Ryan Blue (JIRA)
Ryan Blue created SPARK-27724: - Summary: Add RTAS logical operation Key: SPARK-27724 URL: https://issues.apache.org/jira/browse/SPARK-27724 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-27724) DataSourceV2: Add RTAS logical operation

2019-05-15 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue updated SPARK-27724: -- Summary: DataSourceV2: Add RTAS logical operation (was: Add RTAS logical operation) > DataSourceV2:

[jira] [Updated] (SPARK-24923) DataSourceV2: Add CTAS logical operation

2019-05-15 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue updated SPARK-24923: -- Summary: DataSourceV2: Add CTAS logical operation (was: DataSourceV2: Add CTAS and RTAS logical

[jira] [Assigned] (SPARK-27678) Support Knox user impersonation in UI

2019-05-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-27678: -- Assignee: Marcelo Vanzin > Support Knox user impersonation in UI >

[jira] [Resolved] (SPARK-27678) Support Knox user impersonation in UI

2019-05-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27678. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24582

[jira] [Updated] (SPARK-27720) ConcurrentModificationException on operating with DirectKafkaInputDStream

2019-05-15 Thread ov7a (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ov7a updated SPARK-27720: - Affects Version/s: 2.4.3 Description: If a DirectKafkaInputDStream is started in one thread and is

[jira] [Commented] (SPARK-27720) ConcurrentModificationException on closing DirectKafkaInputDStream

2019-05-15 Thread ov7a (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840558#comment-16840558 ] ov7a commented on SPARK-27720: -- Thank you for reply. I've tried to make a MWE and occasionally hit similar

[jira] [Updated] (SPARK-27631) Avoid repeating calculate table statistics

2019-05-15 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27631: Summary: Avoid repeating calculate table statistics (was: Avoid repeating calculate table

[jira] [Comment Edited] (SPARK-15463) Support for creating a dataframe from CSV in Dataset[String]

2019-05-15 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840018#comment-16840018 ] Ruslan Dautkhanov edited comment on SPARK-15463 at 5/15/19 4:00 PM:

[jira] [Commented] (SPARK-24437) Memory leak in UnsafeHashedRelation

2019-05-15 Thread Eyal Farago (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840498#comment-16840498 ] Eyal Farago commented on SPARK-24437: - [~mgaido], looking at this again I suspect in this case

[jira] [Commented] (SPARK-19728) PythonUDF with multiple parents shouldn't be pushed down when used as a predicate

2019-05-15 Thread Krishna Prasanna Sistla (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840488#comment-16840488 ] Krishna Prasanna Sistla commented on SPARK-19728: - Is it resolved ? I am still seeing

[jira] [Commented] (SPARK-27720) ConcurrentModificationException on closing DirectKafkaInputDStream

2019-05-15 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840487#comment-16840487 ] Gabor Somogyi commented on SPARK-27720: --- Now I see the places where the consumer usage is not

[jira] [Commented] (SPARK-27720) ConcurrentModificationException on closing DirectKafkaInputDStream

2019-05-15 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840479#comment-16840479 ] Gabor Somogyi commented on SPARK-27720: --- Well, had a look on the DirectKafkaInputDStream's stop

[jira] [Updated] (SPARK-27718) incorrect result from pagerank

2019-05-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-27718: -- Target Version/s: (was: 2.4.3) Priority: Minor (was: Major) Fix Version/s:

[jira] [Resolved] (SPARK-27682) Avoid use of Scala collection classes that are removed in 2.13

2019-05-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27682. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24586

[jira] [Commented] (SPARK-27723) Unable to pull the oracle table data using patitionColumn date/timeStamp

2019-05-15 Thread Shyama (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840456#comment-16840456 ] Shyama commented on SPARK-27723: [~yumwang] by the way when I tried with timestamp. I used 

[jira] [Commented] (SPARK-27723) Unable to pull the oracle table data using patitionColumn date/timeStamp

2019-05-15 Thread Shyama (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840442#comment-16840442 ] Shyama commented on SPARK-27723: [~yumwang] wow ..Below is working fine...thank you so much

[jira] [Comment Edited] (SPARK-27720) ConcurrentModificationException on closing DirectKafkaInputDStream

2019-05-15 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840438#comment-16840438 ] Gabor Somogyi edited comment on SPARK-27720 at 5/15/19 2:12 PM: [~ov7a]

[jira] [Commented] (SPARK-27720) ConcurrentModificationException on closing DirectKafkaInputDStream

2019-05-15 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840438#comment-16840438 ] Gabor Somogyi commented on SPARK-27720: --- [~ov7a] Kafka normally does the following in case of

[jira] [Commented] (SPARK-27723) Unable to pull the oracle table data using patitionColumn date/timeStamp

2019-05-15 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840375#comment-16840375 ] Yuming Wang commented on SPARK-27723: - Try to add {{sessionInitStatement}} to option: {noformat}

[jira] [Commented] (SPARK-27714) Support Join Reorder based on Genetic Algorithm when the # of joined tables > 12

2019-05-15 Thread Xianyin Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840370#comment-16840370 ] Xianyin Xin commented on SPARK-27714: - [~hyukjin.kwon] Thanks for reminding. [~hyukjin.kwon]

[jira] [Commented] (SPARK-27689) Error to execute hive views with spark

2019-05-15 Thread Juan Antonio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840353#comment-16840353 ] Juan Antonio commented on SPARK-27689: -- I don't know that I can add to because I wrote how to

[jira] [Commented] (SPARK-27723) Unable to pull the oracle table data using patitionColumn date/timeStamp

2019-05-15 Thread Shyama (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840345#comment-16840345 ] Shyama commented on SPARK-27723: [~yumwang], thanks for reply, you mean to try

[jira] [Updated] (SPARK-27689) Error to execute hive views with spark

2019-05-15 Thread Juan Antonio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juan Antonio updated SPARK-27689: - Priority: Minor (was: Major) > Error to execute hive views with spark >

[jira] [Updated] (SPARK-27723) Unable to pull the oracle table data using patitionColumn date/timeStamp

2019-05-15 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27723: Fix Version/s: (was: 2.4.0) > Unable to pull the oracle table data using patitionColumn

[jira] [Updated] (SPARK-27723) Unable to pull the oracle table data using patitionColumn date/timeStamp

2019-05-15 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27723: Target Version/s: (was: 2.4.1) > Unable to pull the oracle table data using patitionColumn

[jira] [Commented] (SPARK-27723) Unable to pull the oracle table data using patitionColumn date/timeStamp

2019-05-15 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840315#comment-16840315 ] Yuming Wang commented on SPARK-27723: - Could you try this: 

[jira] [Updated] (SPARK-27723) Unable to pull the oracle table data using patitionColumn date/timeStamp

2019-05-15 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27723: Labels: (was: pull-request-available) > Unable to pull the oracle table data using

  1   2   >