[jira] [Created] (SPARK-28600) the method agg of KeyValueGroupedDataset has 4 TypedColumn limits

Zhou Zach created SPARK-28600: - Summary: the method agg of KeyValueGroupedDataset has 4 TypedColumn limits Key: SPARK-28600 URL: https://issues.apache.org/jira/browse/SPARK-28600 Project: Spark

[jira] [Updated] (SPARK-28599) Support pagination and sorting for ThriftServerPage

[ https://issues.apache.org/jira/browse/SPARK-28599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28599: Description: SQLTab support pagination and sorting, but ThriftServerPage missing this feature.

[jira] [Updated] (SPARK-28599) Support pagination and sorting for ThriftServerPage

[ https://issues.apache.org/jira/browse/SPARK-28599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28599: Description: SQLTab  > Support pagination and sorting for ThriftServerPage >

[jira] [Commented] (SPARK-28599) Support pagination and sorting for ThriftServerPage

[ https://issues.apache.org/jira/browse/SPARK-28599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16898509#comment-16898509 ] Yuming Wang commented on SPARK-28599: - cc [~shahid] Would you like to pick up this? > Support

[jira] [Created] (SPARK-28599) Support pagination and sorting for ThriftServerPage

Yuming Wang created SPARK-28599: --- Summary: Support pagination and sorting for ThriftServerPage Key: SPARK-28599 URL: https://issues.apache.org/jira/browse/SPARK-28599 Project: Spark Issue

[jira] [Resolved] (SPARK-28591) Message is incorrect when trying to read shuffle blocks

[ https://issues.apache.org/jira/browse/SPARK-28591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28591. -- Resolution: Won't Fix > Message is incorrect when trying to read shuffle blocks >

[jira] [Created] (SPARK-28598) Few date time manipulation functions does not provide versions supporting Column as input through the Dataframe API

Weichen Xu created SPARK-28598: -- Summary: Few date time manipulation functions does not provide versions supporting Column as input through the Dataframe API Key: SPARK-28598 URL:

[jira] [Updated] (SPARK-28597) Spark streaming terminated when close meta data log error

[ https://issues.apache.org/jira/browse/SPARK-28597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ulysses you updated SPARK-28597: Description: Now, HDFSMetaDataLog is the only implement of MetaDataLog. If

[jira] [Updated] (SPARK-28597) Spark streaming terminated when close meta data log error

[ https://issues.apache.org/jira/browse/SPARK-28597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ulysses you updated SPARK-28597: Description: Now, HDFSMetaDataLog is the only implement of MetaDataLog. If

[jira] [Updated] (SPARK-28597) Spark streaming terminated when close meta data log error

[ https://issues.apache.org/jira/browse/SPARK-28597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ulysses you updated SPARK-28597: Issue Type: Improvement (was: Bug) > Spark streaming terminated when close meta data log error

[jira] [Created] (SPARK-28597) Spark streaming terminated when close meta data log error

ulysses you created SPARK-28597: --- Summary: Spark streaming terminated when close meta data log error Key: SPARK-28597 URL: https://issues.apache.org/jira/browse/SPARK-28597 Project: Spark

[jira] [Updated] (SPARK-28521) Fix error message for built-in functions

[ https://issues.apache.org/jira/browse/SPARK-28521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-28521: -- Priority: Minor (was: Major) Issue Type: Improvement (was: Bug) > Fix error message for

[jira] [Assigned] (SPARK-28521) Fix error message for built-in functions

[ https://issues.apache.org/jira/browse/SPARK-28521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-28521: - Assignee: Yuming Wang > Fix error message for built-in functions >

[jira] [Resolved] (SPARK-28521) Fix error message for built-in functions

[ https://issues.apache.org/jira/browse/SPARK-28521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-28521. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25261

[jira] [Updated] (SPARK-28584) Flaky test: org.apache.spark.scheduler.TaskSchedulerImplSuite

[ https://issues.apache.org/jira/browse/SPARK-28584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-28584: --- Summary: Flaky test: org.apache.spark.scheduler.TaskSchedulerImplSuite (was: Fix thread

[jira] [Comment Edited] (SPARK-28584) Flaky test: org.apache.spark.scheduler.TaskSchedulerImplSuite

[ https://issues.apache.org/jira/browse/SPARK-28584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16898225#comment-16898225 ] Marcelo Vanzin edited comment on SPARK-28584 at 8/1/19 5:42 PM: Please

[jira] [Resolved] (SPARK-28584) Flaky test: org.apache.spark.scheduler.TaskSchedulerImplSuite

[ https://issues.apache.org/jira/browse/SPARK-28584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28584. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25317

[jira] [Updated] (SPARK-28584) Fix thread safety issue in blacklist timer, tests

[ https://issues.apache.org/jira/browse/SPARK-28584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28584: -- Summary: Fix thread safety issue in blacklist timer, tests (was: Flaky test:

[jira] [Assigned] (SPARK-28584) Flaky test: org.apache.spark.scheduler.TaskSchedulerImplSuite

[ https://issues.apache.org/jira/browse/SPARK-28584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-28584: - Assignee: Marcelo Vanzin > Flaky test:

[jira] [Resolved] (SPARK-28586) Make merge-spark-pr script compatible with Python 3

[ https://issues.apache.org/jira/browse/SPARK-28586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28586. --- Resolution: Fixed Assignee: Hyukjin Kwon Fix Version/s: 3.0.0 This is

[jira] [Assigned] (SPARK-28568) Make Javadoc in org.apache.spark.shuffle.api visible

[ https://issues.apache.org/jira/browse/SPARK-28568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-28568: - Assignee: Hyukjin Kwon > Make Javadoc in org.apache.spark.shuffle.api visible >

[jira] [Resolved] (SPARK-28568) Make Javadoc in org.apache.spark.shuffle.api visible

[ https://issues.apache.org/jira/browse/SPARK-28568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28568. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25323

[jira] [Created] (SPARK-28596) Use Java 8 time API in date_trunc

Maxim Gekk created SPARK-28596: -- Summary: Use Java 8 time API in date_trunc Key: SPARK-28596 URL: https://issues.apache.org/jira/browse/SPARK-28596 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-28595) explain should not trigger partition listing

Wenchen Fan created SPARK-28595: --- Summary: explain should not trigger partition listing Key: SPARK-28595 URL: https://issues.apache.org/jira/browse/SPARK-28595 Project: Spark Issue Type:

[jira] [Created] (SPARK-28594) Allow event logs for running streaming apps to be rolled over.

Stephen Levett created SPARK-28594: -- Summary: Allow event logs for running streaming apps to be rolled over. Key: SPARK-28594 URL: https://issues.apache.org/jira/browse/SPARK-28594 Project: Spark

[jira] [Created] (SPARK-28593) Rename ShuffleClient to BlockStoreClient which more close to its usage

Yuanjian Li created SPARK-28593: --- Summary: Rename ShuffleClient to BlockStoreClient which more close to its usage Key: SPARK-28593 URL: https://issues.apache.org/jira/browse/SPARK-28593 Project: Spark

[jira] [Updated] (SPARK-28592) Mark new Shuffle apis as @Experimental (instead of @Private)

[ https://issues.apache.org/jira/browse/SPARK-28592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-28592: - Description: The new Shuffle api is initially marked as {{@Private}}, just to discourage anyone

[jira] [Updated] (SPARK-28592) Mark new Shuffle apis as @Experimental (instead of @Private)

[ https://issues.apache.org/jira/browse/SPARK-28592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-28592: - Summary: Mark new Shuffle apis as @Experimental (instead of @Private) (was: Mark apis as

[jira] [Assigned] (SPARK-25584) Document libsvm data source in doc site

[ https://issues.apache.org/jira/browse/SPARK-25584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-25584: - Assignee: zhengruifeng > Document libsvm data source in doc site >

[jira] [Resolved] (SPARK-25584) Document libsvm data source in doc site

[ https://issues.apache.org/jira/browse/SPARK-25584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25584. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25286

[jira] [Updated] (SPARK-25584) Document libsvm data source in doc site

[ https://issues.apache.org/jira/browse/SPARK-25584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-25584: -- Priority: Minor (was: Major) > Document libsvm data source in doc site >

[jira] [Resolved] (SPARK-26329) ExecutorMetrics should poll faster than heartbeats

[ https://issues.apache.org/jira/browse/SPARK-26329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-26329. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23767

[jira] [Assigned] (SPARK-26329) ExecutorMetrics should poll faster than heartbeats

[ https://issues.apache.org/jira/browse/SPARK-26329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-26329: Assignee: Wing Yew Poon > ExecutorMetrics should poll faster than heartbeats >

[jira] [Created] (SPARK-28592) Mark apis as @Experimental (instead of @Private)

Imran Rashid created SPARK-28592: Summary: Mark apis as @Experimental (instead of @Private) Key: SPARK-28592 URL: https://issues.apache.org/jira/browse/SPARK-28592 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-28366) Logging in driver when loading single large unsplittable file

[ https://issues.apache.org/jira/browse/SPARK-28366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-28366. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25134

[jira] [Assigned] (SPARK-28366) Logging in driver when loading single large unsplittable file

[ https://issues.apache.org/jira/browse/SPARK-28366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-28366: --- Assignee: Weichen Xu > Logging in driver when loading single large unsplittable file >

[jira] [Updated] (SPARK-28590) Add sort_stats Setter for Custom Profiler

[ https://issues.apache.org/jira/browse/SPARK-28590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Albertus Kelvin updated SPARK-28590: Description: When I want to use BasicProfiler with different sorters in sort_stats, I

[jira] [Commented] (SPARK-27689) Error to execute hive views with spark

[ https://issues.apache.org/jira/browse/SPARK-27689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897948#comment-16897948 ] Manu Zhang commented on SPARK-27689: [~lambda], [~hyukjin.kwon], [~yumwang], This issue should be

[jira] [Created] (SPARK-28591) Message is incorrect when trying to read shuffle blocks

Sandeep Katta created SPARK-28591: - Summary: Message is incorrect when trying to read shuffle blocks Key: SPARK-28591 URL: https://issues.apache.org/jira/browse/SPARK-28591 Project: Spark

[jira] [Created] (SPARK-28590) Add sort_stats Setter for Custom Profiler

Albertus Kelvin created SPARK-28590: --- Summary: Add sort_stats Setter for Custom Profiler Key: SPARK-28590 URL: https://issues.apache.org/jira/browse/SPARK-28590 Project: Spark Issue Type:

[jira] [Updated] (SPARK-28512) New optional mode: throw runtime exceptions on casting failures

[ https://issues.apache.org/jira/browse/SPARK-28512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-28512: --- Issue Type: Sub-task (was: Improvement) Parent: SPARK-28589 > New optional mode:

[jira] [Updated] (SPARK-28503) Return null result on cast an out-of-range value to a integral type

[ https://issues.apache.org/jira/browse/SPARK-28503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-28503: --- Issue Type: Sub-task (was: Bug) Parent: SPARK-28589 > Return null result on cast

[jira] [Updated] (SPARK-28495) Follow ANSI SQL on table insertion

[ https://issues.apache.org/jira/browse/SPARK-28495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-28495: --- Issue Type: Sub-task (was: Improvement) Parent: SPARK-28589 > Follow ANSI SQL on

[jira] [Updated] (SPARK-28495) Follow ANSI SQL on table insertion

[ https://issues.apache.org/jira/browse/SPARK-28495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-28495: --- Summary: Follow ANSI SQL on table insertion (was: AssignableCast: A new type coercion

[jira] [Updated] (SPARK-28589) Follow the type conversion rules of PostgreSQL

[ https://issues.apache.org/jira/browse/SPARK-28589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-28589: --- Attachment: (was: type_conversion.txt) > Follow the type conversion rules of PostgreSQL

[jira] [Updated] (SPARK-28589) Follow the type conversion rules of PostgreSQL

[ https://issues.apache.org/jira/browse/SPARK-28589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-28589: --- Attachment: postgreSQL_type_conversion.txt > Follow the type conversion rules of PostgreSQL

[jira] [Created] (SPARK-28589) Follow the type conversion rules of PostgreSQL

Gengliang Wang created SPARK-28589: -- Summary: Follow the type conversion rules of PostgreSQL Key: SPARK-28589 URL: https://issues.apache.org/jira/browse/SPARK-28589 Project: Spark Issue

[jira] [Updated] (SPARK-28589) Follow the type conversion rules of PostgreSQL

[ https://issues.apache.org/jira/browse/SPARK-28589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-28589: --- Attachment: type_conversion.txt > Follow the type conversion rules of PostgreSQL >

[jira] [Updated] (SPARK-28589) Follow the type conversion rules of PostgreSQL

[ https://issues.apache.org/jira/browse/SPARK-28589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-28589: --- Description: The type conversion in current Spark can be improved by learning from

[jira] [Updated] (SPARK-28562) PySpark profiling is not understandable

[ https://issues.apache.org/jira/browse/SPARK-28562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28562: - Priority: Minor (was: Critical) > PySpark profiling is not understandable >

[jira] [Commented] (SPARK-28562) PySpark profiling is not understandable

[ https://issues.apache.org/jira/browse/SPARK-28562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897835#comment-16897835 ] Hyukjin Kwon commented on SPARK-28562: -- Please ask questions to mailing list rather than filing as

[jira] [Commented] (SPARK-28567) pyspark-sql returns error "Paths and fileStatuses are expected to be 1. . Got paths: 1 fileStatuses: 26"

[ https://issues.apache.org/jira/browse/SPARK-28567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897831#comment-16897831 ] Hyukjin Kwon commented on SPARK-28567: -- Please show the stacktrance with a narrowed down and

[jira] [Resolved] (SPARK-28567) pyspark-sql returns error "Paths and fileStatuses are expected to be 1. . Got paths: 1 fileStatuses: 26"

[ https://issues.apache.org/jira/browse/SPARK-28567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28567. -- Resolution: Incomplete > pyspark-sql returns error "Paths and fileStatuses are expected to be

[jira] [Commented] (SPARK-28569) Recommended additions to SECRET_REDACTION_PATTERN

[ https://issues.apache.org/jira/browse/SPARK-28569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897830#comment-16897830 ] Hyukjin Kwon commented on SPARK-28569: -- Can you describe more contexts? For instance the expected

[jira] [Resolved] (SPARK-28575) Spark job time increasing when upgrading Spark from 2.1.1 to 2.3.1

[ https://issues.apache.org/jira/browse/SPARK-28575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28575. -- Resolution: Incomplete Please reopen when those are ready. I am leaving this as incomplete

[jira] [Commented] (SPARK-28575) Spark job time increasing when upgrading Spark from 2.1.1 to 2.3.1

[ https://issues.apache.org/jira/browse/SPARK-28575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897826#comment-16897826 ] Hyukjin Kwon commented on SPARK-28575: -- Please show the reproducer (self-contained if possible) and

[jira] [Resolved] (SPARK-26218) Throw exception on overflow for integers

[ https://issues.apache.org/jira/browse/SPARK-26218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-26218. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21599

[jira] [Assigned] (SPARK-26218) Throw exception on overflow for integers

[ https://issues.apache.org/jira/browse/SPARK-26218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-26218: --- Assignee: Marco Gaido > Throw exception on overflow for integers >

[jira] [Updated] (SPARK-28588) Build a SQL reference doc

[ https://issues.apache.org/jira/browse/SPARK-28588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-28588: Target Version/s: 3.0.0 > Build a SQL reference doc > - > > Key:

[jira] [Commented] (SPARK-28588) Build a SQL reference doc

[ https://issues.apache.org/jira/browse/SPARK-28588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897802#comment-16897802 ] Xiao Li commented on SPARK-28588: - cc [~huaxingao] [~dkbiswal] > Build a SQL reference doc >

[jira] [Created] (SPARK-28588) Build a SQL reference doc

Xiao Li created SPARK-28588: --- Summary: Build a SQL reference doc Key: SPARK-28588 URL: https://issues.apache.org/jira/browse/SPARK-28588 Project: Spark Issue Type: Umbrella Components: