[jira] [Updated] (SPARK-26877) Support user-level app staging directory in yarn mode when spark.yarn.stagingDir specified

2019-02-13 Thread liupengcheng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liupengcheng updated SPARK-26877: - Summary: Support user-level app staging directory in yarn mode when spark.yarn.stagingDir

[jira] [Commented] (SPARK-26855) SparkSubmitSuite fails on a clean build

2019-02-13 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767935#comment-16767935 ] Felix Cheung commented on SPARK-26855: -- possibly. sounds like there are more cases like this, and

[jira] [Assigned] (SPARK-26794) SparkSession enableHiveSupport does not point to hive but in-memory while the SparkContext exists

2019-02-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-26794: --- Assignee: Kent Yao > SparkSession enableHiveSupport does not point to hive but in-memory

[jira] [Resolved] (SPARK-26794) SparkSession enableHiveSupport does not point to hive but in-memory while the SparkContext exists

2019-02-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-26794. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23709

[jira] [Assigned] (SPARK-26851) CachedRDDBuilder only partially implements double-checked locking

2019-02-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-26851: --- Assignee: Bruce Robbins > CachedRDDBuilder only partially implements double-checked

[jira] [Resolved] (SPARK-26851) CachedRDDBuilder only partially implements double-checked locking

2019-02-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-26851. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23768

[jira] [Commented] (SPARK-26868) Duplicate error message for implicit cartesian product in verbose explain

2019-02-13 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767915#comment-16767915 ] Takeshi Yamamuro commented on SPARK-26868: -- Thanks, [~dongjoon]! I just set that number cuz I

[jira] [Issue Comment Deleted] (SPARK-26868) Duplicate error message for implicit cartesian product in verbose explain

2019-02-13 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-26868: - Comment: was deleted (was: Thanks, [~dongjoon]! I might misunderstand the affected

[jira] [Commented] (SPARK-26868) Duplicate error message for implicit cartesian product in verbose explain

2019-02-13 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767912#comment-16767912 ] Takeshi Yamamuro commented on SPARK-26868: -- Thanks, [~dongjoon]! I might misunderstand the

[jira] [Updated] (SPARK-26876) Spark repl scala test failure on big-endian system

2019-02-13 Thread salamani (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] salamani updated SPARK-26876: - Attachment: repl_scala_issue.txt > Spark repl scala test failure on big-endian system >

[jira] [Created] (SPARK-26876) Spark repl scala test failure on big-endian system

2019-02-13 Thread salamani (JIRA)
salamani created SPARK-26876: Summary: Spark repl scala test failure on big-endian system Key: SPARK-26876 URL: https://issues.apache.org/jira/browse/SPARK-26876 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-26650) Yarn Client throws 'ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration'

2019-02-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-26650: Assignee: Marcelo Vanzin > Yarn Client throws 'ClassNotFoundException: >

[jira] [Resolved] (SPARK-26650) Yarn Client throws 'ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration'

2019-02-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26650. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23776

[jira] [Commented] (SPARK-26870) Java : Avro function to_avro and from_avro is undefined

2019-02-13 Thread Amit Baghel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767842#comment-16767842 ] Amit Baghel commented on SPARK-26870: - Yes, I have added the spark-avro dependency in pom.xml and I

[jira] [Commented] (SPARK-26870) Java : Avro function to_avro and from_avro is undefined

2019-02-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767843#comment-16767843 ] Hyukjin Kwon commented on SPARK-26870: -- Yup, I made a PR to fix. Please take a look for that when

[jira] [Assigned] (SPARK-26870) Java : Avro function to_avro and from_avro is undefined

2019-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26870: Assignee: Apache Spark > Java : Avro function to_avro and from_avro is undefined >

[jira] [Assigned] (SPARK-26870) Java : Avro function to_avro and from_avro is undefined

2019-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26870: Assignee: (was: Apache Spark) > Java : Avro function to_avro and from_avro is

[jira] [Assigned] (SPARK-26854) Support ANY subquery

2019-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26854: Assignee: Apache Spark > Support ANY subquery > > >

[jira] [Assigned] (SPARK-26854) Support ANY subquery

2019-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26854: Assignee: (was: Apache Spark) > Support ANY subquery > > >

[jira] [Updated] (SPARK-26874) With PARQUET-1414, Spark can erroneously write empty pages

2019-02-13 Thread Matt Cheah (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Cheah updated SPARK-26874: --- Description: This issue will only come up when Spark upgrades its Parquet dependency to the latest

[jira] [Updated] (SPARK-26874) With PARQUET-1414, Spark can erroneously write empty pages

2019-02-13 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue updated SPARK-26874: -- Summary: With PARQUET-1414, Spark can erroneously write empty pages (was: When we upgrade Parquet to

[jira] [Commented] (SPARK-25355) Support --proxy-user for Spark on K8s

2019-02-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767724#comment-16767724 ] Marcelo Vanzin commented on SPARK-25355: Doesn't that work already? I don't see any checks that

[jira] [Commented] (SPARK-26874) When we upgrade Parquet to 1.11+, Spark can erroneously write empty pages

2019-02-13 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767798#comment-16767798 ] Ryan Blue commented on SPARK-26874: --- To be clear, Parquet has not released any 1.11.x versions so this

[jira] [Assigned] (SPARK-26875) Add an option on FileStreamSource to include modified files

2019-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26875: Assignee: (was: Apache Spark) > Add an option on FileStreamSource to include

[jira] [Assigned] (SPARK-26875) Add an option on FileStreamSource to include modified files

2019-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26875: Assignee: Apache Spark > Add an option on FileStreamSource to include modified files >

[jira] [Commented] (SPARK-26870) Java : Avro function to_avro and from_avro is undefined

2019-02-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767774#comment-16767774 ] Hyukjin Kwon commented on SPARK-26870: -- Did you properly set

[jira] [Updated] (SPARK-26874) When we upgrade Parquet to 1.11+, Spark can erroneously write empty pages

2019-02-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26874: - Priority: Major (was: Critical) > When we upgrade Parquet to 1.11+, Spark can erroneously

[jira] [Resolved] (SPARK-25650) Make analyzer rules used in once-policy idempotent

2019-02-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25650. -- Resolution: Done Please add subtasks and reopen if there are more. > Make analyzer rules

[jira] [Assigned] (SPARK-24894) Invalid DNS name due to hostname truncation

2019-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24894: Assignee: Apache Spark > Invalid DNS name due to hostname truncation >

[jira] [Updated] (SPARK-26875) Add an option on FileStreamSource to include modified files

2019-02-13 Thread Mike Dias (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Dias updated SPARK-26875: -- Summary: Add an option on FileStreamSource to include modified files (was: Add an option on

[jira] [Created] (SPARK-26875) Add an option on FileStreamSource for include modified files

2019-02-13 Thread Mike Dias (JIRA)
Mike Dias created SPARK-26875: - Summary: Add an option on FileStreamSource for include modified files Key: SPARK-26875 URL: https://issues.apache.org/jira/browse/SPARK-26875 Project: Spark

[jira] [Assigned] (SPARK-24894) Invalid DNS name due to hostname truncation

2019-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24894: Assignee: (was: Apache Spark) > Invalid DNS name due to hostname truncation >

[jira] [Updated] (SPARK-26874) When we upgrade Parquet to 1.11+, Spark can erroneously write empty pages

2019-02-13 Thread Matt Cheah (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Cheah updated SPARK-26874: --- Priority: Critical (was: Major) > When we upgrade Parquet to 1.11+, Spark can erroneously write

[jira] [Assigned] (SPARK-25261) Standardize the default units of spark.driver|executor.memory

2019-02-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25261: -- Assignee: Marcelo Vanzin > Standardize the default units of

[jira] [Assigned] (SPARK-25261) Standardize the default units of spark.driver|executor.memory

2019-02-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25261: -- Assignee: (was: Marcelo Vanzin) > Standardize the default units of

[jira] [Updated] (SPARK-26874) When we upgrade Parquet to 1.11+, Spark can erroneously write empty pages

2019-02-13 Thread Matt Cheah (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Cheah updated SPARK-26874: --- Description: This issue will only come up when Spark upgrades its Parquet dependency to the

[jira] [Commented] (SPARK-26874) When we upgrade Parquet to 1.11+, Spark can erroneously write empty pages

2019-02-13 Thread Matt Cheah (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767736#comment-16767736 ] Matt Cheah commented on SPARK-26874: [~rdblue] [~cloud_fan] - was wondering if you had any thoughts

[jira] [Created] (SPARK-26874) When we upgrade Parquet to 1.11+, Spark can erroneously write empty pages

2019-02-13 Thread Matt Cheah (JIRA)
Matt Cheah created SPARK-26874: -- Summary: When we upgrade Parquet to 1.11+, Spark can erroneously write empty pages Key: SPARK-26874 URL: https://issues.apache.org/jira/browse/SPARK-26874 Project: Spark

[jira] [Resolved] (SPARK-26865) DataSourceV2Strategy should push normalized filters

2019-02-13 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-26865. --- Resolution: Fixed Assignee: Dongjoon Hyun Fix Version/s: 3.0.0 This is

[jira] [Assigned] (SPARK-26873) FileFormatWriter creates inconsistent MR job IDs

2019-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26873: Assignee: (was: Apache Spark) > FileFormatWriter creates inconsistent MR job IDs >

[jira] [Assigned] (SPARK-26873) FileFormatWriter creates inconsistent MR job IDs

2019-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26873: Assignee: Apache Spark > FileFormatWriter creates inconsistent MR job IDs >

[jira] [Commented] (SPARK-26150) __spark_conf__XXX.zip doesn't exist

2019-02-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767617#comment-16767617 ] Marcelo Vanzin commented on SPARK-26150: If you can provide the full YARN logs for your

[jira] [Assigned] (SPARK-26650) Yarn Client throws 'ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration'

2019-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26650: Assignee: (was: Apache Spark) > Yarn Client throws 'ClassNotFoundException: >

[jira] [Resolved] (SPARK-25766) AMCredentialRenewer can leak FS clients

2019-02-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25766. Resolution: Not A Problem I took a look at the new code (and some of the {{FileSystem}}

[jira] [Created] (SPARK-26873) FileFormatWriter creates inconsistent MR job IDs

2019-02-13 Thread Ryan Blue (JIRA)
Ryan Blue created SPARK-26873: - Summary: FileFormatWriter creates inconsistent MR job IDs Key: SPARK-26873 URL: https://issues.apache.org/jira/browse/SPARK-26873 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-26650) Yarn Client throws 'ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration'

2019-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26650: Assignee: Apache Spark > Yarn Client throws 'ClassNotFoundException: >

[jira] [Resolved] (SPARK-9209) Using executor allocation, a executor is removed but it exists in ExecutorsPage of the web ui

2019-02-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-9209. --- Resolution: Not A Problem Dead executors are now explicitly kept by Spark, so this is not an

[jira] [Resolved] (SPARK-8622) Spark 1.3.1 and 1.4.0 doesn't put executor working directory on executor classpath

2019-02-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-8622. --- Resolution: Not A Problem This works as designed. {{--jars}} are added to the Spark class

[jira] [Assigned] (SPARK-26817) Use System.nanoTime to measure time intervals

2019-02-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-26817: - Assignee: Maxim Gekk > Use System.nanoTime to measure time intervals >

[jira] [Updated] (SPARK-26872) Use a configurable value for final termination in the JobScheduler.stop() method

2019-02-13 Thread Steven Rosenberry (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Rosenberry updated SPARK-26872: -- Description: As a user of Spark, I would like to configure the timeout that controls

[jira] [Created] (SPARK-26872) Use a configurable value for final termination in the JobScheduler.stop() method

2019-02-13 Thread Steven Rosenberry (JIRA)
Steven Rosenberry created SPARK-26872: - Summary: Use a configurable value for final termination in the JobScheduler.stop() method Key: SPARK-26872 URL: https://issues.apache.org/jira/browse/SPARK-26872

[jira] [Resolved] (SPARK-26817) Use System.nanoTime to measure time intervals

2019-02-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26817. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23727

[jira] [Updated] (SPARK-26866) Support kinesis checkpoint with subSequenceNumber

2019-02-13 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-26866: -- Target Version/s: (was: 2.3.0) > Support kinesis checkpoint with subSequenceNumber >

[jira] [Updated] (SPARK-26866) Support kinesis checkpoint with subSequenceNumber

2019-02-13 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-26866: -- Fix Version/s: (was: 2.3.0) > Support kinesis checkpoint with subSequenceNumber >

[jira] [Commented] (SPARK-26829) In place standard scaler so the column remains same after transformation

2019-02-13 Thread Santokh Singh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767439#comment-16767439 ] Santokh Singh commented on SPARK-26829: --- Yes, you are right, there are workarounds that I had to

[jira] [Comment Edited] (SPARK-26860) RangeBetween docs appear to be wrong

2019-02-13 Thread Jagadesh Kiran N (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767387#comment-16767387 ] Jagadesh Kiran N edited comment on SPARK-26860 at 2/13/19 5:09 PM: --- I

[jira] [Commented] (SPARK-26860) RangeBetween docs appear to be wrong

2019-02-13 Thread Jagadesh Kiran N (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767387#comment-16767387 ] Jagadesh Kiran N commented on SPARK-26860: -- I will the below statements to differentiate the

[jira] [Updated] (SPARK-26867) Spark Support of YARN Placement Constraint

2019-02-13 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26867: -- Affects Version/s: (was: 2.4.0) 3.0.0 > Spark Support of YARN

[jira] [Commented] (SPARK-26867) Spark Support of YARN Placement Constraint

2019-02-13 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767378#comment-16767378 ] Dongjoon Hyun commented on SPARK-26867: --- Hi, [~Prabhu Joseph]. Since this is a new feature, I

[jira] [Updated] (SPARK-26868) Duplicate error message for implicit cartesian product in verbose explain

2019-02-13 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26868: -- Affects Version/s: (was: 2.4.0) 3.0.0 > Duplicate error message

[jira] [Commented] (SPARK-26868) Duplicate error message for implicit cartesian product in verbose explain

2019-02-13 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767377#comment-16767377 ] Dongjoon Hyun commented on SPARK-26868: --- Hi, [~maropu]. Since this is a new feature, I updated the

[jira] [Commented] (SPARK-26860) RangeBetween docs appear to be wrong

2019-02-13 Thread Jagadesh Kiran N (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767371#comment-16767371 ] Jagadesh Kiran N commented on SPARK-26860: -- Sure thankyou , suggest me any feature development

[jira] [Assigned] (SPARK-26871) File Source V2: avoid creating unnecessary FileIndex in the write path

2019-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26871: Assignee: (was: Apache Spark) > File Source V2: avoid creating unnecessary FileIndex

[jira] [Assigned] (SPARK-26871) File Source V2: avoid creating unnecessary FileIndex in the write path

2019-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26871: Assignee: Apache Spark > File Source V2: avoid creating unnecessary FileIndex in the

[jira] [Created] (SPARK-26871) File Source V2: avoid creating unnecessary FileIndex in the write path

2019-02-13 Thread Gengliang Wang (JIRA)
Gengliang Wang created SPARK-26871: -- Summary: File Source V2: avoid creating unnecessary FileIndex in the write path Key: SPARK-26871 URL: https://issues.apache.org/jira/browse/SPARK-26871 Project:

[jira] [Resolved] (SPARK-26798) HandleNullInputsForUDF should trust nullability

2019-02-13 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-26798. -- Resolution: Fixed Fix Version/s: 3.0.0 Resolved by 

[jira] [Commented] (SPARK-24437) Memory leak in UnsafeHashedRelation

2019-02-13 Thread Dave DeCaprio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767221#comment-16767221 ] Dave DeCaprio commented on SPARK-24437: --- Have not gotten to it yet. Dave > Memory leak in

[jira] [Assigned] (SPARK-26721) Bug in feature importance calculation in GBM (and possibly other decision tree classifiers)

2019-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26721: Assignee: Apache Spark > Bug in feature importance calculation in GBM (and possibly

[jira] [Assigned] (SPARK-26721) Bug in feature importance calculation in GBM (and possibly other decision tree classifiers)

2019-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26721: Assignee: (was: Apache Spark) > Bug in feature importance calculation in GBM (and

[jira] [Assigned] (SPARK-26835) Document configuration properties of Spark SQL Generic Load/Save Functions

2019-02-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-26835: - Assignee: Peter Horvath > Document configuration properties of Spark SQL Generic Load/Save

[jira] [Resolved] (SPARK-26835) Document configuration properties of Spark SQL Generic Load/Save Functions

2019-02-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26835. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23742

[jira] [Comment Edited] (SPARK-26858) Vectorized gapplyCollect, Arrow optimization in native R function execution

2019-02-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766941#comment-16766941 ] Hyukjin Kwon edited comment on SPARK-26858 at 2/13/19 1:46 PM: --- cc

[jira] [Commented] (SPARK-23534) Spark run on Hadoop 3.0.0

2019-02-13 Thread Darek (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767195#comment-16767195 ] Darek commented on SPARK-23534: --- It's NOT just Hadoop, it's Java 1.8 which is EOL and needs to be

[jira] [Updated] (SPARK-26863) Add minimal values for spark.driver.memory and spark.executor.memory

2019-02-13 Thread oskarryn (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] oskarryn updated SPARK-26863: - Description: I propose to change `1g` to `1g, with minimum of 472m` in "Default" column for

[jira] [Updated] (SPARK-26772) Delete ServiceCredentialProvider and make HadoopDelegationTokenProvider a developer API

2019-02-13 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-26772: -- Description: HadoopDelegationTokenProvider has basically the same functionality just like 

[jira] [Updated] (SPARK-26772) Delete ServiceCredentialProvider and make HadoopDelegationTokenProvider a developer API

2019-02-13 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-26772: -- Description: HadoopDelegationTokenProvider has basically the same functionality just like  >

[jira] [Updated] (SPARK-26772) Delete ServiceCredentialProvider and make HadoopDelegationTokenProvider a developer API

2019-02-13 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-26772: -- Summary: Delete ServiceCredentialProvider and make HadoopDelegationTokenProvider a developer

[jira] [Updated] (SPARK-26772) Delete ServiceCredentialProvider and make HadoopDelegationTokenProvider a developer API

2019-02-13 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-26772: -- Description: (was: YARNHadoopDelegationTokenManager now loads ServiceCredentialProviders

[jira] [Commented] (SPARK-26856) Python support for "from_avro" and "to_avro" APIs

2019-02-13 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767120#comment-16767120 ] Gabor Somogyi commented on SPARK-26856: --- OK, same understanding. Adapt the patch to the things

[jira] [Comment Edited] (SPARK-26858) Vectorized gapplyCollect, Arrow optimization in native R function execution

2019-02-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766941#comment-16766941 ] Hyukjin Kwon edited comment on SPARK-26858 at 2/13/19 12:21 PM: cc

[jira] [Updated] (SPARK-26870) Java : Avro function to_avro and from_avro is undefined

2019-02-13 Thread Amit Baghel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Baghel updated SPARK-26870: Component/s: SQL > Java : Avro function to_avro and from_avro is undefined >

[jira] [Created] (SPARK-26870) Java : Avro function to_avro and from_avro is undefined

2019-02-13 Thread Amit Baghel (JIRA)
Amit Baghel created SPARK-26870: --- Summary: Java : Avro function to_avro and from_avro is undefined Key: SPARK-26870 URL: https://issues.apache.org/jira/browse/SPARK-26870 Project: Spark Issue

[jira] [Comment Edited] (SPARK-26858) Vectorized gapplyCollect, Arrow optimization in native R function execution

2019-02-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766941#comment-16766941 ] Hyukjin Kwon edited comment on SPARK-26858 at 2/13/19 12:13 PM: cc

[jira] [Updated] (SPARK-26869) UDF with struct requires to have _1 and _2 as struct field names

2019-02-13 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-26869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrés Doncel Ramírez updated SPARK-26869: -- Description: When using a UDF which has a Seq of tuples as input, the struct

[jira] [Created] (SPARK-26869) UDF with struct requires to have _1 and _2 as struct field names

2019-02-13 Thread JIRA
Andrés Doncel Ramírez created SPARK-26869: - Summary: UDF with struct requires to have _1 and _2 as struct field names Key: SPARK-26869 URL: https://issues.apache.org/jira/browse/SPARK-26869

[jira] [Commented] (SPARK-23534) Spark run on Hadoop 3.0.0

2019-02-13 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767077#comment-16767077 ] Steve Loughran commented on SPARK-23534: bq. am curious to know if hadoop3 offers much

[jira] [Commented] (SPARK-24778) DateTimeUtils.getTimeZone method returns GMT time if timezone cannot be parsed

2019-02-13 Thread Renkai Ge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767068#comment-16767068 ] Renkai Ge commented on SPARK-24778: --- {{Since Spark 2.3 dropped support for java versions before Java

[jira] [Comment Edited] (SPARK-24778) DateTimeUtils.getTimeZone method returns GMT time if timezone cannot be parsed

2019-02-13 Thread Renkai Ge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767068#comment-16767068 ] Renkai Ge edited comment on SPARK-24778 at 2/13/19 11:24 AM: - Since Spark

[jira] [Updated] (SPARK-26863) Add minimal values for spark.driver.memory and spark.executor.memory

2019-02-13 Thread oskarryn (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] oskarryn updated SPARK-26863: - Description: I propose to change `1g` to `1g, with minimum of 472m` in "Default" column for

[jira] [Updated] (SPARK-26204) Optimize InSet expression

2019-02-13 Thread Anton Okolnychyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Okolnychyi updated SPARK-26204: - Description: The {{InSet}} expression was introduced in SPARK-3711 to avoid O\(n\) time

[jira] [Created] (SPARK-26868) Duplicate error message for implicit cartesian product in verbose explain

2019-02-13 Thread Takeshi Yamamuro (JIRA)
Takeshi Yamamuro created SPARK-26868: Summary: Duplicate error message for implicit cartesian product in verbose explain Key: SPARK-26868 URL: https://issues.apache.org/jira/browse/SPARK-26868

[jira] [Updated] (SPARK-26204) Optimize InSet expression

2019-02-13 Thread Anton Okolnychyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Okolnychyi updated SPARK-26204: - Description: The {{InSet}} expression was introduced in SPARK-3711 to avoid O(n) time

[jira] [Commented] (SPARK-26777) SQL worked in 2.3.2 and fails in 2.4.0

2019-02-13 Thread Jungtaek Lim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767008#comment-16767008 ] Jungtaek Lim commented on SPARK-26777: -- I actually missed reporter's last comment. Looks like it is

[jira] [Commented] (SPARK-26509) Parquet DELTA_BYTE_ARRAY is not supported in Spark 2.x's Vectorized Reader

2019-02-13 Thread Filipe Gonzaga Miranda (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767016#comment-16767016 ] Filipe Gonzaga Miranda commented on SPARK-26509: HI [~qiaojialin] - thanks for sharing

[jira] [Updated] (SPARK-26867) Spark Support of YARN Placement Constraint

2019-02-13 Thread Prabhu Joseph (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated SPARK-26867: -- Summary: Spark Support of YARN Placement Constraint (was: Spark Support for YARN Placement

[jira] [Updated] (SPARK-26867) Spark Support of YARN Placement Constraint

2019-02-13 Thread Prabhu Joseph (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated SPARK-26867: -- Component/s: YARN > Spark Support of YARN Placement Constraint >

[jira] [Created] (SPARK-26867) Spark Support for YARN Placement Constraint

2019-02-13 Thread Prabhu Joseph (JIRA)
Prabhu Joseph created SPARK-26867: - Summary: Spark Support for YARN Placement Constraint Key: SPARK-26867 URL: https://issues.apache.org/jira/browse/SPARK-26867 Project: Spark Issue Type:

[jira] [Comment Edited] (SPARK-26858) Vectorized gapplyCollect, Arrow optimization in native R function execution

2019-02-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766941#comment-16766941 ] Hyukjin Kwon edited comment on SPARK-26858 at 2/13/19 9:20 AM: --- cc

[jira] [Commented] (SPARK-26858) Vectorized gapplyCollect, Arrow optimization in native R function execution

2019-02-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766946#comment-16766946 ] Hyukjin Kwon commented on SPARK-26858: -- I am going to leave the try I made here soon. My fix is

[jira] [Commented] (SPARK-26858) Vectorized gapplyCollect, Arrow optimization in native R function execution

2019-02-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766943#comment-16766943 ] Hyukjin Kwon commented on SPARK-26858: -- BTW, just for clarification, in case of the way 1., the

  1   2   >