[jira] [Resolved] (SPARK-25178) Directly ship the StructType objects of the keySchema / valueSchema for xxxHashMapGenerator

[ https://issues.apache.org/jira/browse/SPARK-25178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-25178. --- Resolution: Fixed Assignee: Kazuaki Ishizaki Fix Version/s: 2.4.0 Issue reso

[jira] [Commented] (SPARK-25106) A new Kafka consumer gets created for every batch

[ https://issues.apache.org/jira/browse/SPARK-25106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16591170#comment-16591170 ] Tathagata Das commented on SPARK-25106: --- This is interesting! I dont know how this

[jira] [Commented] (SPARK-24630) SPIP: Support SQLStreaming in Spark

[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16591151#comment-16591151 ] Genmao Yu commented on SPARK-24630: --- [~Jackey Lee] I am glad to participate in code re

[jira] [Resolved] (SPARK-4502) Spark SQL reads unneccesary nested fields from Parquet

[ https://issues.apache.org/jira/browse/SPARK-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-4502. Resolution: Fixed Fix Version/s: 2.4.0 > Spark SQL reads unneccesary nested fields from Parquet > --

[jira] [Assigned] (SPARK-4502) Spark SQL reads unneccesary nested fields from Parquet

[ https://issues.apache.org/jira/browse/SPARK-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-4502: -- Assignee: Michael Allman > Spark SQL reads unneccesary nested fields from Parquet > --

[jira] [Closed] (SPARK-25210) spark driver apply task success info cost much time

[ https://issues.apache.org/jira/browse/SPARK-25210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangminfeng closed SPARK-25210. --- > spark driver apply task success info cost much time > ---

[jira] [Assigned] (SPARK-25221) [DEPLOY] Consistent trailing whitespace treatment of conf values

[ https://issues.apache.org/jira/browse/SPARK-25221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25221: Assignee: (was: Apache Spark) > [DEPLOY] Consistent trailing whitespace treatment of

[jira] [Commented] (SPARK-25221) [DEPLOY] Consistent trailing whitespace treatment of conf values

[ https://issues.apache.org/jira/browse/SPARK-25221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16591097#comment-16591097 ] Apache Spark commented on SPARK-25221: -- User 'gerashegalov' has created a pull requ

[jira] [Assigned] (SPARK-25221) [DEPLOY] Consistent trailing whitespace treatment of conf values

[ https://issues.apache.org/jira/browse/SPARK-25221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25221: Assignee: Apache Spark > [DEPLOY] Consistent trailing whitespace treatment of conf values

[jira] [Created] (SPARK-25221) [DEPLOY] Consistent trailing whitespace treatment of conf values

Gera Shegalov created SPARK-25221: - Summary: [DEPLOY] Consistent trailing whitespace treatment of conf values Key: SPARK-25221 URL: https://issues.apache.org/jira/browse/SPARK-25221 Project: Spark

[jira] [Commented] (SPARK-25220) [K8S] Split out node selector config between driver and executors.

[ https://issues.apache.org/jira/browse/SPARK-25220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16591058#comment-16591058 ] Apache Spark commented on SPARK-25220: -- User 'jweaver-personal' has created a pull

[jira] [Assigned] (SPARK-25220) [K8S] Split out node selector config between driver and executors.

[ https://issues.apache.org/jira/browse/SPARK-25220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25220: Assignee: Apache Spark > [K8S] Split out node selector config between driver and executor

[jira] [Assigned] (SPARK-25220) [K8S] Split out node selector config between driver and executors.

[ https://issues.apache.org/jira/browse/SPARK-25220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25220: Assignee: (was: Apache Spark) > [K8S] Split out node selector config between driver a

[jira] [Created] (SPARK-25220) [K8S] Split out node selector config between driver and executors.

Jonathan A Weaver created SPARK-25220: - Summary: [K8S] Split out node selector config between driver and executors. Key: SPARK-25220 URL: https://issues.apache.org/jira/browse/SPARK-25220 Project:

[jira] [Resolved] (SPARK-25209) Optimization in Dataset.apply for DataFrames

[ https://issues.apache.org/jira/browse/SPARK-25209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-25209. --- Resolution: Fixed Assignee: Bogdan Raducanu Fix Version/s: 2.4.0 > O

[jira] [Commented] (SPARK-25210) spark driver apply task success info cost much time

[ https://issues.apache.org/jira/browse/SPARK-25210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16591050#comment-16591050 ] Hyukjin Kwon commented on SPARK-25210: -- If it's unclear whether it is an issue or n

[jira] [Resolved] (SPARK-25210) spark driver apply task success info cost much time

[ https://issues.apache.org/jira/browse/SPARK-25210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25210. -- Resolution: Invalid Let's reopen when it's clear if that's an issue. > spark driver apply tas

[jira] [Resolved] (SPARK-23425) load data for hdfs file path with wild card usage is not working properly

[ https://issues.apache.org/jira/browse/SPARK-23425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-23425. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20611 [https://gi

[jira] [Assigned] (SPARK-23425) load data for hdfs file path with wild card usage is not working properly

[ https://issues.apache.org/jira/browse/SPARK-23425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-23425: Assignee: Sujith > load data for hdfs file path with wild card usage is not working prope

[jira] [Resolved] (SPARK-25205) typo in spark.network.crypto.keyFactoryIteration

[ https://issues.apache.org/jira/browse/SPARK-25205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25205. -- Resolution: Fixed Fix Version/s: 2.3.2 2.4.0 Issue resolved by pull

[jira] [Assigned] (SPARK-25205) typo in spark.network.crypto.keyFactoryIteration

[ https://issues.apache.org/jira/browse/SPARK-25205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-25205: Assignee: Imran Rashid > typo in spark.network.crypto.keyFactoryIteration > -

[jira] [Commented] (SPARK-24814) Relationship between catalog and datasources

[ https://issues.apache.org/jira/browse/SPARK-24814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16591029#comment-16591029 ] Xiao Li commented on SPARK-24814: - Also cc [~cloud_fan] > Relationship between catalog

[jira] [Created] (SPARK-25219) KMeans Clustering - Text Data - Results are incorrect

Vasanthkumar Velayudham created SPARK-25219: --- Summary: KMeans Clustering - Text Data - Results are incorrect Key: SPARK-25219 URL: https://issues.apache.org/jira/browse/SPARK-25219 Project: S

[jira] [Commented] (SPARK-24564) Add test suite for RecordBinaryComparator

[ https://issues.apache.org/jira/browse/SPARK-24564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590923#comment-16590923 ] Apache Spark commented on SPARK-24564: -- User 'henryr' has created a pull request fo

[jira] [Commented] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590921#comment-16590921 ] Apache Spark commented on SPARK-23207: -- User 'henryr' has created a pull request fo

[jira] [Commented] (SPARK-25114) RecordBinaryComparator may return wrong result when subtraction between two words is divisible by Integer.MAX_VALUE

[ https://issues.apache.org/jira/browse/SPARK-25114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590924#comment-16590924 ] Apache Spark commented on SPARK-25114: -- User 'henryr' has created a pull request fo

[jira] [Commented] (SPARK-22905) Fix ChiSqSelectorModel, GaussianMixtureModel save implementation for Row order issues

[ https://issues.apache.org/jira/browse/SPARK-22905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590922#comment-16590922 ] Apache Spark commented on SPARK-22905: -- User 'henryr' has created a pull request fo

[jira] [Updated] (SPARK-25124) VectorSizeHint.size is buggy, breaking streaming pipeline

[ https://issues.apache.org/jira/browse/SPARK-25124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-25124: -- Target Version/s: 2.3.2 > VectorSizeHint.size is buggy, breaking streaming pipeline >

[jira] [Commented] (SPARK-25124) VectorSizeHint.size is buggy, breaking streaming pipeline

[ https://issues.apache.org/jira/browse/SPARK-25124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590919#comment-16590919 ] Joseph K. Bradley commented on SPARK-25124: --- I merged https://github.com/apach

[jira] [Updated] (SPARK-25124) VectorSizeHint.size is buggy, breaking streaming pipeline

[ https://issues.apache.org/jira/browse/SPARK-25124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-25124: -- Target Version/s: 2.3.2, 2.4.0 (was: 2.3.2) > VectorSizeHint.size is buggy, breaking

[jira] [Updated] (SPARK-25124) VectorSizeHint.size is buggy, breaking streaming pipeline

[ https://issues.apache.org/jira/browse/SPARK-25124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-25124: -- Fix Version/s: 2.4.0 > VectorSizeHint.size is buggy, breaking streaming pipeline > ---

[jira] [Updated] (SPARK-25124) VectorSizeHint.size is buggy, breaking streaming pipeline

[ https://issues.apache.org/jira/browse/SPARK-25124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-25124: -- Shepherd: Joseph K. Bradley > VectorSizeHint.size is buggy, breaking streaming pipelin

[jira] [Assigned] (SPARK-25124) VectorSizeHint.size is buggy, breaking streaming pipeline

[ https://issues.apache.org/jira/browse/SPARK-25124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley reassigned SPARK-25124: - Assignee: Huaxin Gao > VectorSizeHint.size is buggy, breaking streaming pipelin

[jira] [Commented] (SPARK-25202) SQL Function Split Should Respect Limit Argument

[ https://issues.apache.org/jira/browse/SPARK-25202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590874#comment-16590874 ] Liang-Chi Hsieh commented on SPARK-25202: - [~phegstrom] No problem. Please submi

[jira] [Commented] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590847#comment-16590847 ] Bruce Robbins commented on SPARK-23207: --- Will we be back-porting this to 2.1, or d

[jira] [Comment Edited] (SPARK-8582) Optimize checkpointing to avoid computing an RDD twice

[ https://issues.apache.org/jira/browse/SPARK-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590733#comment-16590733 ] Baris ERGUN edited comment on SPARK-8582 at 8/23/18 9:46 PM: -

[jira] [Commented] (SPARK-24539) HistoryServer does not display metrics from tasks that complete after stage failure

[ https://issues.apache.org/jira/browse/SPARK-24539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590837#comment-16590837 ] Ankur Gupta commented on SPARK-24539: - Verified this issue is same as SPARK-24415. T

[jira] [Assigned] (SPARK-25218) Potential resource leaks in TransportServer and SocketAuthHelper

[ https://issues.apache.org/jira/browse/SPARK-25218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25218: Assignee: Apache Spark (was: Shixiong Zhu) > Potential resource leaks in TransportServer

[jira] [Assigned] (SPARK-25218) Potential resource leaks in TransportServer and SocketAuthHelper

[ https://issues.apache.org/jira/browse/SPARK-25218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25218: Assignee: Shixiong Zhu (was: Apache Spark) > Potential resource leaks in TransportServer

[jira] [Commented] (SPARK-25218) Potential resource leaks in TransportServer and SocketAuthHelper

[ https://issues.apache.org/jira/browse/SPARK-25218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590766#comment-16590766 ] Apache Spark commented on SPARK-25218: -- User 'zsxwing' has created a pull request f

[jira] [Created] (SPARK-25218) Potential resource leaks in TransportServer and SocketAuthHelper

Shixiong Zhu created SPARK-25218: Summary: Potential resource leaks in TransportServer and SocketAuthHelper Key: SPARK-25218 URL: https://issues.apache.org/jira/browse/SPARK-25218 Project: Spark

[jira] [Created] (SPARK-25217) Error thrown when creating BlockMatrix

cs5090237 created SPARK-25217: - Summary: Error thrown when creating BlockMatrix Key: SPARK-25217 URL: https://issues.apache.org/jira/browse/SPARK-25217 Project: Spark Issue Type: Bug Co

[jira] [Commented] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590748#comment-16590748 ] Apache Spark commented on SPARK-24415: -- User 'ankuriitg' has created a pull request

[jira] [Assigned] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24415: Assignee: (was: Apache Spark) > Stage page aggregated executor metrics wrong when fai

[jira] [Assigned] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24415: Assignee: Apache Spark > Stage page aggregated executor metrics wrong when failures > --

[jira] [Commented] (SPARK-8582) Optimize checkpointing to avoid computing an RDD twice

[ https://issues.apache.org/jira/browse/SPARK-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590733#comment-16590733 ] Baris ERGUN commented on SPARK-8582: +1 when this issue is planned to be resolved. I

[jira] [Resolved] (SPARK-25204) rate source test is flaky

[ https://issues.apache.org/jira/browse/SPARK-25204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-25204. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22191 [https://

[jira] [Assigned] (SPARK-25204) rate source test is flaky

[ https://issues.apache.org/jira/browse/SPARK-25204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-25204: - Assignee: Jose Torres > rate source test is flaky > - > >

[jira] [Updated] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23207: Fix Version/s: 2.2.3 > Shuffle+Repartition on an DataFrame could lead to incorrect answers > -

[jira] [Updated] (SPARK-24564) Add test suite for RecordBinaryComparator

[ https://issues.apache.org/jira/browse/SPARK-24564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24564: Fix Version/s: 2.3.2 2.2.3 > Add test suite for RecordBinaryComparator > --

[jira] [Updated] (SPARK-25114) RecordBinaryComparator may return wrong result when subtraction between two words is divisible by Integer.MAX_VALUE

[ https://issues.apache.org/jira/browse/SPARK-25114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25114: Fix Version/s: 2.2.3 > RecordBinaryComparator may return wrong result when subtraction between two > word

[jira] [Assigned] (SPARK-25216) Provide better error message when a column contains dot and needs backticks quote

[ https://issues.apache.org/jira/browse/SPARK-25216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25216: Assignee: Apache Spark > Provide better error message when a column contains dot and need

[jira] [Assigned] (SPARK-25216) Provide better error message when a column contains dot and needs backticks quote

[ https://issues.apache.org/jira/browse/SPARK-25216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25216: Assignee: (was: Apache Spark) > Provide better error message when a column contains d

[jira] [Commented] (SPARK-25216) Provide better error message when a column contains dot and needs backticks quote

[ https://issues.apache.org/jira/browse/SPARK-25216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590681#comment-16590681 ] Apache Spark commented on SPARK-25216: -- User 'icexelloss' has created a pull reques

[jira] [Updated] (SPARK-25216) Provide better error message when a column contains dot and needs backticks quote

[ https://issues.apache.org/jira/browse/SPARK-25216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Jin updated SPARK-25216: --- Description: The current error message is  often confusing to a new Spark user that a column containing "."

[jira] [Updated] (SPARK-25216) Provide better error message when a column contains dot and needs backticks quote

[ https://issues.apache.org/jira/browse/SPARK-25216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Jin updated SPARK-25216: --- Description: The current error message is  often confusing to a new Spark user that a column containing "."

[jira] [Created] (SPARK-25216) Provide better error message when a column contains dot and needs backticks quote

Li Jin created SPARK-25216: -- Summary: Provide better error message when a column contains dot and needs backticks quote Key: SPARK-25216 URL: https://issues.apache.org/jira/browse/SPARK-25216 Project: Spark

[jira] [Updated] (SPARK-25214) Kafka v2 source may return duplicated records when `failOnDataLoss` is `false`

[ https://issues.apache.org/jira/browse/SPARK-25214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25214: - Description: When there are missing offsets, Kafka v2 source may return duplicated records when

[jira] [Commented] (SPARK-25200) Allow setting HADOOP_CONF_DIR as a spark property

[ https://issues.apache.org/jira/browse/SPARK-25200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590642#comment-16590642 ] Adam Balogh commented on SPARK-25200: - Great! I can work on it, I was thinking of ad

[jira] [Assigned] (SPARK-25214) Kafka v2 source may return duplicated records when `failOnDataLoss` is `false`

[ https://issues.apache.org/jira/browse/SPARK-25214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25214: Assignee: Apache Spark (was: Shixiong Zhu) > Kafka v2 source may return duplicated recor

[jira] [Assigned] (SPARK-25214) Kafka v2 source may return duplicated records when `failOnDataLoss` is `false`

[ https://issues.apache.org/jira/browse/SPARK-25214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25214: Assignee: Shixiong Zhu (was: Apache Spark) > Kafka v2 source may return duplicated recor

[jira] [Assigned] (SPARK-25214) Kafka v2 source may return duplicated records when `failOnDataLoss` is `false`

[ https://issues.apache.org/jira/browse/SPARK-25214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25214: Assignee: Apache Spark (was: Shixiong Zhu) > Kafka v2 source may return duplicated recor

[jira] [Assigned] (SPARK-25214) Kafka v2 source may return duplicated records when `failOnDataLoss` is `false`

[ https://issues.apache.org/jira/browse/SPARK-25214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25214: Assignee: Shixiong Zhu (was: Apache Spark) > Kafka v2 source may return duplicated recor

[jira] [Commented] (SPARK-25214) Kafka v2 source may return duplicated records when `failOnDataLoss` is `false`

[ https://issues.apache.org/jira/browse/SPARK-25214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590629#comment-16590629 ] Apache Spark commented on SPARK-25214: -- User 'zsxwing' has created a pull request f

[jira] [Created] (SPARK-25215) Make PipelineModel public

Nicholas Resnick created SPARK-25215: Summary: Make PipelineModel public Key: SPARK-25215 URL: https://issues.apache.org/jira/browse/SPARK-25215 Project: Spark Issue Type: Wish

[jira] [Commented] (SPARK-25200) Allow setting HADOOP_CONF_DIR as a spark property

[ https://issues.apache.org/jira/browse/SPARK-25200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590621#comment-16590621 ] Marcelo Vanzin commented on SPARK-25200: Sounds like a good idea, just need some

[jira] [Commented] (SPARK-25200) Allow setting HADOOP_CONF_DIR as a spark property

[ https://issues.apache.org/jira/browse/SPARK-25200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590618#comment-16590618 ] Adam Balogh commented on SPARK-25200: - cc [~vanzin] > Allow setting HADOOP_CONF_DIR

[jira] [Assigned] (SPARK-25213) DataSourceV2 doesn't seem to produce unsafe rows

[ https://issues.apache.org/jira/browse/SPARK-25213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25213: Assignee: Apache Spark > DataSourceV2 doesn't seem to produce unsafe rows >

[jira] [Assigned] (SPARK-25213) DataSourceV2 doesn't seem to produce unsafe rows

[ https://issues.apache.org/jira/browse/SPARK-25213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25213: Assignee: (was: Apache Spark) > DataSourceV2 doesn't seem to produce unsafe rows > -

[jira] [Commented] (SPARK-25213) DataSourceV2 doesn't seem to produce unsafe rows

[ https://issues.apache.org/jira/browse/SPARK-25213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590585#comment-16590585 ] Apache Spark commented on SPARK-25213: -- User 'rdblue' has created a pull request fo

[jira] [Created] (SPARK-25214) Kafka v2 source may return duplicated records when `failOnDataLoss` is `false`

Shixiong Zhu created SPARK-25214: Summary: Kafka v2 source may return duplicated records when `failOnDataLoss` is `false` Key: SPARK-25214 URL: https://issues.apache.org/jira/browse/SPARK-25214 Projec

[jira] [Commented] (SPARK-25213) DataSourceV2 doesn't seem to produce unsafe rows

[ https://issues.apache.org/jira/browse/SPARK-25213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590531#comment-16590531 ] Ryan Blue commented on SPARK-25213: --- Sorry, I just realized the point is that the filt

[jira] [Updated] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiang Xingbo updated SPARK-23207: - Affects Version/s: 1.6.0 2.0.0 2.1.0

[jira] [Commented] (SPARK-25202) SQL Function Split Should Respect Limit Argument

[ https://issues.apache.org/jira/browse/SPARK-25202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590500#comment-16590500 ] Parker Hegstrom commented on SPARK-25202: - [~viirya] I do have time so can take

[jira] [Updated] (SPARK-25206) Wrong data may be returned when enable pushdown

[ https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-25206: --- Target Version/s: 2.3.2 > Wrong data may be returned when enable pushdown >

[jira] [Commented] (SPARK-25206) Wrong data may be returned when enable pushdown

[ https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590480#comment-16590480 ] Marcelo Vanzin commented on SPARK-25206: Updating to blocker given recent discus

[jira] [Updated] (SPARK-25206) Wrong data may be returned when enable pushdown

[ https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-25206: --- Labels: correctness (was: ) > Wrong data may be returned when enable pushdown > ---

[jira] [Updated] (SPARK-25206) Wrong data may be returned when enable pushdown

[ https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-25206: --- Priority: Blocker (was: Major) > Wrong data may be returned when enable pushdown >

[jira] [Commented] (SPARK-24768) Have a built-in AVRO data source implementation

[ https://issues.apache.org/jira/browse/SPARK-24768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590441#comment-16590441 ] Antonio Murgia commented on SPARK-24768: Will this support UDT to the extent the

[jira] [Issue Comment Deleted] (SPARK-24772) support reading AVRO logical types - Date

[ https://issues.apache.org/jira/browse/SPARK-24772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antonio Murgia updated SPARK-24772: --- Comment: was deleted (was: Will this support UDT to the extent the parquet reader/writer doe

[jira] [Commented] (SPARK-24772) support reading AVRO logical types - Date

[ https://issues.apache.org/jira/browse/SPARK-24772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590439#comment-16590439 ] Antonio Murgia commented on SPARK-24772: Will this support UDT to the extent the

[jira] [Commented] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590435#comment-16590435 ] Daniel Darabos commented on SPARK-23207: Sorry, could you clarify the fix versio

[jira] [Commented] (SPARK-25213) DataSourceV2 doesn't seem to produce unsafe rows

[ https://issues.apache.org/jira/browse/SPARK-25213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590406#comment-16590406 ] Ryan Blue commented on SPARK-25213: --- [~cloud_fan], that PR ensures that there is a Pro

[jira] [Comment Edited] (SPARK-25073) Spark-submit on Yarn Task : When the yarn.nodemanager.resource.memory-mb and/or yarn.scheduler.maximum-allocation-mb is insufficient, Spark always reports an erro

[ https://issues.apache.org/jira/browse/SPARK-25073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590396#comment-16590396 ] Sujith edited comment on SPARK-25073 at 8/23/18 3:48 PM: -  Yes,

[jira] [Commented] (SPARK-25213) DataSourceV2 doesn't seem to produce unsafe rows

[ https://issues.apache.org/jira/browse/SPARK-25213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590397#comment-16590397 ] Wenchen Fan commented on SPARK-25213: - [~rdblue] I think we have a problem here. In

[jira] [Comment Edited] (SPARK-25073) Spark-submit on Yarn Task : When the yarn.nodemanager.resource.memory-mb and/or yarn.scheduler.maximum-allocation-mb is insufficient, Spark always reports an erro

[ https://issues.apache.org/jira/browse/SPARK-25073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590396#comment-16590396 ] Sujith edited comment on SPARK-25073 at 8/23/18 3:43 PM: -  Yes,

[jira] [Comment Edited] (SPARK-25073) Spark-submit on Yarn Task : When the yarn.nodemanager.resource.memory-mb and/or yarn.scheduler.maximum-allocation-mb is insufficient, Spark always reports an erro

[ https://issues.apache.org/jira/browse/SPARK-25073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590396#comment-16590396 ] Sujith edited comment on SPARK-25073 at 8/23/18 3:42 PM: -  Yes,

[jira] [Commented] (SPARK-25073) Spark-submit on Yarn Task : When the yarn.nodemanager.resource.memory-mb and/or yarn.scheduler.maximum-allocation-mb is insufficient, Spark always reports an error req

[ https://issues.apache.org/jira/browse/SPARK-25073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590396#comment-16590396 ] Sujith commented on SPARK-25073:  Yes, in the executor memory validation check we are di

[jira] [Assigned] (SPARK-25212) Support Filter in ConvertToLocalRelation

[ https://issues.apache.org/jira/browse/SPARK-25212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25212: Assignee: Apache Spark > Support Filter in ConvertToLocalRelation > -

[jira] [Assigned] (SPARK-25212) Support Filter in ConvertToLocalRelation

[ https://issues.apache.org/jira/browse/SPARK-25212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25212: Assignee: (was: Apache Spark) > Support Filter in ConvertToLocalRelation > --

[jira] [Created] (SPARK-25213) DataSourceV2 doesn't seem to produce unsafe rows

Li Jin created SPARK-25213: -- Summary: DataSourceV2 doesn't seem to produce unsafe rows Key: SPARK-25213 URL: https://issues.apache.org/jira/browse/SPARK-25213 Project: Spark Issue Type: Task

[jira] [Commented] (SPARK-25212) Support Filter in ConvertToLocalRelation

[ https://issues.apache.org/jira/browse/SPARK-25212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590364#comment-16590364 ] Apache Spark commented on SPARK-25212: -- User 'bogdanrdc' has created a pull request

[jira] [Assigned] (SPARK-25208) Loosen Cast.forceNullable for DecimalType.

[ https://issues.apache.org/jira/browse/SPARK-25208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-25208: --- Assignee: Takuya Ueshin > Loosen Cast.forceNullable for DecimalType. >

[jira] [Resolved] (SPARK-25208) Loosen Cast.forceNullable for DecimalType.

[ https://issues.apache.org/jira/browse/SPARK-25208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-25208. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22200 [https://gith

[jira] [Assigned] (SPARK-25196) Analyze column statistics in cached query

[ https://issues.apache.org/jira/browse/SPARK-25196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25196: Assignee: Apache Spark > Analyze column statistics in cached query >

[jira] [Commented] (SPARK-25196) Analyze column statistics in cached query

[ https://issues.apache.org/jira/browse/SPARK-25196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590323#comment-16590323 ] Apache Spark commented on SPARK-25196: -- User 'maropu' has created a pull request fo

[jira] [Assigned] (SPARK-25196) Analyze column statistics in cached query

[ https://issues.apache.org/jira/browse/SPARK-25196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25196: Assignee: (was: Apache Spark) > Analyze column statistics in cached query > -

[jira] [Comment Edited] (SPARK-25196) Analyze column statistics in cached query

[ https://issues.apache.org/jira/browse/SPARK-25196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589604#comment-16589604 ] Takeshi Yamamuro edited comment on SPARK-25196 at 8/23/18 2:34 PM: ---

[jira] [Commented] (SPARK-25211) speculation and fetch failed result in hang of job

[ https://issues.apache.org/jira/browse/SPARK-25211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590267#comment-16590267 ] Lijia Liu commented on SPARK-25211: --- In https://issues.apache.org/jira/browse/SPARK-23

[jira] [Resolved] (SPARK-25126) avoid creating OrcFile.Reader for all orc files

[ https://issues.apache.org/jira/browse/SPARK-25126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25126. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22157 [https://gi

  1   2   >