[jira] [Commented] (SPARK-26762) Arrow optimization for conversion from Spark DataFrame to R DataFrame

[ https://issues.apache.org/jira/browse/SPARK-26762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764092#comment-16764092 ] Felix Cheung commented on SPARK-26762: -- does this include head, take etc? > Arrow optimization for

[jira] [Commented] (SPARK-26762) Arrow optimization for conversion from Spark DataFrame to R DataFrame

[ https://issues.apache.org/jira/browse/SPARK-26762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764094#comment-16764094 ] Hyukjin Kwon commented on SPARK-26762: -- Basically yes, I target all the APIs that can return R data

[jira] [Comment Edited] (SPARK-26762) Arrow optimization for conversion from Spark DataFrame to R DataFrame

[ https://issues.apache.org/jira/browse/SPARK-26762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764092#comment-16764092 ] Felix Cheung edited comment on SPARK-26762 at 2/9/19 7:35 AM: -- does this

[jira] [Commented] (SPARK-26840) Avoid cost-based join reorder in presence of join hints

[ https://issues.apache.org/jira/browse/SPARK-26840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764084#comment-16764084 ] Hyukjin Kwon commented on SPARK-26840: -- [~maryannxue], can you fill the description of the JiRA? >

[jira] [Commented] (SPARK-26829) In place standard scaler so the column remains same after transformation

[ https://issues.apache.org/jira/browse/SPARK-26829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764085#comment-16764085 ] Hyukjin Kwon commented on SPARK-26829: -- Please fill the JIRA description. > In place standard

[jira] [Resolved] (SPARK-26844) Parquet Reader exception - ArrayIndexOutOfBound should give more information to user

[ https://issues.apache.org/jira/browse/SPARK-26844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26844. -- Resolution: Incomplete Cannot reproduce this with the current information. Yes, reproducer

[jira] [Assigned] (SPARK-26816) Benchmark for XORShiftRandom

[ https://issues.apache.org/jira/browse/SPARK-26816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26816: Assignee: Apache Spark > Benchmark for XORShiftRandom > > >

[jira] [Assigned] (SPARK-26816) Benchmark for XORShiftRandom

[ https://issues.apache.org/jira/browse/SPARK-26816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26816: Assignee: (was: Apache Spark) > Benchmark for XORShiftRandom >

[jira] [Assigned] (SPARK-26852) CrossValidator: support transforming metrics to absolute values prior to min/max test

[ https://issues.apache.org/jira/browse/SPARK-26852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26852: Assignee: (was: Apache Spark) > CrossValidator: support transforming metrics to

[jira] [Assigned] (SPARK-26852) CrossValidator: support transforming metrics to absolute values prior to min/max test

[ https://issues.apache.org/jira/browse/SPARK-26852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26852: Assignee: Apache Spark > CrossValidator: support transforming metrics to absolute values

[jira] [Updated] (SPARK-26852) CrossValidator: support transforming metrics to absolute values prior to min/max test

[ https://issues.apache.org/jira/browse/SPARK-26852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Weber updated SPARK-26852: -- Description: When writing a custom Evaluator with PySpark, it's often useful to be able to support

[jira] [Created] (SPARK-26852) CrossValidator: support transforming metrics to absolute values prior to min/max test

Ben Weber created SPARK-26852: - Summary: CrossValidator: support transforming metrics to absolute values prior to min/max test Key: SPARK-26852 URL: https://issues.apache.org/jira/browse/SPARK-26852

[jira] [Resolved] (SPARK-26821) filters not working with char datatype when querying against hive table

[ https://issues.apache.org/jira/browse/SPARK-26821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26821. --- Resolution: Not A Problem I don't think there's something to document at this point. The behavior

[jira] [Updated] (SPARK-26851) CachedRDDBuilder only partially implements double-checked locking

[ https://issues.apache.org/jira/browse/SPARK-26851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-26851: -- Description: In CachedRDDBuilder, {{cachedColumnBuffers}} uses double-checked locking to

[jira] [Updated] (SPARK-26851) CachedRDDBuilder only partially implements double-checked locking

[ https://issues.apache.org/jira/browse/SPARK-26851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-26851: -- Labels: (was: con) > CachedRDDBuilder only partially implements double-checked locking >

[jira] [Updated] (SPARK-26851) CachedRDDBuilder only partially implements double-checked locking

[ https://issues.apache.org/jira/browse/SPARK-26851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-26851: -- Labels: con (was: ) > CachedRDDBuilder only partially implements double-checked locking >

[jira] [Comment Edited] (SPARK-26804) Spark sql carries newline char from last csv column when imported

[ https://issues.apache.org/jira/browse/SPARK-26804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763728#comment-16763728 ] Bruce Robbins edited comment on SPARK-26804 at 2/8/19 10:13 PM:

[jira] [Commented] (SPARK-26688) Provide configuration of initially blacklisted YARN nodes

[ https://issues.apache.org/jira/browse/SPARK-26688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763922#comment-16763922 ] Imran Rashid commented on SPARK-26688: -- I haven't heard any more objections to this, so I think its

[jira] [Assigned] (SPARK-26766) Remove the list of filesystems from HadoopDelegationTokenProvider.obtainDelegationTokens

[ https://issues.apache.org/jira/browse/SPARK-26766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26766: -- Assignee: Gabor Somogyi > Remove the list of filesystems from >

[jira] [Resolved] (SPARK-26766) Remove the list of filesystems from HadoopDelegationTokenProvider.obtainDelegationTokens

[ https://issues.apache.org/jira/browse/SPARK-26766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26766. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23698

[jira] [Resolved] (SPARK-26845) Avro to_avro from_avro roundtrip fails if data type is string

[ https://issues.apache.org/jira/browse/SPARK-26845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved SPARK-26845. --- Resolution: Not A Problem > Avro to_avro from_avro roundtrip fails if data type is string >

[jira] [Commented] (SPARK-26845) Avro to_avro from_avro roundtrip fails if data type is string

[ https://issues.apache.org/jira/browse/SPARK-26845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763868#comment-16763868 ] Gabor Somogyi commented on SPARK-26845: --- I think I've found the reason for the second question:

[jira] [Reopened] (SPARK-26849) Introduce new option to Kafka source: offset by timestamp (starting/ending)

[ https://issues.apache.org/jira/browse/SPARK-26849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-26849: --- > Introduce new option to Kafka source: offset by timestamp (starting/ending) >

[jira] [Commented] (SPARK-26845) Avro to_avro from_avro roundtrip fails if data type is string

[ https://issues.apache.org/jira/browse/SPARK-26845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763848#comment-16763848 ] Gabor Somogyi commented on SPARK-26845: --- [~attilapiros] Thanks for the help, this explains why the

[jira] [Resolved] (SPARK-26849) Introduce new option to Kafka source: offset by timestamp (starting/ending)

[ https://issues.apache.org/jira/browse/SPARK-26849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-26849. --- Resolution: Duplicate > Introduce new option to Kafka source: offset by timestamp

[jira] [Closed] (SPARK-26849) Introduce new option to Kafka source: offset by timestamp (starting/ending)

[ https://issues.apache.org/jira/browse/SPARK-26849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-26849. - > Introduce new option to Kafka source: offset by timestamp (starting/ending) >

[jira] [Updated] (SPARK-26837) Pruning nested fields from object serializers

[ https://issues.apache.org/jira/browse/SPARK-26837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26837: -- Issue Type: Sub-task (was: Improvement) Parent: SPARK-25603 > Pruning nested fields

[jira] [Commented] (SPARK-21492) Memory leak in SortMergeJoin

[ https://issues.apache.org/jira/browse/SPARK-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763825#comment-16763825 ] Tao Luo commented on SPARK-21492: - Can someone add 'affects version' 2.4.0 as well?  > Memory leak in

[jira] [Created] (SPARK-26851) CachedRDDBuilder only partially implements double-checked locking

Bruce Robbins created SPARK-26851: - Summary: CachedRDDBuilder only partially implements double-checked locking Key: SPARK-26851 URL: https://issues.apache.org/jira/browse/SPARK-26851 Project: Spark

[jira] [Updated] (SPARK-21492) Memory leak in SortMergeJoin

[ https://issues.apache.org/jira/browse/SPARK-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21492: -- Affects Version/s: 3.0.0 > Memory leak in SortMergeJoin > > >

[jira] [Commented] (SPARK-21492) Memory leak in SortMergeJoin

[ https://issues.apache.org/jira/browse/SPARK-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763820#comment-16763820 ] Tao Luo commented on SPARK-21492: - If SortMergeJoinScanner doesn't consume the iterator from

[jira] [Commented] (SPARK-26851) CachedRDDBuilder only partially implements double-checked locking

[ https://issues.apache.org/jira/browse/SPARK-26851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763818#comment-16763818 ] Bruce Robbins commented on SPARK-26851: --- [~maropu] [~cloud_fan] I will let this Jira marinate for

[jira] [Commented] (SPARK-24657) SortMergeJoin may cause SparkOutOfMemory in execution memory because of not cleanup resource when finished the merge join

[ https://issues.apache.org/jira/browse/SPARK-24657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763808#comment-16763808 ] Dongjoon Hyun commented on SPARK-24657: --- Hi, [~wangzjie1], [~maropu], [~taoluo]. I close this

[jira] [Updated] (SPARK-21492) Memory leak in SortMergeJoin

[ https://issues.apache.org/jira/browse/SPARK-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21492: -- Affects Version/s: (was: 3.0.0) 2.2.0 2.3.0

[jira] [Closed] (SPARK-24657) SortMergeJoin may cause SparkOutOfMemory in execution memory because of not cleanup resource when finished the merge join

[ https://issues.apache.org/jira/browse/SPARK-24657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-24657. - > SortMergeJoin may cause SparkOutOfMemory in execution memory because of not > cleanup resource

[jira] [Resolved] (SPARK-24657) SortMergeJoin may cause SparkOutOfMemory in execution memory because of not cleanup resource when finished the merge join

[ https://issues.apache.org/jira/browse/SPARK-24657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-24657. --- Resolution: Duplicate > SortMergeJoin may cause SparkOutOfMemory in execution memory

[jira] [Resolved] (SPARK-26389) Add force delete temp checkpoint configuration

[ https://issues.apache.org/jira/browse/SPARK-26389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-26389. --- Resolution: Fixed Assignee: Gabor Somogyi Fix Version/s: 3.0.0 This is

[jira] [Updated] (SPARK-26389) Add force delete temp checkpoint configuration

[ https://issues.apache.org/jira/browse/SPARK-26389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26389: -- Issue Type: Improvement (was: Bug) > Add force delete temp checkpoint configuration >

[jira] [Updated] (SPARK-26389) Add force delete temp checkpoint configuration

[ https://issues.apache.org/jira/browse/SPARK-26389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26389: -- Summary: Add force delete temp checkpoint configuration (was: temp checkpoint folder at

[jira] [Assigned] (SPARK-26185) add weightCol in python MulticlassClassificationEvaluator

[ https://issues.apache.org/jira/browse/SPARK-26185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk reassigned SPARK-26185: --- Assignee: Huaxin Gao > add weightCol in python MulticlassClassificationEvaluator >

[jira] [Commented] (SPARK-26804) Spark sql carries newline char from last csv column when imported

[ https://issues.apache.org/jira/browse/SPARK-26804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763758#comment-16763758 ] Raj commented on SPARK-26804: - Hi [~bersprockets]    Any idea on when 3.0 will be available? I am using

[jira] [Commented] (SPARK-26804) Spark sql carries newline char from last csv column when imported

[ https://issues.apache.org/jira/browse/SPARK-26804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763728#comment-16763728 ] Bruce Robbins commented on SPARK-26804: --- v2.4.0: Fails as described Tip of branch-2.4: Fails as

[jira] [Commented] (SPARK-26841) Timestamp pushdown on Kafka table

[ https://issues.apache.org/jira/browse/SPARK-26841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763697#comment-16763697 ] Tomas Bartalos commented on SPARK-26841: Thank you for letting me know, I've filed the PR. I'm

[jira] [Assigned] (SPARK-26841) Timestamp pushdown on Kafka table

[ https://issues.apache.org/jira/browse/SPARK-26841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26841: Assignee: Apache Spark > Timestamp pushdown on Kafka table >

[jira] [Assigned] (SPARK-26841) Timestamp pushdown on Kafka table

[ https://issues.apache.org/jira/browse/SPARK-26841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26841: Assignee: (was: Apache Spark) > Timestamp pushdown on Kafka table >

[jira] [Reopened] (SPARK-26804) Spark sql carries newline char from last csv column when imported

[ https://issues.apache.org/jira/browse/SPARK-26804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raj reopened SPARK-26804: - Hi,   I sent all details to you. Let me know if you need more info.   Thanks, Raj > Spark sql carries newline

[jira] [Commented] (SPARK-26841) Timestamp pushdown on Kafka table

[ https://issues.apache.org/jira/browse/SPARK-26841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763633#comment-16763633 ] Jungtaek Lim commented on SPARK-26841: -- FYI: I've just filed issue and submitted PR regarding

[jira] [Resolved] (SPARK-26849) Introduce new option to Kafka source: offset by timestamp (starting/ending)

[ https://issues.apache.org/jira/browse/SPARK-26849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-26849. -- Resolution: Invalid Didn't notice SPARK-26848 is created. Closing. > Introduce new option to

[jira] [Created] (SPARK-26850) Make EventLoggingListener LOG_FILE_PERMISSIONS configurable

Hua Zhang created SPARK-26850: - Summary: Make EventLoggingListener LOG_FILE_PERMISSIONS configurable Key: SPARK-26850 URL: https://issues.apache.org/jira/browse/SPARK-26850 Project: Spark Issue

[jira] [Comment Edited] (SPARK-23619) Document the column names created by explode and posexplode functions

[ https://issues.apache.org/jira/browse/SPARK-23619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763564#comment-16763564 ] Jash Gala edited comment on SPARK-23619 at 2/8/19 2:14 PM: --- I've fixed this

[jira] [Assigned] (SPARK-23619) Document the column names created by explode and posexplode functions

[ https://issues.apache.org/jira/browse/SPARK-23619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23619: Assignee: Apache Spark > Document the column names created by explode and posexplode

[jira] [Assigned] (SPARK-23619) Document the column names created by explode and posexplode functions

[ https://issues.apache.org/jira/browse/SPARK-23619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23619: Assignee: (was: Apache Spark) > Document the column names created by explode and

[jira] [Commented] (SPARK-26836) Columns get switched in Spark SQL using Avro backed Hive table if schema evolves

[ https://issues.apache.org/jira/browse/SPARK-26836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763625#comment-16763625 ] Tamas Nemeth commented on SPARK-26836: -- In the meantime I checked if using the same hive version in

[jira] [Assigned] (SPARK-26848) Introduce new option to Kafka source - specify timestamp to start and end offset

[ https://issues.apache.org/jira/browse/SPARK-26848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26848: Assignee: Apache Spark > Introduce new option to Kafka source - specify timestamp to

[jira] [Assigned] (SPARK-26848) Introduce new option to Kafka source - specify timestamp to start and end offset

[ https://issues.apache.org/jira/browse/SPARK-26848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26848: Assignee: (was: Apache Spark) > Introduce new option to Kafka source - specify

[jira] [Created] (SPARK-26849) Introduce new option to Kafka source: offset by timestamp (starting/ending)

Jungtaek Lim created SPARK-26849: Summary: Introduce new option to Kafka source: offset by timestamp (starting/ending) Key: SPARK-26849 URL: https://issues.apache.org/jira/browse/SPARK-26849 Project:

[jira] [Created] (SPARK-26848) Introduce new option to Kafka source - specify timestamp to start and end offset

Jungtaek Lim created SPARK-26848: Summary: Introduce new option to Kafka source - specify timestamp to start and end offset Key: SPARK-26848 URL: https://issues.apache.org/jira/browse/SPARK-26848

[jira] [Commented] (SPARK-23619) Document the column names created by explode and posexplode functions

[ https://issues.apache.org/jira/browse/SPARK-23619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763564#comment-16763564 ] Jash Gala commented on SPARK-23619: --- I'll fix this and raise a PR > Document the column names created

[jira] [Assigned] (SPARK-26761) Arrow optimization in native R function execution at gapply

[ https://issues.apache.org/jira/browse/SPARK-26761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26761: Assignee: Apache Spark (was: Hyukjin Kwon) > Arrow optimization in native R function

[jira] [Assigned] (SPARK-26761) Arrow optimization in native R function execution at gapply

[ https://issues.apache.org/jira/browse/SPARK-26761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26761: Assignee: Hyukjin Kwon (was: Apache Spark) > Arrow optimization in native R function

[jira] [Updated] (SPARK-26836) Columns get switched in Spark SQL using Avro backed Hive table if schema evolves

[ https://issues.apache.org/jira/browse/SPARK-26836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Nemeth updated SPARK-26836: - Affects Version/s: 2.3.1 > Columns get switched in Spark SQL using Avro backed Hive table if

[jira] [Updated] (SPARK-26836) Columns get switched in Spark SQL using Avro backed Hive table if schema evolves

[ https://issues.apache.org/jira/browse/SPARK-26836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Nemeth updated SPARK-26836: - Environment: I tested with Hive and HCatalog which runs on version 2.3.4  and with Spark 2.3.1

[jira] [Commented] (SPARK-26836) Columns get switched in Spark SQL using Avro backed Hive table if schema evolves

[ https://issues.apache.org/jira/browse/SPARK-26836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763438#comment-16763438 ] Tamas Nemeth commented on SPARK-26836: -- I just tried with Spark 2.3.1 and the same issue. I did not

[jira] [Commented] (SPARK-26841) Timestamp pushdown on Kafka table

[ https://issues.apache.org/jira/browse/SPARK-26841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763435#comment-16763435 ] Jungtaek Lim commented on SPARK-26841: -- Honestly I'm also working on the patch to add option on

[jira] [Commented] (SPARK-26841) Timestamp pushdown on Kafka table

[ https://issues.apache.org/jira/browse/SPARK-26841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763424#comment-16763424 ] Tomas Bartalos commented on SPARK-26841: Yes, I'm working on the patch, I have a working

[jira] [Commented] (SPARK-26815) run command "Spark-shell --proxy-user " failed in kerberos environment

[ https://issues.apache.org/jira/browse/SPARK-26815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763422#comment-16763422 ] Dongjoon Hyun commented on SPARK-26815: --- Hi, [~KaiXinXIaoLei]. Did you do `kinit` properly for

[jira] [Comment Edited] (SPARK-26815) run command "Spark-shell --proxy-user " failed in kerberos environment

[ https://issues.apache.org/jira/browse/SPARK-26815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763422#comment-16763422 ] Dongjoon Hyun edited comment on SPARK-26815 at 2/8/19 8:50 AM: --- Hi,

[jira] [Comment Edited] (SPARK-26819) ArrayIndexOutOfBoundsException while loading a CSV to a Dataset with dependencies spark-core_2.12 and spark-sql_2.12 (with spark-core_2.11 and spark-sql_2.11 : wo

[ https://issues.apache.org/jira/browse/SPARK-26819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763419#comment-16763419 ] Dongjoon Hyun edited comment on SPARK-26819 at 2/8/19 8:48 AM: --- Hi,

[jira] [Closed] (SPARK-26819) ArrayIndexOutOfBoundsException while loading a CSV to a Dataset with dependencies spark-core_2.12 and spark-sql_2.12 (with spark-core_2.11 and spark-sql_2.11 : working fi

[ https://issues.apache.org/jira/browse/SPARK-26819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-26819. - > ArrayIndexOutOfBoundsException while loading a CSV to a Dataset with > dependencies

[jira] [Resolved] (SPARK-26819) ArrayIndexOutOfBoundsException while loading a CSV to a Dataset with dependencies spark-core_2.12 and spark-sql_2.12 (with spark-core_2.11 and spark-sql_2.11 : working

[ https://issues.apache.org/jira/browse/SPARK-26819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-26819. --- Resolution: Duplicate Hi, [~mlebihan]. Yes. This is a known issue with Scala-2.12

[jira] [Commented] (SPARK-26847) Pruning nested serializers from object serializers: MapType support

[ https://issues.apache.org/jira/browse/SPARK-26847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763415#comment-16763415 ] Dongjoon Hyun commented on SPARK-26847: --- Thank you, [~viirya]! > Pruning nested serializers from

[jira] [Closed] (SPARK-26846) Empty Strings in dataframe are written as "" in CSV

[ https://issues.apache.org/jira/browse/SPARK-26846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-26846. - > Empty Strings in dataframe are written as "" in CSV >

[jira] [Resolved] (SPARK-26846) Empty Strings in dataframe are written as "" in CSV

[ https://issues.apache.org/jira/browse/SPARK-26846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-26846. --- Resolution: Invalid I'll close this issue because there is an option for you, [~ariyer]. >

[jira] [Commented] (SPARK-26846) Empty Strings in dataframe are written as "" in CSV

[ https://issues.apache.org/jira/browse/SPARK-26846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763412#comment-16763412 ] Dongjoon Hyun commented on SPARK-26846: --- Hi, [~ariyer]. Could you try the option `emptyValue`?

[jira] [Updated] (SPARK-26835) Document configuration properties of Spark SQL Generic Load/Save Functions

[ https://issues.apache.org/jira/browse/SPARK-26835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26835: -- Priority: Minor (was: Trivial) > Document configuration properties of Spark SQL Generic

[jira] [Updated] (SPARK-26835) Document configuration properties of Spark SQL Generic Load/Save Functions

[ https://issues.apache.org/jira/browse/SPARK-26835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26835: -- Priority: Trivial (was: Major) > Document configuration properties of Spark SQL Generic

[jira] [Issue Comment Deleted] (SPARK-26817) Use System.nanoTime to measure time intervals

[ https://issues.apache.org/jira/browse/SPARK-26817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26817: -- Comment: was deleted (was: A user of thincrs has selected this issue. Deadline: Thu, Feb 14,

[jira] [Updated] (SPARK-26835) Document configuration properties of Spark SQL Generic Load/Save Functions

[ https://issues.apache.org/jira/browse/SPARK-26835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26835: -- Affects Version/s: (was: 2.4.0) 3.0.0 > Document configuration

[jira] [Updated] (SPARK-26835) Document configuration properties of Spark SQL Generic Load/Save Functions

[ https://issues.apache.org/jira/browse/SPARK-26835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26835: -- Component/s: (was: SQL) Documentation > Document configuration

[jira] [Commented] (SPARK-26845) Avro to_avro from_avro roundtrip fails if data type is string

[ https://issues.apache.org/jira/browse/SPARK-26845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763391#comment-16763391 ] Dongjoon Hyun commented on SPARK-26845: --- cc [~Gengliang.Wang] > Avro to_avro from_avro roundtrip

[jira] [Updated] (SPARK-26836) Columns get switched in Spark SQL using Avro backed Hive table if schema evolves

[ https://issues.apache.org/jira/browse/SPARK-26836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26836: -- Labels: correctness (was: ) > Columns get switched in Spark SQL using Avro backed Hive table

[jira] [Commented] (SPARK-26836) Columns get switched in Spark SQL using Avro backed Hive table if schema evolves

[ https://issues.apache.org/jira/browse/SPARK-26836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763393#comment-16763393 ] Dongjoon Hyun commented on SPARK-26836: --- Thank you for reporting, [~treff7es]. Did you see this