[jira] [Comment Edited] (SPARK-13510) Shuffle may throw FetchFailedException: Direct buffer memory

2019-04-24 Thread belvey (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825756#comment-16825756 ] belvey edited comment on SPARK-13510 at 4/25/19 5:58 AM: - @Mike in my case I set

[jira] [Comment Edited] (SPARK-13510) Shuffle may throw FetchFailedException: Direct buffer memory

2019-04-24 Thread belvey (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806307#comment-16806307 ] belvey edited comment on SPARK-13510 at 4/25/19 5:57 AM: - [~shenhong] , hello

[jira] [Commented] (SPARK-13510) Shuffle may throw FetchFailedException: Direct buffer memory

2019-04-24 Thread belvey (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825756#comment-16825756 ] belvey commented on SPARK-13510: @Mike in my case I set it to '536870912' (512m) ,and it can be set to

[jira] [Updated] (SPARK-27553) Operation log is not closed when close session

2019-04-24 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27553: - Description: On Window 1. start spark-shell 2. start hive server in shell by

[jira] [Commented] (SPARK-27553) Operation log is not closed when close session

2019-04-24 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825751#comment-16825751 ] Hyukjin Kwon commented on SPARK-27553: -- The cause will probably be a file not being closed

[jira] [Resolved] (SPARK-27554) org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]

2019-04-24 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27554. -- Resolution: Invalid No evidence that it's an issue in Spark. Probably it will be a problem

[jira] [Updated] (SPARK-27554) org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]

2019-04-24 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27554: - Description: {code} #basic env JAVA_HOME=/usr/java/jdk1.8.0_181

[jira] [Updated] (SPARK-27554) org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]

2019-04-24 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27554: - Description: {code} #basic env JAVA_HOME=/usr/java/jdk1.8.0_181

[jira] [Commented] (SPARK-27551) Uniformative error message for mismatched types in when().otherwise()

2019-04-24 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825749#comment-16825749 ] Dongjoon Hyun commented on SPARK-27551: --- Hi, [~huonw]. I updated this from `BUG` to `Improvement`.

[jira] [Updated] (SPARK-27551) Uniformative error message for mismatched types in when().otherwise()

2019-04-24 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27551: -- Priority: Minor (was: Major) > Uniformative error message for mismatched types in

[jira] [Commented] (SPARK-27557) Add copybutton to spark Python API docs for easier copying of code-blocks

2019-04-24 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825746#comment-16825746 ] Hyukjin Kwon commented on SPARK-27557: -- please go ahead for a PR. > Add copybutton to spark Python

[jira] [Updated] (SPARK-27551) Uniformative error message for mismatched types in when().otherwise()

2019-04-24 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27551: -- Issue Type: Improvement (was: Bug) > Uniformative error message for mismatched types in

[jira] [Commented] (SPARK-27555) cannot create table by using the hive default fileformat in both hive-site.xml and spark-defaults.conf

2019-04-24 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825747#comment-16825747 ] Hyukjin Kwon commented on SPARK-27555: -- can you post a self-contained reproducer please? > cannot

[jira] [Updated] (SPARK-27557) Add copybutton to spark Python API docs for easier copying of code-blocks

2019-04-24 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27557: - Description: Add a non-intrusive button for python API documentation, which will remove ">>>"

[jira] [Commented] (SPARK-27558) NPE in TaskCompletionListener due to Spark OOM in UnsafeExternalSorter causing tasks to hang

2019-04-24 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825743#comment-16825743 ] Hyukjin Kwon commented on SPARK-27558: -- Please avoid to set Critical+ which is usually reserved for

[jira] [Updated] (SPARK-27558) NPE in TaskCompletionListener due to Spark OOM in UnsafeExternalSorter causing tasks to hang

2019-04-24 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27558: - Priority: Major (was: Critical) > NPE in TaskCompletionListener due to Spark OOM in

[jira] [Resolved] (SPARK-27559) Nullable in a given schema is not respected when reading from parquet

2019-04-24 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27559. -- Resolution: Duplicate > Nullable in a given schema is not respected when reading from parquet

[jira] [Commented] (SPARK-27559) Nullable in a given schema is not respected when reading from parquet

2019-04-24 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825742#comment-16825742 ] Hyukjin Kwon commented on SPARK-27559: -- It's a dup of SPARK-16472 > Nullable in a given schema is

[jira] [Updated] (SPARK-27559) Nullable in a given schema is not respected when reading from parquet

2019-04-24 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27559: - Component/s: (was: Spark Core) SQL > Nullable in a given schema is not

[jira] [Commented] (SPARK-27505) autoBroadcastJoinThreshold including bigger table

2019-04-24 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825739#comment-16825739 ] Hyukjin Kwon commented on SPARK-27505: -- I meant copy-and-past-able code to reproduce this error >

[jira] [Commented] (SPARK-13510) Shuffle may throw FetchFailedException: Direct buffer memory

2019-04-24 Thread Mike Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825721#comment-16825721 ] Mike Chan commented on SPARK-13510: --- Hi [~belvey] I'm having similar issue and our 

[jira] [Created] (SPARK-27563) automatically get the latest Spark versions in HiveExternalCatalogVersionsSuite

2019-04-24 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-27563: --- Summary: automatically get the latest Spark versions in HiveExternalCatalogVersionsSuite Key: SPARK-27563 URL: https://issues.apache.org/jira/browse/SPARK-27563

[jira] [Commented] (SPARK-20462) Spark-Kinesis Direct Connector

2019-04-24 Thread Long Sun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825669#comment-16825669 ] Long Sun commented on SPARK-20462: -- [~gaurav24], [~arushkharbanda], Any update on this? "The user

[jira] [Updated] (SPARK-27562) Complete the verification mechanism for shuffle transmitted data

2019-04-24 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27562: Description: We've seen some shuffle data corruption during shuffle read phase. As described in

[jira] [Updated] (SPARK-27562) Complete the verification mechanism for shuffle transmitted data

2019-04-24 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27562: Description: SPARK-27562 We've seen some shuffle data corruption during shuffle read phase. As

[jira] [Updated] (SPARK-27562) Complete the verification mechanism for shuffle transmitted data

2019-04-24 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27562: Description: We've seen some shuffle data corruption during shuffle read phase. As described in

[jira] [Updated] (SPARK-27562) Complete the verification mechanism for shuffle transmitted data

2019-04-24 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27562: Description: I will update this lately. > Complete the verification mechanism for shuffle transmitted

[jira] [Commented] (SPARK-24192) Invalid Spark URL in local spark session since upgrading from org.apache.spark:spark-sql_2.11:2.2.1 to org.apache.spark:spark-sql_2.11:2.3.0

2019-04-24 Thread Aseem Patni (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825616#comment-16825616 ] Aseem Patni commented on SPARK-24192: - export SPARK_LOCAL_HOSTNAME=localhost should also fix this. 

[jira] [Created] (SPARK-27562) Complete the verification mechanism for shuffle transmitted data

2019-04-24 Thread feiwang (JIRA)
feiwang created SPARK-27562: --- Summary: Complete the verification mechanism for shuffle transmitted data Key: SPARK-27562 URL: https://issues.apache.org/jira/browse/SPARK-27562 Project: Spark

[jira] [Assigned] (SPARK-27312) PropertyGraph <-> GraphX conversions

2019-04-24 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-27312: - Assignee: Weichen Xu > PropertyGraph <-> GraphX conversions >

[jira] [Assigned] (SPARK-27300) Create the new graph projects in Spark and set up build/test

2019-04-24 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-27300: - Assignee: Martin Junghanns (was: Xiangrui Meng) > Create the new graph projects in

[jira] [Assigned] (SPARK-27300) Create the new graph projects in Spark and set up build/test

2019-04-24 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-27300: - Assignee: Xiangrui Meng > Create the new graph projects in Spark and set up build/test

[jira] [Updated] (SPARK-27561) Support "lateral column alias references" to allow column aliases to be used within SELECT clauses

2019-04-24 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27561: --- Description: Amazon Redshift has a feature called "lateral column alias references":

[jira] [Updated] (SPARK-27561) Support "lateral column alias references" to allow column aliases to be used within SELECT clauses

2019-04-24 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27561: --- Description: Amazon Redshift has a feature called "lateral column alias references:

[jira] [Created] (SPARK-27561) Support "lateral column alias references" to allow column aliases to be used within SELECT clauses

2019-04-24 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-27561: -- Summary: Support "lateral column alias references" to allow column aliases to be used within SELECT clauses Key: SPARK-27561 URL: https://issues.apache.org/jira/browse/SPARK-27561

[jira] [Commented] (SPARK-27542) SparkHadoopWriter doesn't set call setWorkOutputPath, causing NPEs when using certain legacy OutputFormats

2019-04-24 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825560#comment-16825560 ] Josh Rosen commented on SPARK-27542: [~shivuson...@gmail.com], unfortunately I don't have a

[jira] [Updated] (SPARK-27530) FetchFailedException: Received a zero-size buffer for block shuffle

2019-04-24 Thread Adrian Muraru (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrian Muraru updated SPARK-27530: -- Description: I'm getting this in a large shuffle: {code:java}

[jira] [Created] (SPARK-27560) HashPartitioner uses Object.hashCode which is not seeded

2019-04-24 Thread Andrew McHarg (JIRA)
Andrew McHarg created SPARK-27560: - Summary: HashPartitioner uses Object.hashCode which is not seeded Key: SPARK-27560 URL: https://issues.apache.org/jira/browse/SPARK-27560 Project: Spark

[jira] [Created] (SPARK-27559) Nullable in a given schema is not respected when reading from parquet

2019-04-24 Thread colin fang (JIRA)
colin fang created SPARK-27559: -- Summary: Nullable in a given schema is not respected when reading from parquet Key: SPARK-27559 URL: https://issues.apache.org/jira/browse/SPARK-27559 Project: Spark

[jira] [Updated] (SPARK-27558) NPE in TaskCompletionListener due to Spark OOM in UnsafeExternalSorter causing tasks to hang

2019-04-24 Thread Alessandro Bellina (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessandro Bellina updated SPARK-27558: --- Description: We see an NPE in the UnsafeInMemorySorter.getMemoryUsage function (due

[jira] [Created] (SPARK-27558) NPE in TaskCompletionListener due to Spark OOM in UnsafeExternalSorter causing tasks to hang

2019-04-24 Thread Alessandro Bellina (JIRA)
Alessandro Bellina created SPARK-27558: -- Summary: NPE in TaskCompletionListener due to Spark OOM in UnsafeExternalSorter causing tasks to hang Key: SPARK-27558 URL:

[jira] [Commented] (SPARK-25888) Service requests for persist() blocks via external service after dynamic deallocation

2019-04-24 Thread Attila Zsolt Piros (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825248#comment-16825248 ] Attila Zsolt Piros commented on SPARK-25888: I am working on this. > Service requests for

[jira] [Created] (SPARK-27557) Add copybutton to spark Python API docs for easier copying of code-blocks

2019-04-24 Thread Sangram G (JIRA)
Sangram G created SPARK-27557: - Summary: Add copybutton to spark Python API docs for easier copying of code-blocks Key: SPARK-27557 URL: https://issues.apache.org/jira/browse/SPARK-27557 Project: Spark

[jira] [Resolved] (SPARK-27550) Fix `test-dependencies.sh` not to use `kafka-0-8` profile for Scala-2.12

2019-04-24 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27550. --- Resolution: Fixed Assignee: Dongjoon Hyun Fix Version/s: 2.4.3 This is

[jira] [Updated] (SPARK-27556) Exclude com.zaxxer:HikariCP-java7 from hadoop-yarn-server-web-proxy

2019-04-24 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27556: Description: {noformat} [INFO] | +-

[jira] [Commented] (SPARK-25262) Make Spark local dir volumes configurable with Spark on Kubernetes

2019-04-24 Thread Udbhav Agrawal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825134#comment-16825134 ] Udbhav Agrawal commented on SPARK-25262: Thanks [~rvesse] i will try to work on this > Make

[jira] [Created] (SPARK-27556) Exclude com.zaxxer:HikariCP-java7 from hadoop-yarn-server-web-proxy

2019-04-24 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-27556: --- Summary: Exclude com.zaxxer:HikariCP-java7 from hadoop-yarn-server-web-proxy Key: SPARK-27556 URL: https://issues.apache.org/jira/browse/SPARK-27556 Project: Spark

[jira] [Commented] (SPARK-25262) Make Spark local dir volumes configurable with Spark on Kubernetes

2019-04-24 Thread Rob Vesse (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825132#comment-16825132 ] Rob Vesse commented on SPARK-25262: --- [~Udbhav Agrawal] Yes I think an approach like that would be

[jira] [Commented] (SPARK-25262) Make Spark local dir volumes configurable with Spark on Kubernetes

2019-04-24 Thread Udbhav Agrawal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824993#comment-16824993 ] Udbhav Agrawal commented on SPARK-25262: [~rvesse] for the second part, can we move out

[jira] [Updated] (SPARK-27354) Move incompatible code from the hive-thriftserver module to sql/hive-thriftserver/v1.2.1

2019-04-24 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27354: Summary: Move incompatible code from the hive-thriftserver module to sql/hive-thriftserver/v1.2.1

[jira] [Updated] (SPARK-27555) cannot create table by using the hive default fileformat in both hive-site.xml and spark-defaults.conf

2019-04-24 Thread Hui WANG (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui WANG updated SPARK-27555: - Description: I already seen https://issues.apache.org/jira/browse/SPARK-17620  and 

[jira] [Created] (SPARK-27555) cannot create table by using the hive default fileformat in both hive-site.xml and spark-defaults.conf

2019-04-24 Thread Hui WANG (JIRA)
Hui WANG created SPARK-27555: Summary: cannot create table by using the hive default fileformat in both hive-site.xml and spark-defaults.conf Key: SPARK-27555 URL: https://issues.apache.org/jira/browse/SPARK-27555

[jira] [Assigned] (SPARK-27512) Decimal parsing leads to unexpected type inference

2019-04-24 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-27512: Assignee: Hyukjin Kwon > Decimal parsing leads to unexpected type inference >

[jira] [Resolved] (SPARK-27512) Decimal parsing leads to unexpected type inference

2019-04-24 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27512. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24437

[jira] [Updated] (SPARK-27554) org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]

2019-04-24 Thread Jepson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jepson updated SPARK-27554: --- Description: {code:java}  {code} #basic env JAVA_HOME=/usr/java/jdk1.8.0_181